It's not merely the agent (or airline) refusing to book it. It's illegal per Canadian law.
Honestly, this part surprises me
They are willing to book me on two separate itineraries where I leave Montreal and go somewhere in the U.S. and then on a separate itinerary proceed from the U.S. location to Vancouver.
That's still illegal if you're not actually spending some time in the U.S. A short layover doesn't count and can cause the airline to be fined. Even flying, say, American from Montreal to Chicago and United from Chicago to Vancouver is still illegal. My guess would be that the agent you spoke with simply wasn't familiar with why the system wasn't letting her book it on a single ticket for you. The U.S. has fined airlines based in Korea over that in the past when passengers were purchasing tickets on, say, Korean Air or Asiana from the mainland U.S. to Incheon Airport in Seoul and then taking a Korean low-cost carrier to Guam or Saipan, which are part of the U.S.
As Kate's answer describes, this sort of situation is known as cabotage and almost every country in the world refuses to allow it both for air transport and boat transport except with specific exceptions. The U.S. and Canada do have a few such specific exceptions for each other, but it's mostly just limited to cases where a route operated by a domestic carrier would not be economically feasible. For example, there are some small Alaskan islands that are served by Canadian-flagged ferry service, as I recall. As Matthew pointed out, perhaps the most notable exception is the European Common Aviation Area, wherein EU member countries and a few others in the region allow carriers of other ECAA member countries to operate domestic flights within their countries.
There is, however, one way to use your Delta miles to fly within Canada or to earn Delta miles for flights between Canadian cities: Delta is a partner with WestJet. While you can't buy a revenue ticket from Toronto to Vancouver through Delta, they will happily let you redeem your Delta SkyMiles for a WestJet flight from YYZ to YVR. From a quick search, they appear to run 10,000 miles + about $36 CAD in taxes and fees each way.
For example, when I just searched YYZ to YVR one-way with miles on Delta's website, this option (along with several others at the same price, all operated by WestJet) came up:
WestJet flight redemption option with Delta SkyMiles
If you want to earn Delta miles (and/or status credit) for flights within Canada, again, WestJet is the way to do it. You can't book a revenue WestJet domestic flight through Delta, but you can add your Delta SkyMiles number to a reservation booked through WestJet in order to earn Delta miles and status credit instead of earning in WestJet's program. The tables telling you how much you'll earn for a given WestJet fare class are located on Delta's website.