Contrary to the popular opinion I would strongly recommend against the trains. Reason being the trains run slower then buses, for example the Maple Leaf which runs NYP-Toronto runs 12h 30min and makes a bunch of stops. Adirondak which runs NYP-Montreal is kind of in the same boat and runs 10h 30min. Granted that a big chunk of both runs along the Hudson river and provides quite spectacular views. But you lose a day nontheless.
The bus in my opinion is a better option because both destinations are served by multiple companies.
To Montreal, the traditional Greyhound (6 buses daily; 7h 20 - 8h 50min, $73-105) plus you can get on New York Trailways (2 additional buses daily; 7½ - 8½ h, $84).
To Toronto there are 3 options: 2 of them I mentioned above. Another one is Megabus (I hate their website) which is low cost express service offered by Coach USA. The cost per ticket to Toronto for example could be about half of what Amtrak will charge with purchase 4 days in advance.
Some of the buses run overnight so it may be more advantageous to do that then the train which leaves NYC in the morning and run by day.
On top of all of this there are websites like Amovens which allow you to find a possible car ride to your destination too. Predictability of schedule in this case is unfortunately out the window.