I am a Swiss (= not EU) citizen and I'm going to fly to Montreal, rent a RV and then cross into the US (Vermont). Since I won't be included in the ESTA program, what do I need in order to pass the US border? Are they going to give me a visa at the border?
Officially if entering the US via land border then you do not need an ESTA, however you are still able to use the Visa Waiver Program as you come from a country that is allowed under the Visa Waiver Program.
At the border you will be required to fill in a I94W form, be photographed/finger-printed, and pay a US$6 fee.
There have recently been reports that people who have an ETSA have been able to bypass the process above. Given that an ESTA is only US$14 it may be worth applying for one anyway - at worst you end up $14 out of pocket. As best, you save yourself some trouble at the border.
Also make sure that your rental company is aware that you are driving into the US, and that you have suitable insurance coverage for the vehicle. Not all companies allow you to cross the border, although most do.