It depends how you enter the US. ESTA is only valid and useful if you enter by air or sea (and ferries do not always count in this category).
So, as I suppose you will enter the US with a car (or via bus/train), you do not need ESTA at all. You just need to show up at the border and you will be given authorization to enter if everything is all right. The price of the entrance is around $6.
But you still have the issue of the 90 days. What will happen is that when you enter the US you will be given a I-94 which is a document used to track your entering/leaving dates. It should be given back to the Canadian custom agent if you leave for Canada, or to the airline if you leave by air.
So in your case, you should get an I-94 when first entering US, then give your I-94 when you leave the first time, and the next time you enter you get a new entrance authorization with a new I-94 that will be valid another 3 months.