Yes, the U.S. accepts bus tickets as onward travel. I've used it myself. They were even OK accepting a bus ticket from Seattle to Vancouver when I flew into Boston.
The trick you'll have though is that you're running very close to the 90 days limit on the VWP, which does not reset if you go to Canada. As a result, if you stay any later than three months in total in North America, you'll have to ensure your flight out of Canada does not go through the US - you'd need a direct flight to the UK for example.
As for needing a return flight, another time I went to Canada from the UK (Gatwick -> Edmonton -> Vancouver) I had a one-way ticket. I also then purchased a bus ticket from Vancouver to Seattle as my onward travel - remember, Canada will also want to know how you're getting out.
What I've found though is you do want printed proof of this travel. I tried flying to Argentina from New Zealand (I had a weird combination of tickets) and it looked like I only had a one-way in, and even though I suggested I'd just grab a bus to Chile, they wouldn't accept that - I had to go and print out my other flight that I had out of Argentina.
Evidence of planning, dates, and bookings is what they're looking for.