According to the US Department of State, Bureau of Counsular Affairs,
Trips to Canada, Mexico, or nearby Islands
If you are admitted to the United States under the VWP, you may take a short trip to Canada, Mexico, or a nearby island and generally be readmitted to the United States under the VWP for the remainder of the original 90 days granted upon your initial arrival in the United States. Therefore, the length of time of your total stay, including the short trip, must be 90 days or less. See the Customs and Border Protection (CBP) website.
The last sentence is a link to the Customs and Border Patrol website, but it discusses what they call Closed-Loop Voyages. C-L V's are, for instance, where tourists from Europe may fly into Miami and board a cruise ship going to some Carribean Islands and returning to Miami. The rules allow for this. What you are discussing is a personal trip, at your own convenience and itinerary, and using a vehicle operated by you. In that case, you are taking yourself out of the US to another country, not as part of an organized group or tour. If your trip to Mexico takes you beyond the allowed 90-days, yes, you will not be allowed back into the US, without applying for and being granted a new visa. I do not believe you would be elligible for a second waived visa/second 90-day period.