If I travel to the USA on an au pair J1 visa, but don't complete the year of the contract, I understand that I have to leave the country immediately and forfeit the 30 day grace period that comes at the end of the year. However if I leave and go to Canada or Mexico and then re-enter on an existing ESTA that I have that is still valid, could I stay for 90 days? I am a Brit.
Technically, yes you can - with one major condition which is that anyone entering the US under the Visa Waiver Program (VWP) by air must hold an onward ticket that takes them outside of North America. ie, you will require an onward ticket to the UK (or somewhere else) in order to re-enter the US on VWP. If you are entering by land then this is not technically a requirement, but not having such a ticket can still be used against you.
As with all border crossings - especially when using the Visa Waiver Program (VWP) - your entry is at the discretion of the immigration officials.
If they believe that you are entering the US for purposes that are not allowed under the VWP (eg, if they suspect you are intending to work), or if they believe that you are likely to not leave within 90 days, then they can refuse your entry into the US, and when entering using the VWP there is no way to appeal such a decision.
The fact that you are leaving once your J1 is no longer valid is likely a positive sign (it shows that you're not trying to break the rules) so I suspect that you will be fine doing this - but be ready to answer questions regarding what you intend to do in the US, and how/when you intend to leave, and as mentioned above be ready to show proof of onward travel which is mandatory if entering by air, and still probably a good idea if entering by land.