Can I enter the US with a ESTA waiver, leave after 10 days to go to Mexico for 12 days, and then re-enter the US under a J-1 visa to study for four months?
I am entering the US more then 30 days before beginning of J-1 visa.
In principle, this will not work. When entering the US by air under the VWP, you're required to have a round-trip ticket that takes you out of the US, and according to the explicit definition in the regulations, a trip that ends in Mexico is not "round" unless you're a resident there. Since airlines are supposed to enforce these rules, there's a risk that you'll be denied boarding at your origin airport.
In practice, there are anecdotal reports of the round-trip-ticket requirement not being enforced to the letter by either airlines or border guards. So it may work, but it'll be hit-and-miss at best.