For Mexico:
As per Timatic, the database used by Airlines:
Passport Exemptions:
Nationals of Mexico with a:
national ID card; or
certified copy of birth certificate; or
consular registration; or
certificate of Mexican nationality; or
naturalization letter; or
Matricula Consular; or
Constancia de Identificacion; or
voter's certificate with photo.
Any of These, in other words, are accepted for entry to Mexico.
So no, your children do not need Mexican passports to enter Mexico, even by air
For the US:
As per Timatic (which, for the US, covers air travel only):
Passport Exemptions:
- Passengers with a Transportation Letter.
- Passengers with an emergency or a temporary passport.
So to enter the US by air, they do need US passports.
By land, however, US passport cards, enhanced state ID cards and (for children under 16) CRBA:s are valid too, as well as possibly other IDs that I'm unaware of
In summary:
To travel to and from Mexico by air, they need a US passport and any Mexican document mentioned above. By land, a US passport/passport card/enhanced state ID/CRBA if under 16, and any Mexican document mentioned above