Hello and welcome to Travel Stackexchange. I'll try to answer your questions, however I currently can't provide sources, just from memory:
DON'T ATTEMPT IT!
if I apply for ESTA Visa Waiver how would they know I have been to Syria?
Because it's a French passport, the risk is quite high that they have the info from the French authorities. I would not recommend trying to get an ESTA, the B2 visa would probably be less of a hassle.
As Traveller mentions in the comments, the VWP (Visa Waiver Program) participating countries "are required to share intelligence and cooperate in matters of security with the United States [and] you can’t be sure exactly what immigration data is shared".
What are the risks doing so ?
The almost 100% risk is being banned from the VWP formally - which you technically already are, as you aren't eligible with your travel history - and maybe also being banned from any other visitor's visa to the U.S. for misrepresentation, fines, deportation etc. when discovered.
Would it be better to apply for ESTA with my British or French passport ?
The best way would be to apply for a B visa as officially required.