My boyfriend overstayed his visa in the US without realizing (long story) and he was traveling on his British passport via ESTA. His ESTA was revoked as a result. He has now applied for a visa on his South African passport. Will he be able to enter the country if his visa is approved?
Once your friend has overstayed a VWP visit once he is forever barred from entering under the VWP again, no matter which passport he is using.
The only way to be readmitted in this situation is indeed to apply for a visa in advance. It probably doesn't matter much which of the two passports he applies for the visa with; but he will need to be honest about the overstay in his application in either case.
For the best chances of success, he should be sure to append a convincing explanation of the earlier overstay to his application. Do not wait to be asked!
(Applying with the South African passport while pretending that the overstay didn't happen would probably be a quick recipe for being refused for deception, which would make it somewhere between extremely hard and impossible to get any U.S. visa afterwards).