If you have a British passport and applying for university in the US, do you need a visa or is ESTA enough?
You need an F-1 visa for Independent study and a J-1 visa for exchange study.
You will apply for it after having been admitted to the university and received an I-20 (for F-1) or DS-2019 (for J-1) document.
When entering the US (even when undertaking trips during your studies) always present your passport at the visa page (and not the ID page) as well as the I-20 or DS-2019 document. Otherwise this might happen
The terms of the visa waiver program, which govern your entry after you fly to the US using ESTA, do not permit enrollment in a university. They also do not permit you to remain in the US for longer than 90 days.
To travel for an interview or other meetings associated with your application, however, you may indeed use the visa waiver program (and ESTA) if you qualify.
If you want to know what kinds of visa do apply in your situation, you should ask at https://expatriates.stackexchange.com.