Theoretically, you need a passport, but it's very unlikely that anyone will ask for it.
Hungarian law says (10. §)
A külföldi a magyarországi tartózkodása alatt köteles személyazonosító
okmányait, illetve a tartózkodása jogszerűségét igazoló okmányokat a
törvényben erre feljogosított hatóságoknak ellenőrzés céljából átadni.
which means (my translation, so not legal text)
A foreign person during their stay in Hungary is obligated to present their identification documents and documents supporting the
lawfulness of their presence to the entitled authorities for
verification.
According to the relevant page (couldn't find it in English) of the Consular Services, you need a passport or identity card for traveling to Hungary, and since there's no ID card in the UK, that leaves only your passport as "documents supporting the lawfulness of your presence", so you must have it on you.
On the other hand, unless you walk around shady places at night alone or with a visibly drunk company, police won't care about you. (I live in Budapest and I can't think right now of anyone I know that was asked by police to identify themselves).