Absolutely no certainty about this, but this is probably due to the fact your trip was booked from Vienna and you were on the return leg.
This means you don't have a return ticket (to leave Austria as a UK citizen) they know of. Now while the SchegenSchengen rules don't actually require you to have a return/onward ticket to leave the Schengen Area when you are a visitor (they instead require that you have the means to leave the Schengen Area, having a ticket out being just one way to demonstrate that), Timatic still states:
Warning:
Visitors not holding return/onward tickets could be refused entry.
And some airlines are known to take that as "you must have a return/onward ticket".
So now they could ask you for a return/onward ticket, but:
- This is the return trip on a VIE-LHR-VIE trip
- It was probably booked from Vienna
- Maybe you even have an address in Vienna registered with them
So, all in all, it is much more likely that you are a resident in Austria rather than someone going there as a visitor, so instead, they ask for the residence permit. I suppose if you had told them you didn't have one they would have switched to asking for a ticket.