In general if there is a seat available, you should aim to remain seated while the train, bus or tram is moving. This is a simple safety consideration: it's more dangerous to be standing than seated in the event of a sudden stop or crash. This includes utilising any seats designated as priority.
Priority seating labelling varies across networks, with the more modern signage saying something to the effect of
Priority Seating. Please be prepared to give up your seat to passengers with mobility needs
I think the key word here is prepared, you should be on the look out for others that might need it more at stops or stations if you are seated in a priority seat, not glued to a phone, asleep or otherwise oblivious to your surroundings. Obviously if you're sat in a normal seat and someone asks for it you should of course oblige, that's just common courtesy, but the be prepared part is less relevant.
Transport for London says the following:
All buses, Tubes, trains and trams have clearly marked priority seats for anyone who needs them. If one isn't available, ask if someone will give up a seat.
Source: Transport for London website
The implication here is that they can be used by anyone but they should move if asked.
CrossCountry trains is more explicit:
Other passengers may sit in these seats, but wherever possible they should be offered to anyone who needs them as a priority.
Source: CrossCountry Trains website
West Midlands Trains also indicates the able bodied may use priority seats:
When asked politely, most people will be quite happy to move from the seat (unless they need it too) to allow you to sit there instead.
Source: West Midlands Trains website
Other networks I looked into were less explicit in their language, merely referencing priority seating, but equally none of them state able bodied passengers making use of those seats is prohibited. Virgin Trains (which typically runs longer, inter-city routes) states that priority seating must be reserved in advance (Virgin Trains).
Priority seating is not to be confused with the blue badge disabled parking system where it is a criminal offence to park in (publicly owned) disabled spaces without displaying a valid blue badge.