I have a long layover in Heathrow, like 6 hours, and I have a Priority Pass card to access several lounges on terminals 1 and 3. But I will be arriving to T5, and leaving from there as well. Is there any way to change terminals so I can relax at some lounge?
As has been pointed out by several people in the comments, your question is similar to the question Can we switch terminals at Heathrow (LHR) for shopping only?, and indeed that question does give some relevant advice - however there is a little bit more to it in this case.
Priority Pass currently have 4 lounges available at LHR - A ServisAir lounge in each of terminals 1 and 3, the No 1 Heathrow lounge in terminal 3, and the Air Canada Arrivals Lounge in Terminal 3.
The first 3 of these lounges are AIRSIDE, and thus the "shopping" question above will give you your answer - you can not access them from terminal 5.
The last lounge - the Air Canada lounge - is LANDSIDE, and thus presuming you are able to enter the UK (ie, you have a relevant visa and/or do not require a visa) then at first it may appear that you could go through immigrations, and then catch the bus to terminal 3 to access this lounge.
However, unfortunately, that's not the case. The Air Canada lounge only allows access for Priority Pass customers who are arriving on a flight that arrives into terminal 3. As you arrived into terminal 5 you will NOT be allowed access this lounge.
So unfortunately, no, you will not be able to access any of the lounges in terminal 1 or 3.