No you don't. You can transfer to other terminals and exit there. However, you must be travelling with hand luggage only, otherwise you will have to return to the original terminal land-side to collect your luggage, which would be difficult as you would need to get past customs to collect it.
Here's an explanatory post on flytalk.
One of the quirks of the airside transfer buses and the flight
connection areas in each terminal is that you can arrive on an INT
flight at T5 but cross the UK border at another terminal. In fact you
can also do this in reverse, so arrive on an INT at T3 for example,
and cross the UK border at T5. Whilst this is not really recommended
normally, you may wish to do this if you have your car or left luggage
at another terminal or you find the UK border at the terminal you
arrive at to be very busy.