Can I travel back to the UK from Middle East with transit in Amsterdam with 4 months left on my British passport ?
Assuming you're travelling on a single ticket, UK entry requirements is what matters - the Netherlands doesn't impose any document requirements of its own for transit.
TIMATIC, the database used by airlines, states:
Passengers with a British passport with nationality British Citizen are allowed to enter with an expired passport.
Now, the Middle Eastern country likely won't let you exit on an expired passport, but no country imposes a 3/6-month requirement for foreigners exiting.
As such, you're all good, before and after the "definite" Brexit.
Providing you travel on or before 31 December 2020, your passport should be valid for the proposed duration of your stay; you do not need any additional period of validity on your passport beyond this.
From 1 January 2021, on the day you travel, you’ll need your passport to both:
have at least 6 months left
be less than 10 years old (even if it has 6 months or more left)