UK uses left-hand traffic. Traffic flows on the left side, people walk on the left. Is there any reason to stand on the right while using escalators?
Why do people do it today? Because of the multitude of signs that say to do so!
If you're asking why it started, this BBC article (includes video) holds what may be the answer...
If if you've ever wondered why we do it, and why so many tourists get lost in translation, the answer may lie in a piece of film from the 1920s ... partly shot at Waterloo station and it was one of the first times the Tube system had appeared on film.
The escalator design had a diagonal step-off, clearly meant for the right foot first so standing on the right made sense.
(the video also includes clips of 1920s tube-escalator-ettiquette cinema humour such as a soldier tripping up at the bottom of an escalator because he's drilled to always march left foot first)
I moved to the UK for 4ish years back in 2007. In advance, I read the Wikitravel and Wikipedia articles to prepare a bit, and in one of them it mentioned that you stand on the right, and walk on the left. And I laughed and thought - 'no way would people actually follow that'.
Then I got to London, got my first barge done quickly when I was standing on the wrong side. I was amazed - could people really get upset about it?
Soon enough, and sure enough, it was me getting annoyed when I come across someone on the wrong side, and I'll admit to a bit of barging occasionally too. It's funny how quickly you adjust.
Now, as to the why? I also wondered at first - if you drive on the left, why not walk on the left?
And then it hit me, that's exactly what you're doing. Escalators, remember, aren't meant to substitute for walking, but to help. And you walk on the left, up the escalator. It's when you're NOT walking, that you stand on the right.
And that was how I rationalised it. After that, it worked fine in my head.
Now that a system is in place, as you mentioned it needs to make sense to others. Australia, for example has the reverse. And thus all the signs you'll see all over the tube - 'keep to the right', 'walk on the left' and so on.
And so society continues on.
Psychologically for right handed human beings it's a big thing to allow someone to stand on their right. Multiple researches have shown, that usually it is the person with superiority, that is allowed to stand on the right on the lesser "qualified" one. On the old continent, superiority is a big thing (especially in countries with long history such as the UK). This is why in most western civilizations you'll notice people protecting their right side and allowing more traffic on the left.