With regard to sites devoted to Charles Babbage, he has been awarded a blue plaque in London which you can visit at 1 Dorset Street (more info: http://openplaques.org/plaques/3061) at the house where he lived for 40 years. This is though just the plaque, the house its self is not open to the general public and there isn't an information board.
I don't belive there are any museums devoted entirely to him in the UK: I quickly searched through this list and didn't notice anything obvious. The Science Museum in London appears to be your best bet and you are already aware of it: according to Atlas Obscura they have a working difference engine and a section of his brain. There is also the Center for computing history in Cambridge and the Bletchley park museam near Milton Keynes.