I don't have any specific insights into how it's enforced on the ground but it doesn't seem particularly difficult.
You cannot climb a mountain like that in a few hours or on your own. Even assuming you're a specialist and don't want to rely on local help (i.e. you're eshewing the “expedition style” climbing that's typical of the Himalayas), you would need special equipment, which can easily be detected at the border or somewhere on the way.
As most mountains in the Himalayas are relatively remote, even the approach might be challenging. Again, unless you are being very deliberate about it (think special operation forces), you cannot travel places like that without assistance or interaction with the locals, so many opportunities for people to wonder what you are trying to do and either report you or get involved themselves.
That's especially true in Bhutan as it sees relatively few tourists (deliberately so) and requires them to book their trip through an approved tour operator. You cannot just show up there and hope to hide in a huge crowd.