If you are caught with smaller amounts of alcohol exceeding the allowed amount, the customs will offer you to accept a "forenklet forelegg", a kind of fixed rate fine used in the Norwegian legal system for minor issues, e.g. traffic violations or breach of the custom regulations.
For beverages with 22.1 to 60% alcohol by volume, the rates are as such (updated December 2018):
- up to 1 litre more than the allowed amount: 400 NOK (41€)
- up to 2 litres: 800 NOK (83€)
- up to 3 litres: 1600 NOK (165€)
- up to 4 litres: 2500 NOK (260€)
- up to 5 litres: 3400 NOK (350€)
- up to 6 litres: 4500 NOK (465€)
- up to 8 litres: 7000 NOK (725€)
- up to 10 litres: 9500 NOK (980€)
For beverages with 7.1 to 22% alcohol by volume, the rates are the same, but the limits are doubled, e.g. 400 NOK for up to 2l too much, 800 NOK for up to 4l too much, etc.
For beverages with 2.5 to 7% alcohol by volume, the amount are ten times higher, e.g. 400 NOK for up to 10l too much, 800 NOK for up to 20l too much, etc.
For beverages with less than 2.5% alcohol, there are no special customs regulations in Norway. Beverages with more than 60% alcohol are generally prohibited and attempting to smuggle such beverages will be punishable similar to smuggling of drugs and controlled substances.
There are similar fixed rate fines defined for exceeding the allowed amounts of cigarettes, tobacco, meat, fuel and exceeding the value limit for other goods. A complete list can be found in Norwegian here.
If you exceed the allowed amounts in multiple categories, the fixed rate fines (forenklet forelegg) will be added up to 20000 NOK (2070€). If the accumulated fixed fines exceed 20000 NOK or you exceed the amounts in the fixed fine list within any of the categories, the customs will hand you over to the police and demand a legal proceeding, in which you will be fined by a court. If you deny or refuse to accept the "forenklet forelegg", the customs will also commence a legal proceeding and you will be judged by a court.
If the issue is resolved with a "forenklet forelegg", you are allowed to keep the allowed amount of alcohol, but the exceeding amount will be confiscated.
So to answer your specific question, if you are caught with 10l of beer (with somewhere between 2.5 and 7% alcohol) and don't bring any other alcoholic beverages, you are allowed to bring 5l and have exceeded that amount by 5l. According to the law, if you accept the "forenklet forelegg", you will be fined 400 NOK, 5l of beer will be confiscated and you may keep the 5l of the allowed amount. In reality, it is not unlikely that you (at least for such a small violation) will only get an oral reprimand and be waved through.