It's not strictly forbidden but could be tricky. The visa is valid and you should not be automatically bounced just because it was issued by Norway.
But it is also a single-entry visa, which implies that you are not a regular/trusted traveller and that it was issued with a specific trip in mind. When you applied for that visa, you must have provided the Norwegian consulate with details of a trip which cannot have been a 4-day stopover on the way to the US (for they would have directed you to the German consulate if that was the case).
The German border guards will know this precisely because the visa was issued by Norway and are likely to quiz you about it and generally examine your situation more carefully than usual. If they are not satisfied that you still fulfil the conditions and had a good reason to change your plans or worse think that you lied to get the visa in the first place, they will refuse entry and possibly revoke or annul the visa.
Additionally, anecdotal evidence suggest Germany is strict about this and other users have run into problems there.