Your passport is an official government document, and when marked for any non-official purposes (like collecting a souvenir stamp), you are technically defacing or altering it. In theory you could have it voided (or worse), or in theory, an immigration official could refuse to admit you if they believed that it was a signal the document was in any way invalid.
In practical terms, there are many places in the world that issue many thousands of unofficial stamps. I know they are popular to get at Checkpoint Charlie, various Antarctica stations, Timbuktu, Easter Island, and other iconic world locations. I got one myself at Machu Picchu, which raised no eyebrows at any of the half-dozen countries I visited subsequent to Peru. I've never heard any firsthand accounts of problems, and tend to think that if it were indeed a serious issue, the guidebooks and State Department advisories would include a mention of it.
On the other hand, that stamp was the only non-official one— and what does hold you up at immigration is a passport full of unusual stamps. The more of these meaningless stamps you collect, the longer it takes for the agent to find a blank page, and the more likely s/he'll think something is "off" and refer you to the most persnickety bureaucrat you've ever encountered.
While it's cool to have a full passport, I try not to think of mine as a souvenir. Above all, it is a working document that I need in order to travel. Some countries require entire pages to be blank before they will issue you a visa; I've had some flip through to find a blank page just to stamp me in. As a result, I'm extremely wary of filling up any more squares or pages than strictly necessary— and I only travel internationally once or twice a year.
For souvenir stamps such as those you collect at museums, temples, national parks, hostels, distilleries, and so on, I would simply invest in a small journal or passbook. These don't expire, aren't subject to the whims of any officious officials, and are entirely yours to mark up however you wish.