You require a visa to visit Switzerland. Only travel document for refugees issued by an EU member state, Norway, Iceland, Liechtenstein or the United Kingdom exempts the holder from visa requirement for short stays (<90 days within any 180-day period) in combination of a residence permit or other proof of residence.
While there is no systematic check between Germany and Switzerland, sporadic passport control (often in conjunction with customs control, which is allowed between the EU and Switzerland) does take place, especially when travelling by train or long-distance bus. Practically, you likely won't encounter too much problem, but you are still violating the law, and if caught, it may have consequences on your future visa applications or even visit to visa-exempt countries within the Schengen.