The easiest solution is depositing the funds (beyond what you'll need in the near future) in a bank account you already have, if you can find an ATM or something which supports that. The rest of this answer is only relevant if that's not possible.
If possible, open an account with a reputable large international bank that has a presence in Japan, and deposit the funds there. Even if you wind up finding a place where you have to withdraw the funds before moving on to the next country, you can breathe easier while you're in Japan.
Once you have the money electronically, see if you can get it to a credit card company (if you have one) and make it as a payment so you have a credit balance. Then you might not have as much in interest fees on "cash advances" and/or card purchases.