Japan is an extremely safe and largely cash-based society. As I remember, if you use a cash machine the smallest amount you can withdrawal is the equivalent of 200 USD. Also, there is virtually no crime. So it is quite normal to walk around with a lot of money in cash.
Second, it is no problem to pay for a small thing with a giant bill. In some countries if you pull out a £50 note or a 100 USD bill to pay for a magazine, there is no end of scrutiny, as though you are doing the cashier a personal slight. In Japan, it's perfectly fine, except at some vending machines. So make sure you get large bills when you change your currency---to reduce the physical bulk in your wallet!
Personally, I would not leave this kind of money unattended in a hostel. That is simply asking for trouble. There will be no disadvantage in keeping it with you, except I suppose the possibility of losing it.
If you do keep it on your person, use the typical precaution of breaking it into bundles and keeping most of it somewhere else on your person or in your bags where it won't be discovered every time you pay for something.
All of this said, even though Japan is largely cash-based, I think it is more sensible to arrange with your bank a card with a 0% forex rate, pay with card when you can and make cash withdrawals as you need them.