Is it, generally speaking, safe to carry large amounts of cash (US$4000) in Japan?
Generally speaking, yes. However the same common-sense rules apply everywhere. Don't leave it alone in a bag somewhere, don't flash it about, etc. The only time I had anything stolen was my bag, in an ex-pat bar in Roppongi. Says a lot.
More: Japan is still a rather cash-based society given how early and frequently banks close, and I found that my UK bank card would only work in post office cash machines. Basically post-5pm you'll struggle to get cash. This particularly applies to anywhere rural or villages / small towns - part of the reason for this is Japan still has a very large percentage of small / family businesses (much higher than the US / UK), and if it's anything like the UK the cost of processing card payments is far too high for small-value purchases. Of course in large chains and cities this is much less of a problem. As such, people tend to carry around larger quantities of cash than in other countries... though $4000 is quite a lot!
EDIT I should add 7/11 convenience stores seem to accept many foreign-issued cards. I live in Japan but get paid in Pounds Sterling, and usually withdraw it there. Of course any foreign transaction fees will be applied by your bank. Best to check the terms and conditions of your account.
Japan has ATMs, but couple of things to know
Citibank and Shinsei Bank will be the main 2 banks accepting foreigners ATM cards. You can find the map of ATMs on their respective website in english: http://www.shinseibank.com/english/atm/.
SevenEleven convenient stores spread al over the country will accept some foreign ATM cards
Japan post will accept Visa/Master/Plus cards
Now, in Japan Banks will shutdown ATMs during some public holidays (May and September mostly).
I would recommend taking 2 ATM cards, one always in one of your pockets, the other one in your backpack. Japan is pretty safe, in case you lose anything, report it to the Police (Koban) immediately, they usually find wallets within 1 to 2 weeks. Have a safe trip.