I will be spending four weeks in Japan, visiting all of the "main islands" except for Hokkaido. Some of the activities would be hiking across the woods and mountains as well.
The vaccine is for "Japanese Brain Fever" or Japanese Encephalitis.
The short answer is no, you don't need it.
The long answer is that all Japanese have been vaccinated for it since the 1960s and statistical incidence is down to background noise levels: this study figures less than one case per year since the 1990s. So unless you're planning on spending a lot of time in rural pig farms, it's not worth worrying about.
The CDC is, as ever, more cautious, but even they state that "vaccine not routinely recommended for travel limited to Tokyo or other major cities". The Australian Dep't of Health goes a bit further, stating that "the risk of travellers in Asia acquiring JE is extremely low" and only recommends vaccination for "travellers spending 1 month or more in rural areas of Asia" (unless there are significant other risk factors).
All that said, the vaccine is reasonably effective and provides lifetime protection. So if money and needles aren't a concern and (probably more importantly) you plan on traveling in places where the disease is more common, like Vietnam or Indonesia, you might as well go for it.
The French governmental website "advices to travellers" says about Japan (my translation):
For long stays in rural zone (and at some seasons), vaccinations against hepatitis A, B, typhoid fever or japanese encephalitis can be advised.
Just thought I'd give a U.K. perspective.
You almost certainly don't need the vaccine, particularly if you're visiting soon - the transmission season is June - September, except in Okinawa / Ryukyu islands where it's April - December.
I was advised by the gap year charity I went to Japan with to get these vaccines, but my doctor informed me that they were unnecessary (for Tokyo).
More specifically from the U.K. Government on who is at risk:
JAPANESE ENCEPHALITIS: ... Short-term travellers and those who restrict their visits to urban areas are at very low risk. Those at higher risk are travellers who visit or work in rural agricultural areas such as rice fields and marshland. Long-term travellers and expatriates are also at higher risk.
and on risk management procedures
Travellers should take mosquito bite avoidance measures. Culex mosquitoes feed predominantly during the hours from dusk to dawn. Vaccination should be given to travellers whose planned activities put them at higher risk (see above). There are specific contraindications and adverse events associated with JE vaccine. A careful risk assessment should be made before administration and specialist advice sought as appropriate.
From The National Travel Health Network and Centre, part of the U.K Health Protection Agency.