It really depends where you want to go. I have been to Japan seven times and I have only needed a car once.
In urban areas, like Kansai (Osaka and nearby cities like Kyoto, Kobe, Nara, Himeji) or Tokyo, public transportation is quite convenient. And navigating within an urban area (for example, from Osaka to Kyoto) is not expensive (there are actually various trains with different fares depending on how fast they go).
In rural areas, like Okinawa, a car is very useful. There are many beaches and places that cannot be easily reached by public transportation. Even going from Naha (the regional prefecture) to the famous Churaumi aquarium would take a lot of time without a car or booking a tour. I haven't been there yet, but I guess that the situation would be similar in Hokkaido, though it seems the train network is more developed there.
One element you should factor in your decision: parking lots. You should check if your accommodation has a parking lot, as it is usually not possible to park in the (narrow) streets and parking lots can be quite expensive.
I have no experience with going from one big city to another one (for example, Tokyo to Osaka) but, according to @lambshaanxy, it is quite expensive. Shinkansen is not cheap though.
but also that apart from major highways, it might be difficult for foreigners to navigate the Japanese back-country.
From my experience in Okinawa, navigation by GPS is really pleasant in Japan. You can find a place by using its phone number or using its Mapcode, a series of numbers that identifies it. No need to type an address using any Japanese alphabet! And most tourist maps (such as the ones you can find in the airports) have the mapcodes of touristic areas.
In addition to GPS if you want to go outside cities, you may want to rent a pocket wifi at the airport. Most of them are cheap and comes with unlimited data, a charger that you may use with other devices,...