In this I can only give my own experience but I think it is telling enough.
I have traveled in Japan for most of a month in 1994, with no Japanese at all and traveling by train and hostels, using English as my native Dutch is not understood there at all.
I had arranged my first week with a short tour Tokyo/Kyoto, but did the rest without any reservations made from home.
It was only when going off of the tourist routes that I met one hostel where the people picking up the phone did not understand enough English to get a reservation, a tourist information office made that call and reservation for me.
Upon arriving at the hostel it turned out the landlady spoke 6 words of English:
Yes, No, Bed, Bath, Dinner and Toilet. But she used those words very efficiently and when she had to convey more involved concepts she asked one of the other guests to translate. There was one other guest each day I was there, so limited choice, but the stay was pleasant and successful.
And that was the worst language situation I met.
I have not been in any shop, restaurant, museum, station booking office, train or underground station where there was not enough English with the staff and in the signs for me to be able to find my way and get my things.
My Japanese is restricted to 'Arigato' (not even sure of spelling) which I am told is the shortest way to say thanks and still be polite.
I am sure English is more widely spoken by now, with so many more years of young Japanese people learning it in school and so many more years of tourists coming to see the country.
In Tokyo it was not a problem back then, not at all, although you did find quite a few people who did not speak English and a lot of the rest was not sure about their ability to use English and might have avoided using it rather than losing face using it poorly. By now many of those people will be used to using their English much more. There will still be people who do not speak English, but not enough to stop you from finding your way around.
Added after reading the answer by the person living in Tokyo:
My experience was a tourist, doing touristy things, getting food in restaurants with plastic models of food in the few cases I did not get an English language menu and going to shops where tourists were common. If you need to communicate with people who do not interact with tourists often I appreciate you will need more Japanese.