There are Ginkgo trees in a lot of areas of Japan. So it depends a lot on where you want to go otherwise. Since the other answers focus on Tokyo I will just throw some answers for the Kansai area.
There are many Ginkgo trees lining Midosuji which is the central street going from Namba to Umeda which is a nice stroll.
There are also Ginkgo trees as well as many other tree's such as maple in the Osaka castle garden which makes for a nice view.
There are many places in the central town and around temples. Such as kiyomizu-dera.
My favorite place in Kyoto for Autumn colors is Arashiyama. You can also take the Torokko which is a train that goes through the valley where you can see a lot of foliage. Then you can also take a boat ride on the river to get back to Arashiyama
Nara park contains a lot of Ginkgo trees as well.