I live in Tokyo (seven years now) and just started commuting to work by bicycle.
Mostly safe. But some drivers (cars, motorbikes, vans, large vehicles, construction vehicles, buses - can be anyone) think its perfectly fine to whiz right past you with cms of gap. Some drivers also think its OK to pull-out in front of you (even though you made eye contact) or overtake and turn left or pull out in front of you turning right whilst looking left causing near crashes but usually just heavy braking on your part. Other drivers will honk at you if you are turning right e.g. in the middle of the lane rather than against the curb on the left.
Residential roads are the most dangerous.
Many pedestrians are also unfriendly generally, will not move to give you space (those group of three hogging the entire width of the pavement for instance) or will walk into you because they are staring down into their mobile phones. Also, the few pedestrians who jay-walk although may wait for vehicles to pass before crossing, will pass in front of you even if your going full-pelt and they performed their green-cross-code and saw you coming.
Cycling very early in the morning is the safest and most fun because there are far less traffic and pedestrians on the roads.