Places selling cannabis (“coffeshops“) aren't restricted to a particular part of town and can't be avoided entirely (to be accurate, their locations are in fact regulated through a permit system and they are required, e.g., not to be too close from a school, but they are not restricted to one specific neighbourhood). On the other hand, there is not much to see apart from a sign, sometimes a bit of music or a smell for the larger ones. From the outside, many look more like a closed disco or café than anything else. It might not help if you find the very idea disturbing but it's very easy to just move on, or in many cases, not notice them at all.
Avoiding the red light districts (especially the larger one called “De Wallen“) would therefore be the most important. The one in the Singel area in particular is a bit surprising. Whereas you can “feel” you are getting closer to De Wallen (many neon lights, seedy shops, drunken tourists…) and avoid it easily, I once stumbled upon the other one while getting out of a car park. There were a few “windows“ in quiet tree-lined streets in a picturesque neighbourhood, not far from some upscale restaurants and a conference venue, which was a bit disconcerting.
More generally, the Netherlands have become (in)famous for their liberal approach to many things but at the end of the day, all of it is limited in scope, inconspicuous or easy to avoid. Putting things into perspective, the ads or street prostitution I occasionally came across in Germany seemed more obnoxious.
