Is there any flight search website that allows to exclude results with layovers in certain countries?
In ITA Matrix there is an advanced routing code for filtering entire countries. Just enter
~l:nUS+ in the routing codes section to avoid layovers in USA for example.
These are the results without any routing code.
These are the results when specifying no layovers in USA.
The only site I know with remotely close functionality is ITA Matrix (http://matrix.itasoftware.com). There, you can use "Advanced routing codes" to specify certain airports (though not countries) to not transit through.
Let's say you're going from San Francisco to Delhi, but for whatever reason, would like to avoid China, Japan, and Korea. Off the top of my head, I know that the major hubs in China, Japan, and Korea are ICN (Seoul Incheon), NRT (Tokyo Narita), PEK (Beijing), PVG (Shanghai), HKG (Hong Kong). We can use advanced routing codes to exclude these:
The results predictably go through Europe rather than Asia (e.g., through Munich, Istanbul):
If you missed any airports the first go-around, you can go back and add them to the search filters. Once you have a result that you'd like to book, copy and paste it into BookWithMatrix to book the ITA Matrix itinerary.
Google flights gives you a list of transfer airports it has found and lets you include/exclude them from your search.
Unfortuately it doesn't tell you what countries those airports are in or offer a quick way to (de)-select all the airports in a given country. So excluding all the options in a country like the US may take a fair bit of effort.
I don't know about excluding particular layover countries. However, some sites will let you pick only certain airlines. I believe these option will let you filter out some countries.
For example, a flight search on https://www.travelocity.com/ will let you do the basic search (from, to, and dates) and then let you restrict the search to specific airlines; whether the routing is direct, 1 stop or 2+ stops; and whether the depart/arrive times are morning, afternoon or evening.