Do immigration officers care when leaving to a country you are obviously not allowed to visit?
Typically no, but that's probably not universal and they are likely some exceptions to the rule.
Any exception would be likely at a land border since enforcement at ports or airports are handled by the airlines or cruise ship lines and most governments put hefty fines in place to make sure they do a thorough (but not perfect) job at it
Consider a land border from A to B. Exceptions could be triggered by cases where any problem entering B would also affect the country A. Let's say you have a single entry visa for A and no Visa for B. At exit immigration, the officer country A would check that you were legally in the country have not overstayed AND they would invalidate your single entry Visa.
If you now move on to country B, they won't let you in since you don't have a Visa. Typically they would just sent you back to A, but since your single-entry Visa has been invalidated you can't go back there either. So you are stuck between A and B and someone has to deal with you.
In many cases, the exiting country has a process for this, where you would get an emergency transit Visa or some such, but that's a lot of extra hassle and paperwork and so it's a lot easier for country A if the exit immigration officer checks your credentials for entry into B BEFORE they invalidate your single entry Visa.
Another example: Exit immigration in Hongkong does a fairly thorough check of your papers at any land border to mainland China. I'm pretty sure they would refuse you exit if there is a problem with your entry credentials into China. Granted, I have never tried, and I'll make darn sure I never will!
Overall there are 186 options for country "A" and for country "B", 200 or so possibilities for your citizenship plus a many different visa options and all of these are governed by a different set of laws and rules. These are tens of millions of combinations and making any general statements is always likely to be incomplete