Yes, Serbian border officials will look at your passport and if they find a stamp from a border checkpoint between Kosovo and Albania / Montenegro / North Macedonia, they can refuse your entry into Serbia. The stamp can be either an entry stamp from Kosovo, or it can be an exit stamp from Albania / Montenegro / North Macedonia. Every passport stamp anywhere in the world displays the name of the border checkpoint (and sometimes also a mode of transport: road, rail, air, sea...). Serbian officials of course know the names of all Kosovo border checkpoints. They will see that you entered Kosovo from another country, which is a violation of a Serbian law.
I could go on and explain in great details things such as Balkan rule-bending mentality, probability of passing through a border checkpoint without getting any stamps to your passport, countries that do not stamp passports at all and instead use only computer databases to record arrivals and departures of foreigners, but it's not necessary.
My advice: leave your passport at home and take your plastic ID card only. Problem solved.