Can I mix passports when entering EU/Poland, i.e. sometimes enter on British sometimes on Belarus passport into the EU?
This should not be a problem in itself. In general, your status in a Schengen country is independent of how you entered the Schengen area. You can use whichever passport to enter, as long as your documents are sufficient to be admitted.
If you are only a visitor, for a single short stay, you should enter and leave the Schengen area with the same passport, as the passport stamps may be checked. Additionally, in your case, one passport requires visa and the other not.
However, since you have a residence permit issued by a Schengen country, it is easier to enter and exit with your residence permit and the corresponding travel document. Your residence permit with the Belarusian passport is the most straightforward way to establish your ability to enter the Schengen area.
When entering Poland, would border control be confused about Polish residence permit and British passport if presented (bc the permit is attached to the Belarus passport, at least in their database on Polish border)? Never tried it.
Yes. They may be confused and may ask you to explain the situation (e.g. they may need to update your information). As it is a Polish residence permit, the border guard in Poland has more power to examine your situation than the border guards of other Schengen countries.
When entering EU countries (not Poland, more like France) would it make sense to provide British passport or Belarus+permit? For me it feels like British is the straightforward passport for EU border control
Before Brexit, a British national would have had an essentially unlimited right to enter any EU country. But now it is considered as a third-country with a visitor visa waiver in effect. If you present only the British passport, you will be treat as a visitor and technically may be asked to present e.g. evidences justifying your purpose of stay, sufficient financial means, appropriate health insurance (though practically rare), as well as the passport validity requirement, even if in practice this is relatively rare.
while Belarus+residence may cause too much of the confusion on non-Polish borders.
Not at all. Tens of thousands of people probably do it everyday (enter a Schengen country with their residence permit issued by another Schengen country) without any problem. Your residence permit also gives you a stronger right to enter if your final destination is Poland; France or another Schengen country must allow you to enter to transit to Poland if you have a Polish national visa or residence permit except under exceptional circumstances related to public safety etc.
If you present your British passport with the residence permit, you can also be allowed to enter; there is no EU law requiring the residence permit and the travel document to state the same nationality. And often, the border guard does not check the residence permit so carefully on all the details. Nonetheless, the risk of confusion and additional hassle remains.