The short answer is that I would consider this to be very unlikely.
Let's start by taking a look at what the machine readable zone actually contains. From wikipedia, the first line contains:
| Positions |
Length |
Characters |
Meaning |
| 1 |
1 |
alpha |
P, indicating a passport |
| 2 |
1 |
alpha + < |
Type (for countries that distinguish between different types of passports) |
| 3–5 |
3 |
alpha + < |
Issuing country or organization (ISO 3166-1 alpha-3 code with modifications) |
| 6–44 |
39 |
alpha + < |
Surname, followed by two filler characters, followed by given names. Given names are separated by single filler characters |
and the second line contains:
| Positions |
Length |
Characters |
Meaning |
| 1–9 |
9 |
alpha+num+< |
Passport number |
| 10 |
1 |
numeric |
Check digit over digits 1–9 |
| 11–13 |
3 |
alpha+< |
Nationality (ISO 3166-1 alpha-3 code with modifications) |
| 14–19 |
6 |
numeric |
Date of birth (YYMMDD) |
| 20 |
1 |
numeric |
Check digit over digits 14–19 |
| 21 |
1 |
alpha+< |
Sex (M, F or < for male, female or unspecified) |
| 22–27 |
6 |
numeric |
Expiration date of passport (YYMMDD) |
| 28 |
1 |
numeric |
Check digit over digits 22–27 |
| 29–42 |
14 |
alpha+num+< |
Personal number (may be used by the issuing country as it desires) |
| 43 |
1 |
numeric+< |
Check digit over digits 29–42 (may be < if all characters are <) |
| 44 |
1 |
numeric |
Check digit over digits 1–10, 14–20, and 22–43 |
You can see that more or less everything in the MRZ is fixed. The only free field is the "personal number".
In principle, is it possible that the personal number contains some kind of code? Sure. And if it did, that's the kind of information that I wouldn't necessarily expect to be publicly stated. But I don't believe it's at all likely that there's some kind of secret code hidden here which is used to reject your passport by the machines, for one main reason:
When the machines reject you, you go and see a human. And they check your passport, and let you through. If we assume there is some code, it doesn't seem to be having any effect (I guess you'd have mentioned if you seemed to face heightened scrutiny after the machines reject you). That could be explained by
a) the agents can quickly perform some look up that shows you're not a concern
b) agents can't tell the code is there
c) there is no code
a) seems implausible, as if the lookup is so quick you can't tell it's going on it's almost certainly something that just shows the agents a yes/no answer, and the machines should be able to do it themselves
b) would mean the code is completely and utterly pointless in the first place
c) seems compatible with everything we know
Similar logic applies to your ESTA application. If the MRZ contained some kind of code suggesting you're a danger, it wouldn't make any sense for it to be described as "unreadable" and then you have no trouble getting the ESTA when you input your details manually. The MRZ would surely be read just fine, and quite likely set some internal flag to alert them to the problem so it can be considered when they decide whether or not to grant you the ESTA.
I think the above lays out a few reasons that the idea of the MRZ having some code marking your passport as banned doesn't make much sense, given your experiences. Ultimately I see no reason to believe that there's anything going on beyond what the ESTA application told you: your MRZ is unreadable. I can't give you a certain answer as to why that is, but some possibilities have been raised by various users in the comments.
A printing error seems unlikely but isn't entirely out of the question. It might be recognisable if you took a close look at your MRZ (perhaps with reference to the font).
An error in the check digits (as @phoog mentioned) also seems rather unlikely (and I would expect it to lead to an error describing it as "invalid" rather than "unreadable", but you never know), but certainly wouldn't be easily recognisable to humans.
The pattern behind the text may be causing some difficulty for the OCR process. This would presumably have been noticed if it were widespread, but the pattern varies across the MRZ - it's conceivable that a particular character becomes difficult to identify only when it occurs at some specific location in the MRZ.
@Traveller mentioned having similar experience of a passport repeatedly being rejected, but a photocopy of that passport being accepted. Even a small imperfection on the surface could in principle cause weird reflections that interfere with the OCR process but aren't easily visible when looking at it under normal lighting conditions.
