You are a British Overseas Citizen (BOC) travelling on a BOC passport and you had difficulty clearing UK immigration controls. Ultimately you were issued leave to enter for 6 weeks. Based upon what you wrote, you applied for ILR and were refused and your refusal was accompanied by a removal notice which gave you the option of leaving voluntarily.
When the IO scanned your passport at the primary control point (PCP) the system came back with a stop indicator. It's a simple yes/no flag and a code; no other information is available to the IO at the control point. They have to go to a secured control point to get the details and that's why the IO was unable to answer your question "Why?"... She didn't know the details, all she knew at that point is that you were a flight risk.
This type of situation is normal for someone who has been previously removed from the UK. In your case they ultimately determined that the chances of your absconding were below the threshold for an immediate removal from port.
For your specific questions...
Just because I applied for the ILR?
Yes, probably. As I mentioned a failed ILR application leaves the Home Office with the impression that you will settle in the UK (lawfully or unlawfully) if you have the chance.
Where are my refused entry records logged?
You were not refused entry, but your Home Office transcript would be amended to show a summary of this incident and how it was resolved. If you want to see the unclassified portion of your Home Office transcript you can submit a Subject Access Request (SAR) and they will send it to you. Since you are outside the UK there are extra data privacy issues to resolve, but they should issue the information within 3 - 6 months. It costs 10 pounds. All the 'juicy stuff' is usually restricted but at least you can get a general idea.
Are they stored in the chip of my BOC passport?
No. The chip contains coded biometrics and vanilla id info only.
Will I be refused entry on my next visit?
There's no way of knowing. The fact that you were given LTE may be helpful to your case, but it would not remove the stop indicator and they still have the option to remove you from port. It depends upon the conditions of your next arrival and the decision is down to the IO (and the Chief on duty at the time). The law says that their decision is final. See General provisions regarding entry clearance, leave to enter or remain in the United Kingdom (paragraphs 7 to 39c).
What should I do? How do I know they still continue to refuse my entry or not?
You have two options...
- Show up at a British port and seek leave to enter under Paragraph 16
of the rules (linked above). This is what happened to you the last
time. As explained, they can land you or remove you.
- Apply for entry clearance before you leave your country. This tells
you for sure what will happen (but it does not remove the stop
indicator until you have performed successfully with an entry
clearance). When you have an entry clearance, the IO uses a
different procedure which is much more relaxed and approaches a
formality. Having an entry clearance prevents the stress of removal from port, but it's not a requirement for most BOC's.
If either option is refused, they will give you some paperwork explaining the rationale for their decision.
If you were to go to an immigration lawyer, they would tell you that any adverse decision from the Home Office should be followed up with an SAR and 'option 2'.
Adding...
Your Home Office transcript is available to the authorities of any country signed up to the Five Eyes agreement or any Schengen member.
And you added...
How does the system work?
A secured database with preset queries. Overall it doesn't matter, but the specs are out there and you can take this question to the appropriate StackOverflow site.
Does it store and 'flag' my name + date of birth or my BOC Passport number?
It has your passport number, but they don't care about it. Also your name and dob are incidental. It matches on biometrics.
It is extremely unpleasant and tense to go through that kind of situation, maybe I should not use my BOC Passport anymore.
You got advice on how to reduce the stress of clearing UK immigration controls; please see above...
Update 2 Jan 2016
The OP returned and posted scans of the stamps in his passport. The OP was not refused entry in the way he described in this question. Instead he was given a coded landing and a 6 week "Leave to Enter" (curtailed from the ordinary 6 months LTE).
Despite the misuse of terminology, this answer remains correct.