To answer my question myself after my experience from two subsequent visits:
Is this better than applying for entry clearance ahead of time,
bearing in mind obtaining entry clearance ahead of time does not mean
I will automatically be allowed entry into the U.K.
March 15 2017 Visit
For any visa free national with a previous dust-up with UK Immigration, I will not recommend you try the visa free route unless you have an appetite for uncertainty. I just got released from detention at Heathrow after being held for Six hours. The whole ordeal was a nightmare which was one missed phone call away from ending disastrously.
Ultimately I chose to fly in directly to the UK from the USA, abandoning the entry via Calais in France. I landed at 7:30 AM and after a couple of questions from the clearing officer (including have you ever had any problems with immigration anywhere which I answered honestly with a yes including recounting my first refusal 19 years ago), I was promptly perp walked to a cordoned off area in the immigration control hall in full view of arriving passengers.
After stewing for about twenty minutes, the I/O came over, asked me a couple more questions and took the number of my friend I am visiting. Unfortunately my friend was at work and did not pick up the call after two tries (I believe this was a major cause of my ordeal). I was marched over to the fingerprinting room and my mugshot and fingerprints taken, then returned to the cordoned off area. I sat there about another hour before another I/O came for me and escorted me to the detention room, confiscated all my electronic equipment and belongings etc and locked me up.
I remained there for another two hours before two different I/O's came for me, asked me to get my bags and went through the bags in detail and inventoried everything. Back again to the detention room for another hour, then one of the I/O's came for me to an interview room.
I was interviewed for about thirty minutes with all kinds of probing questions. Fortunately I had a printout of all documents required for a visa interview and remembered the advice here to be exceedingly polite. After the interview the I/O said he was going to his Chief Immigration Officer to make a final decision. I asked him how is it looking and he said he couldn't give me an answer, then relented a bit and said don't worry too much, I will be back with the decision. I was returned to the detention center and I started thinking about how to request temporary admission.
After another thirty minutes he came back and said, I have good news for you. We finally reached your host/friend and your answers all tallied with his except a couple minor discrepancies. We have thus decided to land you. I thanked him for his professionalism throughout the ordeal (the second I/O had been yelling at an Iraqi girl who was also in detention for not having a transit visa while enroute to Canada, making her burst into tears, she had been held from the previous day and was going to be there another day or more while the storm in the East USA cleared) and was on my way.
Attempting a visa free entry is asking for unnecessary trouble and rolling dice stacked against you. Heed Gayot Fow's advise from the answer below and get entry clearance before traveling.
DOCUMENTS I CARRIED ALONG
I had the original visa refusal letter from 2015, three months of current bank statements from each of my three checking accounts plus brokerage account , three months of mortgage statements, signed invitation letter, marriage certificate, return ticket, CASH plus credit cards (they did say they prefer debit card to credit card because it's access to your own money and not a loan unlike a credit card), previous international hotel stays spanning the time from my previous refusal to now.
The detention centre was not pleasant although it was clean and big enough. There was a television and books We were not segregated by sex, so we had both males and females in there.One man was sleeping and snoring like a locomotive and disturbing everyone and he clearly had not taken a bath in a few days. I confess I could be viewed as another crazy since I couldn't sit still and was milling around trying to do tai chi, shadow boxing, and talking to myself etc to keep my mind off my predicament. The female guard did comment later that I talk to myself a lot.
Another woman was on the only allowed public phone for about three straight hours preventing other detainees(including myself) from receiving vital incoming phone calls. There were two mattresses on the floor I guess for long term detainees. You get a drink and a bun and you're told you could be held there anywhere from 15 minutes to 24 hours without a specific time. Basically, you don't want to be there
UPDATE 2 - July 8 2017 Visit
Followup visit after first visit in March. Today at Belfast airport I was detained for about twenty minutes while the border officer left her terminal at the primary control point (PCP) and went to review my profile/files at the secure control point.
My trip today was once again as a non-visa national without obtaining advance entry clearance. I was asked the standard what are you visiting for, how long are you staying, and then have you ever had any problem with UK immigration after which she took my passport and issued me with the form IS 81, asked me to take a seat, and went off into their offices.
After the officer returned, she apologized for holding me up and explained it was because of my entry clearance refusal back in 2015. Without request, she went on to say she would request the stop indicator flag on my profile be removed. There was no interview except the initial questions and they also did not ask for any documents.
My experience makes me believe unless I somehow get the stop flag removed, subsequent visits will be always involve relatively short detentions while they check their system.
UPDATE 3 - July 3 2018 Visit
Experience was like second visit in Belfast. Detained for about fifteen minutes. Officer went to check record at secondary control point, came back and let me go after an apology and said he had also recommended Home Office remove the stop flag on my name however I am not counting my chicks on that.
UPDATE - March 3 2019 Visit
I was just admitted into the UK for the first time today after my complaint to UK Border Force last year. I spent the standard one minute on my landing interview at immigration control like most other travelers. This confirms that the flag on my profile has indeed been removed and eliminates any future mal-effects stemming from my original entry clearance refusal.
UPDATE - November 2019 Visit
Successfully used e-gate after it was expanded to American citizens and citizens of other Five Eyes countries.