The answer appears to be a combination of unrealistic scheduling and a flurry of mergers.
A Travel Weekly article from 2003 (the record low year at that time) gives some clues:
The 2003 decline is even more dramatic when compared with the peak year, 1987, when the DOT received 44,845. That was a year in which on-time problems were rampant, and the DOT determined many carriers were publishing unrealistic schedules. Airline mergers also had caused consumer problems that year, a DOT spokesman said.
As an aside, it is worth noting that US air traffic in 1987 was only about half of what it is today, meaning that 1987 had a particularly high "per capita" complaint rate.
Concerning details on the unrealistic scheduling, this GAO report from 1990 provides details:
DOT’s investigation of airline scheduling practices during 1986 and 1987 resulted in the on-time reporting requirement. At that time, DOT found that airlines often scheduled unrealistic flying times because the computerized reservation systems used by travel agents gave priority listings to flights with the shortest elapsed time. Travel agents usually book passengers on one of the flights that appear on the first few display screens of a reservation system. Flights with longer scheduled elapsed times would appear on subsequent screens and agents would be less likely to book these flights. DOT officials concluded that unrealistic scheduling was an unfair and deceptive trade practice. In August 1987, they obtained a commitment from the computerized reservation system vendors to stop listing flights in the order of scheduled elapsed times. This eliminated an incentive for airlines to underestimate flight times.
In September 1987, DOT began recording the on-time performance of U.S. airlines to provide consumers with information on airline flight timeliness.
Later in the report, the GAO report notes that "on-time performance was lowest in December 1987, when 66 percent of flights were on time." For comparison, the on-time performance in December 2019 was 78%.
It may also be worth noting that the airlines had moved to a hub-and-spoke system over the 5–10 years prior as a result of airline deregulation, rather than the older point-to-point system. I would speculate that this caused more travelers to end up taking connecting flights, and these travelers were more exposed to major hassles when flights were delayed or luggage was mishandled.
So many mergers
For further details on the mergers, we can look at this page, which notes that the major carriers acquired ten (!) smaller airlines in about a one-year period around 1987:
- Northwest merged with Republic Airlines in October 1986
- TWA merged with Ozark Air in October 1986
- Delta merged with Western Airlines in December 1986
- Alaska merged with Horizon Air in December 1986, and with Jet America in October 1987
- Continental merged with People Express, New York Air, and Frontier Airlines in February 1987
- American merged with AirCal in May 1987
- US Air merged with Pacific Southwest Airlines in October 1987
The logistical challenges of merging airlines often lead to a higher level of mishaps and delays.
Note that "Frontier Airlines" above does not refer to the current airline, but an earlier airline of the same name.