Flights are routinely overbooked, but more commonly in coach than business (or first). When airlines find that more confirmed and ticketed passengers have shown up than they have seats for, they will try to "roll the cabins" to make seats available (move people from business to first to free seats in business, move people from premium economy to business to free seats in premium economy, etc.) As a last resort, they will downgrade people or "bump" them (deny boarding). In general, but especially for major airlines such as BA, they have a defined order in which to downgrade or bump passengers (just as they have a defined order for upgrading people to free a seat in a lower cabin). They will generally start with those who are not members or have no status in their frequent-flyer program (Executive Club) or those of partners (OneWorld), and among them, start with those who checked in last.
In your case, I suspect that the reason you couldn't check in online is because your cabin was oversold and they couldn't assign you a seat (rather than the other way around, as you suggest in your question).
Did BA compensate you for being downgraded? They should have offered to fly you on an alternative flight in business or first, or on an alternate carrier. In addition, they should have offered you a refund for the fare difference between the classes (in either cash or points, whichever was used to pay for your ticket) and thrown in a bunch of points for the inconvenience.