There is nothing in the EU regulation, wich requires airlines to make sure that passengers who have booked together also can travel together. There is also in no EU country any law or regulation requiring a 17 year old to be under uninterrupted supervision of a parent or legal guardian. Many of the larger European airlines allow children of age 12 or older to fly alone without supervision.
When it comes to deciding who is allowed to fly and who has to stay back in case of an overbooked flight, decisions must often be made within minutes immediately before take off and it is not unlikely or unreasonable that airline staff will make this decision without having time for any discussions with the involved passengers. There may also be other relevant factors not immediately obvious to uninvolved 'witnesses', e.g. handling of the involved passenger's checked luggage, which makes it easier for the airline to let (or require) one specific passenger to fly instead of another.
What should one do in such a situation?
Regardless of the passengers age, such a 'forced' split of a group travelling together is of course inconvenient in any situation, but is not a situation, in which any special considerations are required from the airline.
The options available are really as suggested by the airline. The passenger could fly with the ticket he has booked, and with which he is allowed to board, or he can voluntarily stay, but will then of course not be entitled to any compensation from the airline.
If the involved passengers had any kind of travel insurance, it is not impossible that it may offer some kind of compensation in a situation like this if the entire 'group' decides to stay home if a part of the group is prevented from travelling, but I would even find that unlikely.