The Schengen agreement is entirely unrelated to customs and goods, it's only about the movement of people. The UK isn't in the Schengen area and maintains its own visa policy but as a full member of the EU, it's part of the common market, which means you can import any quantity of alcoholic beverages from other EU countries (for personal use). Conversely, Switzerland entered the Schengen area and agreed to follow the Schengen visa policy and to abolish border checks but still enforces restrictions on the movement of goods.
Strictly speaking, entering Switzerland with so much wine would therefore indeed be illegal. But Geneva airport also has a small French sector that is technically on Swiss territory but is treated as a part of France for customs and immigration purposes (part of the airport is built on what was earlier French territory but it was transferred to Switzerland at the time).
Whether you can use it depends on the flight you want to catch. You can check-in for flights to France and international flights from a few airlines but not all of them. From the official website:
Can baggage be checked in for an international flight in the French sector?
Yes, checking in is possible in the French sector for some international flights (other than France), but only those operated by Aeroflot, Alitalia, Etihad, KLM, Luxair, Tunisair, and Twinjet by making a call (telephone available on the check-in counter).
In any case, if you want to avoid Swiss duty or transit paperwork, you need to avoid entering Swiss territory and approach the airport from the French side. There was another question about that recently. If you have a French rental car, returning it in the French sector is actually to your advantage (no surcharge for crossing the border) but you need to make sure the company you got if from is present on both sides as some only have desks in the Swiss sector.
Note that there should not be any systematic ID check on the border and you can often cross it without seeing anyone but Switzerland can still do some spot checks from time to time. And if you are flying to a non-Schengen airport, Swiss customs can of course inspect your bags in the airport. You could still try your luck but I am not sure what the odds of being caught really are.
Alternatively, you could enter Switzerland and stay in Geneva, hoping not to get caught on the border (perhaps avoiding the motorway) and then leave on the last day to enter the airport from the French side. I don't think the odds are too bad but it's obviously illegal and you could be liable for a hefty fine. Beware that many rental cars often have exotic registration numbers (e.g. 60 in France or AI in Switzerland) that might invite extra scrutiny.
Finally, you might also stay for some time in Geneva and simply import the bottles officially. VAT is low in Switzerland and the duty on wine for less than 20 litres (26 bottles) is not very high either and all that can be refunded if you declare them as goods in transit. The only trouble is taking care of the paperwork.