You are allowed to make minor adjustments to your itinerary after you get the visa.
You are not allowed to misrepresent your itinerary to apply for your visa at a different main destination (a practice sometimes called "visa shopping").
In all likelihood, there will be no questions. If there are, it would be useful if you could demonstrate that the change became necessary after you got your visa, and that changes were really minor. Were France and Switzerland on your original itinerary? Then you are just changing the date of an internal border crossing by a few days.