It might be worth breaking the trip into two or more separate trips and searching for them separately; keeping the main outgoing and return flights together (plus all the connections), but looking at the travel within the trip separately.
For example, when I'm travelling from Seattle in USA to Europe to catch up with family and also do my own travel, I usually book an 'open-jaw' for the two end-points of the trip with the connections: so for example, Seattle-Brussels via Amsterdam, and returning Dublin-Seattle via Atlanta.
Then, for the travel within Europe during the trip - Brussels to Dublin in this case - I may use some combination of trains, or flights booked directly on budget airline websites (Ryanair, EasyJet, etc.). These budgets lines usually won't show up on any of the usual flight search engines (Expedia, Orbitz, etc), and usually have to be booked directly. Also, if you haven't used a budget airline before, be sure to read up on the pros and cons before using one of them. Sometimes the airports are out-of-the-way and may not have good transit connections, and they may require you to print out a boarding pass in advance (or charge you extra at the desk!) or charge you more for checked baggage; and they often won't check baggage through to another flight, so are sometimes only useful for single point-to-point legs on a larger itinerary.
I did something similar in Japan a while back, using one of the major sites to book the flight to/from the country, then I found a budget carrier within Japan to do a Tokyo/Sapporo flight; and used a railpass to work my way back to Tokyo and beyond. None of the major USA search engines knew about the internal airline that I used and only knew about other major carriers that were several times more expensive.
(If using this approach, though, make sure you have at least a day between any two separately booked trips; never try doing two separate bookings back-to-back, because the parties involved won't recognize the separate bookings as related, so won't check luggage through from one to another, and won't reschedule flights on the later one if you're delayed on an earlier one.)