I've faced this same problem before. I spent 5 years in Europe, and traveled quite extensively. Navigating RyanAir's website can only get you so far (literally). Once I learned about other cheap airlines, I tried to figure out the best way to ENSURE I was finding the cheapest flight.
I wish I could tell you "use Kayak" or something, but as far as I know, there is no catch-all tool. The best tip I could give you is this:
Choose your destination, look up all the airports within an acceptable distance, and navigate the airport's website for what airlines it serves. From there, you can start down the path of identifying those airlines that best suit your needs, and your pocket book. Remember, a lot of destinations have multiple airports, and most cheap airlines serve the "b" and "c" airports.
For example: want to head to Berlin? Searching Berlin's 3 airports for airlines they service will lead you to AirBerlin. A large percentage of AirBerlin flights are very comparable in price to RyanAir, and they flight to a ton of destinations. If you are like me, you've heard of RyanAir, but not of AirBerlin, so mission accomplished.
Another tip: When you are on an airport's website, look for transfer busses that get you into the city on the cheap. For instance, flying RyanAir into Frankfurt actually lands you about an hour's drive from Frankfurt. There is a bus that will take you into Frankfurt proper for just a few Euro.