Assuming that you really are from one of the few countries from which a biometric passport is required for visa-free entry to Lithuania or the Schengen area and that the refused entry was not a mistake by the immigration officer:
The Ryanair document check before boarding is not primarily there to avoid your inconvenience, but for Ryanair to avoid all the hassle they have being required to handle your case if you are denied entry. Ryanair's terms and conditions are similar to most airlines:
Article 13 - Administrative formalities
13.1 General
13.1.1 You are responsible for obtaining all required travel documents and visas and for complying with all laws, regulations, orders, demands and travel requirements of countries to be flown from, into or through which you transit.
13.1.2 We shall not be liable for the consequences to any Passenger resulting from his or her failure to obtain such documents or visas or to comply with such laws, regulations, orders, demands, requirements, rules or instructions.
13.2 Travel documents
Prior to travel, you must present all exit, entry, health and other documents required by law, regulation, order, demand or other requirement of the countries concerned and permit us to take and retain copies thereof. We reserve the right to refuse carriage if you have not complied with these and such other requirements as are set out in our Regulations, or your travel documents do not appear to be in order. (click here for Regulations concerning Travel Documentation).
13.3 Refusal of entry
If you are refused entry into any country due to immigration inadmissibility, you will be required to pay any fines levied against us by the Government or immigration authority concerned, plus the cost of transporting you from that country back to your point of origin or elsewhere. We will not refund any flight you are unable to use as a result of your refused entry. Any costs incurred by us on your behalf will be levied via deduction from the credit/debit card used to make the booking.
13.4 Passenger responsible for fines, detention costs, etc.
If we are required to pay or deposit any fine or penalty or incur any expenditure by reason of your failure to comply with laws, regulations, orders, demands or other travel requirements of the countries concerned, you shall reimburse us on demand, any amount paid or expenditure incurred. We may apply towards such payment or expenditure the value of any unused flights, or any of your funds in our possession e.g. gift vouchers, credit vouchers, etc. Any costs incurred by us on your behalf will be levied via deduction from the credit/debit card used to make the booking.
Ryanair's boarding agent could and should have recognized that your travel document doesn't entitle you to enter Lithuania, but their failure to do so does not free you from any liability.
Considering Ryanair's goodwill attitude, I would unfortunately suppose that it is one of the airlines least likely to waive any valid claims they may have against you.
Addition: Since there are a few other heavily upvoted answers here with IMHO insubstantial interpretations of the situation (paragraph 13.3 of Ryanair's T&C is allegedly violating ICAO recommendations - which by the way are only recommendations and not binding regulations, EU regulations and being an unfair contract term), furthermore Dean McGregor's heavily upvoted comment on my answer claiming that United Airlines does not forward the fine to their customers, here some more thoughts:
Even if disputed in the following comments and being simply wrong, Dean McGregors's comment has for some reason received a great number of upvotes. Contradicting the comment, United Airlines does as well in their Contract of Carriage, section 19.A forward such fines to the customer. All the larger airlines in Europe, e.g. Lufthansa, British Airways, Air France and KLM have similar regulations in their terms and conditions.
I have not found any references to disputes on this matter in UK courts, but there have been several court rulings on the subject in Germany, a country with similar consumer rights regulations as the UK (some of which even apply EU/EEA wide). A discussion on the same subject can be found on frag-einen-anwalt.de, where Lufthansa forwarded the fine for transporting an inadmissible passenger to Donetsk to the passenger. The lawyer answering the question, Holger Hopperdietzel, points out that a dispute in court can be ruled both ways, but does not sound overly optimistic:
That Lufthansa levies the fine for transporting inadmissible passenger
to you is not an isolated case. This happens now and then and I have
had several cases in my law office.
Unfortunately, there is only one court ruling in favour of the
passenger from the District Court of Aschaffenburg. During my research
so far, I have not found any other court ruling in favour of the
passenger.
The question if Lufthansa can pass the immigration fine directly to
the passenger is until now not judged upon in a higher court.
Therefore, both results can be possible, it all depends on the
argumentation.
So unless German courts regularly blatantly ignore ICAO documents, EU regulations, customer's rights and companies' unfair terms and conditions, the situation is not nearly as obvious as some of the other answers pretend to impose.
