I had two tickets one from Mumbai to Amsterdam via Istanbul on Turkish airlines and Amsterdam to Quito on KLM airlines. Turkish airlines ground staff cleared me at Mumbai after verification of all documents and issued boarding passes. At Istanbul, I was not allowed to board the flight to Amsterdam since I did not have Schengen visa. Indian passport holders don't need a transit visa at Amsterdam. Why did Turkish Airlines ground staff at Istanbul take such a decision?
You have made significant changes to your question since it was originally posted. In light of that new information:
Without checked baggage, you could have transited airside in Amsterdam. However, Turkish Airlines staff in Istanbul would have no way to know that you were planning to transit AMS and go elsewhere, unless you explicitly informed them. Since you say you did inform the staff of your other flight, I think this was a mistake and you should not have been denied boarding.
Based on the information you have provided, you should NOT have been denied boarding of your flight. You've stated that :
You had no plans to exit the airside part of the airport in Amsterdam, and as you were traveling with carry-on bags only there would have been no need to do so.
You had and showed the flight information for your connecting flight to the TK staff in Istanbul.
Your connecting flight departed the next day, but this was only ~16 hours later.
Additional facts that you haven't explicitly mentioned is that there is only one flight from AMS to Quito per day, being the one that you were booked on.
The Netherlands allows an Indian citizen to Transit Without Visa (TWOV) in the conditions that you've described above.
Timatic, which is generally the source that airlines use for visa requirements, states that :
TWOV (Transit Without Visa): Visa required, except for Holders of onward tickets transiting by the same or first connecting aircraft:
As you were transiting on the first connecting aircraft, you meet the exception and thus do not require a visa.
I suspect that the date difference confused the staff - the fact you were arriving into Amsterdam on one day and not departing until the next day is not allowed in some airport - however in Amsterdam is it, and there are even 2 airside hotels that you could have spent the night at without entering the country.