As a dual citizen, I have an EU passport and a non-EU passport. I have to leave and enter my home country on my non-EU passport. After leaving on my non-EU passport, can I then enter the EU on my EU passport, or will there be a security glitch because of the advance passenger info provided by airlines?
There is no problem to enter the EU as long as you are able to (and you are since you are an EU citizen).
Countries like Venezuela are forcing by law its citizens to use the Venezuelan passport to enter/leave Venezuela. However this doesn't force you to enter the EU with that passport.
It could be possible that some EU countries authorities ask you some questions, then just show them the non-EU passport and everything is ok.
I voted to close this question, but retracted my close vote because I realized that there is a distinct angle in the Schengen area. The Schengen Borders Code was recently amended to require more thorough checking of travelers from the EU when they enter the Schengen area. The code explicitly provides that this checking can be done before passengers arrive by using information supplied in passenger manifests.
Therefore, if you check in for a flight to the Schengen area using your non-EU passport, and then present your EU passport at the border, the border authorities may run additional database queries for your EU passport, because they will not have been able to do that earlier. This could delay your entry by a few seconds or minutes, and could be considered a "security glitch."
This would be only an administrative delay, however, because as an EU citizen, your legal right to enter the Schengen area can be curtailed only if you are a threat to public safety or public health, or for reasons of public policy. Checking in for a flight with a non-EU passport does not implicate any of those reasons.
The current text of the Schengen Borders Code, Article 8, paragraph 2e:
2e. The checks against the databases referred to in points (a) and (b) of paragraph 2 may be carried out in advance on the basis of passenger data received in accordance with Council Directive 2004/82/EC [footnote removed] or in accordance with other Union or national law.
Where those checks are carried out in advance on the basis of such passenger data, the data received in advance shall be checked at the border crossing point against the data in the travel document. The identity and the nationality of the person concerned, as well as the authenticity and the validity of the travel document for crossing the border, shall also be verified.