I live with my wife since 2012 in Latvia. My wife and two kids are Latvian citizens. I hold a Latvian permanent residence card. On my residence card is written permanent family member. According to article 20, it should be clearly written "family member of EU citizen" but on my card is only written "family member." As my wife and kids are Latvian citizens, can I travel with them to Ireland or the UK?
The card does really need to say "family member of a union/EEA citizen" before it has any effect for the UK. Those cards are only supposed to be issued when the union citizen lives in a different member state than the one of his citizenship, which is why your card does not say that.
Even if you had one of those cards, the UK have been sort of reluctant to accept them, and the official guidance from the UK government recommends bringing separate documentation for your relation to the union citizen in addition to the article 10 card.
In principle you can just show up at the UK border and ask to be let in based on any convincing evidence that you're a family member of the union citizen you're traveling with -- such as a marriage certificate -- and they have to consider this evidence rather deny you entry immediately. But this can easily take hours and quite ruin the beginning of your holiday. And that's assuming that you even make it to the border, because without an article 10 card (or EEA family permit) you probably won't be allowed to board a plane towards the UK. So this only works if you go by train from France or Belgium (in which case delays at immigration will make you miss your train unless you plan for them and arrive hours in advance).
The sane thing for you to do is to apply for an EEA family permit (or its Irish equivalent, depending on where you're going) before leaving.
Your card is not an article 20 card, because those are issued to people whose family member lives in an EU/EFTA state other than that of their nationality.
Your wife is Latvian and lives in Latvia, so you don't qualify.
As such, you'll need an EEA Family Permit for the UK, and an equivalent visa for Ireland.
The UK is unlikely to accept this card for visa-free travel, and they are not required to under the directive. The same is probably true for Ireland. More importantly, airlines are unlikely to let you board with your permit. The safest thing for you to do is get an EEA family permit for the UK and the analogous visa for Ireland.