I'm wondering if it's possible for passport holders of countries that do not recognise other countries to travel to that particular country. Israel is not recognised by the Malaysian passport, so would it be possible for a Malaysian (passport holder) to travel to Israel and just not have their passport stamp, or would Israel refuse the bearer entry/visa?
Israel allows Citizens of all countries to enter its borders, as long as they have an appropriate visa. There are countries for whose citizens Israel doesn't require a visa for a tourist stay of no more than 90 days. Malaysia and the other countries that don't recognize Israel and don't have diplomatic relations with Israel are not on that list so a Malaysian citizen would need to acquire a visa prior to their visit to Israel.
There is no Israeli representation in Malaysia, so a visa request would need to be filed at a representative at another country. Israel does have an embassy in Singapore. Or one might search the Israeli Foreign Affair Ministry site for more information.
Also it should be noted that while Israel may allow you to enter its borders, Malaysia has restrictions on visiting Israel. As noted in answers to other questions on this site, Israel allows for border control to not stamp a visitor's passport if they ask. This permits the the passport to have no record of the visit to Israel.