Unless you have leave to remain in the UK, a visa doesn't give you the right to enter the UK by itself. It's certainly a big help. However, the Border Force maintains the right to question your circumstances to ensure that your visa was granted on legitimate grounds, and that your purposes for being in the UK are compatible with what the visa permits.
As a non-citizen, no, you're not legally obliged to answer their questions, but if you refuse to answer their questions, you run the risk that you won't be permitted to enter the country.
The family connection matters because, often, family members will work for businesses owned by their other family members. This is alright if your visa permits it but it increases the risk of a person staying longer in a country than they're legally allowed. Also, of course, some visas don't permit any employment at all.
If your long-term plans involve staying in the UK, and you're eligible, you might consider getting citizenship, which will give you a right of entry. Otherwise, and (regardless) until then, you need to be patient with them and answer their questions patiently and honestly. Impatience is not your friend when it comes to border inspection. Indulge them, and vent your frustrations out of ear- and eye-shot of them, once you are admitted back into the UK.
when they get pissy when I tell them that I don't rememberThey would, why wouldn't they. You are a visa national and they would want to know something about you. The more you don't answer, more suspicious they would get because then they assume you are evading questions which in turn means you have something to hide. Answer and move on. The more you try to take them on their game, trouble will come your way.