As an Australian citizen, you have an absolute right to live in Australia, and thus cannot "overstay" in your own country. This is regardless of what passport you used to enter: you don't stop being an Australian citizen just because you used a British passport.
Your hypothetical situation of entering and exiting on a different passport would be quite unlikely in practice, because all visitors to Australia need to get a visa (ETA), which will (should) not be granted if you are an Australian citizen.
But if you do manage to get one, stay longer than the allowed period and then try to leave, you will most likely stopped at the airport and held up until they can confirm that you are indeed an Australian citizen and thus have not overstayed. Once they do, that's it, you've committed no crime. And while you "should" use an Australian passport to enter and leave Australia, anecdotally you will be allowed to leave without one, although this will occasion considerable grumbling -- mostly because they're rightfully concerned that airlines will not let you return without one.