My simple answer is: estimate. I also offer here a more in-depth explanation in case you wonder why a gas station would put their customer through such an experience.
I imagine that the reason you've struggled with this is that the system isn't exactly optimized for your particular usage scenario.
Many people use credit cards or debit cards, which allows the equipment at the gas station to simply verify that the card is in good enough of standing (e.g., not reported as stolen), then remember the number, and then actually charge the exact right amount. Gas stations probably get a lot more people paying amounts like $20.82 or $23.16, compared to the days when more people paid cash and would try to pay $20.00 even (or $20.01, and spend a coin they find in a pocket or ash tray or something). People who paid cash would often try to hit an even dollar amount; customers with cards probably pay more on average. Especially since the next gas payment may be to a competing gas station, they want to get the money now. Supporting the use of a payment (credit or debit) card is a system that works perfectly well for very many customers these days.
Out of the people who do still pay cash, many of them probably just pay $20 or $40 or whatever, depending on the price of gas, and then they don't fill the car up. Then the car isn't absolutely filled up, but that's okay; they'll just go to another gas station a bit sooner and pay more cash there, then. So the system (of requiring pre-payment) is working out decently well for these customers as well.
As a traveler who rents a car, you have a somewhat unique need to fill the car up, because you don't want to get charged a fee for returning a car that isn't filled up.
Many of the gas station customers are not travelers. But even among travelers, you may be unique. Actually, many travelers probably use cards; even if they do carry cash also, it is because they want to keep that cash for emergency situations, and so they don't want to just dump all the cash on a purchase like gasoline. So they will use the card for that.
Out of the travelers who do prefer to pay cash for things, some of them probably just said "yes" when the clerk at the car rental asked if they want to purchase "gas protection" or whatever they call it, where instead of a super high fee for returning an empty tank, they charge a flat rate which is based on a medium-rate fee per gallon, multiplied by the number of gallons of an empty tank. So the only people in your position are the people who want to pay the lower rate of normal gas station prices, but who want to pay cash, and who are near the end of their trip (so they are trying to precisely estimate how much it will cost).
So after taking care of non-travelers who use cards, non-travelers who pay with cash, travelers who pay with cards, and travelers who don't need to precisely estimate how much will fill up a tank, we're left to people in your position. Surely this position does not represent the majority of gasoline purchases. Some gas stations might also only require pre-payment for cash after a certain time of the day (like 5pm or 7pm), further narrowing down who this even affects.
For people in your position, the "expected behavior" (which was asked about) is to simply estimate. Yes, that may mean doing a bit of math, which might be a bit more of a struggle than what most companies want to put their customers through. But, remember, you're not representing the average common case that they made the regulations around. The regulations aren't really designed around people in your situation. The regulations to demand pre-payment are probably mostly designed to reduce situations of non-payment. Maybe too many cash payers will go to McDonald's, then forget that they did, and later fill their tank without remembering that they now have less money than they expected.
What are the gas stations expecting? They expect that the system in place works well enough for most people, and that people in less common situations will simply tolerate the idiosyncrasies/inconveniences imposed by the system. That is the typical "etiquette" that you were asking about. So, to be specific for this particular scenario, what people generally expect a person would do in your situation is to perform a bit of estimating. And, if your estimation is a bit off, then you'll somehow live with the consequences.
I hope that mostly answers the main question. Since you threw in a few specific questions, here are a few specific matching answers.
Am I supposed not to fill my tank entirely every time?
I'm sure that the owners of a gas station are happy to maximize income by having you fill a tank. If that means you pre-pay $30, then find out that wasn't enough, and you walk to the cash register to pay a bit more, then that's fine.
Can I prepay a large amount and ask for the difference between what I prepaid and what I filled? Could I pay a way larger amount when i don't know
Usually, yes, but that might not be guaranteed to work in all cases. There may be some limitations, particularly if the cash register doesn't have much money to be giving a certain amount of change. You can simply ask the cashier. For instance, many cashiers get unhappy when someone springs a $100 bill on them. But if you ask ahead of time, they usually aren't offended. Just ask if you can pay $50, but mention you don't know just how thirsty your car's gas tank is, and you might need up to $35 of it back. Ask, "Is that okay?" If they say yes, you ought to be fine.
I part this analysis with the following idea. If you pay for more gasoline than what you pump, and then you're too embarrassed to want to walk all the way back over to the cash register just to get your $1.37, another option may be to walk back to the cash register with an unpurchased snack in hand. Then, instead of the cashier just giving away money that's in the till, at least the cashier will feel like (s)he is completing another purchase, which is good for the company. Additional bonuses for you may include deliciousness (yay!) and obesity (hmmm... grrr).