This is from my experience, which is still limited (only European lines, some standard, some low-cost, >90% of my trips had hand luggage only)
Weighing
If you have both checked and carry on luggage, usually both are going to be weighed at the check-in/luggage drop.
If you only have a carry on and check-in online you have small chances to have your luggage weighed, however it might not be true if you apparently struggle with it. My hand luggage has never been checked in such circumstances.
Even though you haven't asked - checked luggage is always weighed at the check-in/drop off.
How much can I stretch
Well, assume anything over the limit will be charged or rejected if captured. I don't know what is the accuracy of the scales.
How can I increase my chances
Check in on-line, don't use checked luggage. If you have to either check in a luggage or use a check in counter, leave hand luggage with a friend while dropping your checked luggage but it's not guaranteed. You may be asked if you have a hand-luggage with you and requested to weigh it anyway.
You may use a wheeled bag to conceal weight but it may not work as well.
Of course the most secure approach is to keep within limits.
Size
As others have already mentioned there are those boxes to measure your hand-luggage size. If it is measured and fits, it passes to the plane. If not there are up to 3 options: luggage stays behind, pay excess or send as a check-in. Availability of last options depends when you do the measurement and airline policy.
Now in my experience the luggage was controlled this way just once or twice for me (so a rate of 2-3%), on a low-cost flights only and not even all of them. The size of my bag was then carefully chosen and the bag went in leaving some 1-2mm spare only (on each dimension).
How much can I stretch
I have travelled with a soft bag that I knew was slightly oversize on a regular line plenty of times and never had it checked. I've seen people with even bigger luggage go on-board without any problem.
But I've heard from fellow travellers that rules depend here. Some airlines on some flights enforce far more strict policy. They measure almost every bag that might be close to the limit and nothing goes by if doesn't squeeze into the measuring box. Moreover I've heard of one airline who refused squeezing - the bag had to fit itself without any pressure. I've been flying with the same carrier though and never experienced anything that restrictive myself.
So on a regular line you can usually stretch to some limits (probably few cm will go through). It may also be that the dimension not physically limited in the measurement box is treated with more flexibility but the other two dimensions have to fit accurately.
On a low-cost assume that nothing extra will pass if checked and check is quite possible.
How can I increase my chances
Pick regular lines over low-cost.
Use soft bags rather than rigid and put soft somewhat compressible thing on luggage sides to allow squeezing into the measurement box. An alternative (IMO better) is to use a rigid case that fits into the dimensions and squeeze things inside of it.
Find out which size of the luggage is unbounded during measurement and stretch there.
Again - fitting to the limits is the most foolproof method.
If you're uncertain about size of your bag you can go to the airport and either find a measuring tool yourself or ask at the airline counter. Make sure to use the tool for the correct airline as the dimensions differ significantly. You will also see if your bag just fits (in which case you should not overpack it) ar there is a bit extra space allowing you to squeeze in that additional pair of...
Bottom line reads
On flights with full load the chance of more strict approach is higher as all the luggage might not fit into the cabin.
The limits are there for a purpose. Actually there are three of them:
- Squeezing the bag into place for it on a plane (size)
- Balancing the plane (weight)
- Getting some extra bucks (both on low-cost airlines)
While third point one can treat as a rip-off, the other two are serious concerns. So rather than looking for ways to outsmart the system it's really better to keep to the limits.