U.S. CBP does care about soil on your shoes, but as long as you've made a reasonable effort to clean them, then it won't be a problem, especially if the soil was not from a farm or some such thing.
As gerrit mentioned, two of the questions on the customs declaration form that you fill out when entering the United States are:
I am bringing (...) (d) soil or have been on a farm/ranch/pasture
I have (We have) been in close proximity of livestock (such as touching or handling)
When I actually had visited a farm in Thailand and so answered yes to the first of those questions, the immigration officer asked about it. When I told him I had been to the farm in Thailand, he had me go to the secondary agricultural inspection.
At agriculture inspection, they asked what sort of animals I had been around and when I told them that it was elephants rather than livestock, they weren't very concerned with that. However, they did still ask whether I had mud and such on the shoes or clothes that I was wearing while there. In my case, it had been dry while I was there and the ground was mostly covered in grass, so I (truthfully) told them that my shoes probably had some light dust, but no caked on mud. They said that wasn't a problem at all and didn't even bother physically looking at anything in my bag.
If it had been rainy while I was there and my shoes had actually been muddy, I expect they'd have had me clean them as gerrit experienced, especially if I had been around cows or other such livestock rather than just elephants.
Given that they weren't concerned at all about the light dust and such, I don't think you'll have any problem at all if you've thoroughly cleaned your shoes but were just unable to get them completely clean. Since you haven't even been on farms, it sounds like you should truthfully be able to answer 'no' to the inspection questions, in which case they're unlikely to ask about, look at, or care about your shoes at all.