You should have a look at the Commonwealth of Dominica. Hotels are not really cheap. However, there are some guesthouses and cottages in the capital Roseau, but also in other villages around the island. I was there three years ago, but I stayed in a hotel, so I cannot comment on guesthouses.
Dominica has some nice diving spots. There is no real mass tourism in Dominica. In this respect, the biggest annoyance you could have are the big cruise ships calling in Roseau. Note that in January/February this can happen 2-3 times a week. But it is easy to escape the crowds ... Just go to less popular places. If you want somewhere really calm, you should settle down in Portsmouth. There you will find accommodations and also a diving school.
If you are not diving, you can do some hiking and enjoy Dominica's wild and splendid nature!
Here is a website, where you will find some information on accommodation and things to do in Dominica.