You would want to make certain that your Family visit visa wasn't curtailed when your settlement application was refused. If it wasn't, there is a risk of it being cancelled by an Immigration Officer where you to present for entry at the border.
As the refusal was in 8 May and the appeal filed on 31 May, understand that it can take up to 12 weeks for the visa section to send your appeal bundle to the Tribunal, another 6-8 weeks, or longer, for a hearing (either oral or on paper without an oral hearing).
Going for a visit might not be a problem, but there's no way to predict what could occur when seeking entry. Because of the failed application, it may not go well. You could be refused entry or allowed to enter for a limited period; either would another strike against you on your immigration record. Depending on the reasons for the settlement refusal, it may be wiser to wait for the appeal decision.