Yes, you can. My email asking your question, including all of the specifics, to the ministry in Japan which oversees such, received this response:
This is the Immigration Center at “Immigration Bureau of Japan” of “Ministry of Justice”. In regards to your inquiries, we will answer as following.
As for Immigration Bureau of Japan, as long as temporary visitors leave Japan by the expiration dates of landing permissions, there is no limitation on transportation people use.
In addition, several shared their experience on Lonely Planet.
Giora: I took a boat recently from Busan to Shimonoseki, in Japan. The boat company wouldn't let me buy a one-way Busan-Shimonoseki boat ticket without showing them proof that I had some kind of flight/boat ticket out of Japan. I suspect it's the same thing for the Busan-Fukuoka ferry. Japanese immigration didn't ask me for anything, but I wouldn't have gotten on the boat in the first place without some kind of a ticket out of Japan.
Bamse: ...[A]s a tourist, you need to have a ticket out of Japan when entering Japan (whether by flight of boat does not matter). Also as far as I know usually the airline company (or boat company in your case) and not Japanese immigration would check this requirement, basically because they are the ones risking a penalty if they let you in on a one way ticket.
As you appear to be from a country (UK) whose nationals can enter Japan visa free, and remain for 88 days, the feedback from the immigration authorities would suggest that you would want to carry with you details of your flight departure from Japan, and book a round-trip ferry passage for the trip to Korea..
On the return, according to USAToday travel writer Eric Som, Korean law requires all ferries leaving South Korea to Japan to check the immigration paperwork of all passengers.
And The Mad Traveler Kevin Revolinski has an excellent article on Taking the Ferry from Korea to Japan (Busan-Fukuoka), with everything you could possibly need to know, including the terminal GPS coordinates.