Single-entry visa allows you to enter Chinese territory under Chinese border control exactly once. Hong Kong and Macao are not administered by Chinese border control authorities, and obviously neither is Taiwan.
As long you have exited through a Chinese border control checkpoint, Chinese authorities consider that you have left China. The checkpoints at "internal" borders between mainland China and Hong Kong/Macao are exactly the same as, say, checkpoints on the China-Russian border, so entering Hong Kong is considered leaving China as far as Chinese immigration authorities are concerned.
EU citizens generally don't need visas to enter Taiwan, ROC or Hong Kong, so, yes, according to your definition, you are only entering China once and you exit China when you leave for Taiwan. You are entitled to 90 days of visa free stay in Hong Kong and Taiwan each, unless you are a British citizen, in that case you would be able to stay for up to 180 days in Hong Kong.
You can obtain a 5-day visa for Shenzhen at the Luohu, Huanggang, Shekou, and Fuyong Pier border checkpoints only. Note that the Shenzhen Bay Port and Futian Port border checkpoints do not have visa offices and do not issue visas; you will be refused entry if you attempt to enter at these checkpoints. If you take a bus to Shenzhen, make sure the bus goes to Huanggang. If you take the MTR, make sure you buy your ticket to Lo Wu, not Lok Ma Chau.
In addition, you could visit the Pearl River Delta cities of Guangzhou, Shenzhen, Zhuhai, Foshan, Dongguan, Zhongshan, Jiangmen, Zhaoqing, Huizhou and Shantou for up to 6 days provided you travel with a tour group organized by a Hong Kong travel agency. If you visit to Shenzhen is short, you should be fine.
However some sources (possibly outdated) - like this one- claim that French and American citizens are not eligible for the 5-day Shenzhen visa (but eligible for the 6 day visa-free group tour), so you might want to confirm this. Some other sources claim that citizens of certain countries are subject to a higher visa fee for the Shenzhen visa (such as Chinese Wikipedia), and I personally have memories of seeing Americans getting visas at Huanggang.