I used a sniffer to capture the scan response packets in the office. Nowadays there are a lot of BLE devices everywhere. Only the peripheral was powered up and the central was not powered. Below shows the scan response transmitted by the peripheral.
It shows that the peripheral was working hard to entertain the scan requests coming from other BLE devices. If the central is placed in a busy environment, it will compete with other devices to make the scan request. It is not surprising that the central may need considerable attempts before getting a response.