After encountering this problem, I ran sudo apt update && sudo apt full-upgrade, which resolved the issue with the Hailo device. However, now the NVMe no longer shows up with the lsblk command.
eli@raspberrypi:~ $ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
mmcblk0 179:0 0 58.3G 0 disk
├─mmcblk0p1 179:1 0 512M 0 part /boot/firmware
└─mmcblk0p2 179:2 0 57.8G 0 part /
eli@raspberrypi:~ $
Has anyone faced a similar issue after adding an NVMe ? Any suggestions on how to resolve this?
The -28 error typically maps to ENOSPC (No space left on device), which often means that system resources like interrupt lines or MSI (Message Signalled Interrupts) are exhausted or misconfigured—likely due to the addition of the NVMe device.
Then, after you did a full upgrade, the Hailo device started working—but now the NVMe is not detected by lsblk.
Just remove the Hailo device , and go again, Adding it LAST!
Have a look at the WIKI also, for more information.
Do follow the steps and only use images from there.
I have the same or similar problem with the Seeed Studio reComputer AI R2000 model R2130-12. I use the Seeed Studio Raspberry PI 27 watt AC/DC adapter with the R2000. The Hailo-8 accelerator works fine, that is the Hailo-8 driver always loads if the Hailo-8 accelerator is the only card in the Seeed Studio PCIe3.0 to dual M.2 hat for Raspberry PI 5. The Hailo-8 driver also loads regardless of which M.2 key the Hailo-8 accelerator is installed.
Note: The PI OS and eeprom is always the latest version (updated/upgraded with sudo apt update && sudo apt upgradeable and with the sudo raspi-config command), checked many times.
The problem is the Hailo-8 driver never loads if any type of NVME SSD driver is in either M.2 key (blank, boots from, etc… For the Hailo-8 driver, I find the message "Failed to enable MSI -28 " listed in the startup log (dmesg | grep -i hailo). I have tried several versions of the Hailo driver. Reinstallation of the Hailo-8 accelerator and its driver is not the solution, although I have tried this multiple times. If I power down the R2000, remove the NVME SSD, leaving only the Hailoi-8 accelerator in the dual M.2 hat, restart the rR2000, the Hailo-8 driver loads and the Hailo-8 accelerator works fine - same Hailo-8 software, ex. the rpi examples tests from Hailo run correctly. When I put any NVME SSD back in the R2000 (power down, etc) and start up the R2000, the Hailo-8 driver does not load. I have also used the lowest power, Raspberry PI certified dramless NVME SSD I have, a 512 GB 2230 NVMD SSD (Biwin Storage, uses 3.3V at 2 amps). Note. The NVME SSD always works.
This started out when I loaded with loading the Raspberry PI OS with a network installation on a SD card (press shift while the R2000 is powering on for the first time and selected the network installation option for a blank SD card). The Hailo-8 accelerator worked fine after the initial installation. I then moved the PI OS image to a NVME SSD and booted from the NVME SSD. I find that using a SD card for AI type work is not a viable solution (too many writes to the SD card and it is too slow).
The Hailo-8 accelerator is always displayed with the lspci command, without a NVME SSD installed and also with a NVME SSD installed.
0001:06:00.0 Co-processor: Hailo Technologies Ltd. Hailo-8 AI Processor (rev 01)
When the NVME SSD is also in the M.2 hat, the Hailo-8 driver starts to load and then fails. regardless of the version of the Hailo-8 driver (version 4.19 displayed below on a fresh installation of the PI OS; I also used version 4.20 and 4.21 in other tests).
[ 4.367572] hailo_pci: loading out-of-tree module taints kernel.
[ 4.376702] hailo: Init module. driver version 4.19.0
[ 4.390804] hailo 0001:06:00.0: Probing on: 1e60:2864…
[ 4.390811] hailo 0001:06:00.0: Probing: Allocate memory for device extension, 11632
[ 4.390830] hailo 0001:06:00.0: enabling device (0000 → 0002)
[ 4.390836] hailo 0001:06:00.0: Probing: Device enabled
[ 4.390854] hailo 0001:06:00.0: Probing: mapped bar 0 - 0000000064485481 16384
[ 4.390858] hailo 0001:06:00.0: Probing: mapped bar 2 - 00000000e06a8bbc 4096
[ 4.390860] hailo 0001:06:00.0: Probing: mapped bar 4 - 00000000ec0a1d16 16384
[ 4.390863] hailo 0001:06:00.0: Probing: Force setting max_desc_page_size to 4096 (recommended value is 16384)
[ 4.390873] hailo 0001:06:00.0: Probing: Enabled 64 bit dma
[ 4.390875] hailo 0001:06:00.0: Probing: Using userspace allocated vdma buffers
[ 4.390879] hailo 0001:06:00.0: Disabling ASPM L0s
[ 4.390882] hailo 0001:06:00.0: Successfully disabled ASPM L0s
[ 4.390945] hailo 0001:06:00.0: Failed to enable MSI -28
[ 4.390947] hailo 0001:06:00.0: Failed Enabling interrupts -28
[ 4.390949] hailo 0001:06:00.0: Failed activating board -28
[ 4.390957] hailo 0001:06:00.0: probe with driver hailo failed with error -28
The NVME SSD always works. The R2000 boots fine from the NVME SSD, with the Hailo-8 accelerator installed or without it, and regardless of which M.2 key is used. The led light on the Hailo-8 accelerator is always red when the power is on.
As insufficient power could be a cause, on the PCIe 3.0 to dual M.2 hat for Raspberry PI 5, I checked if the power was also being received at the pogo pins. The M.2 hat gets power from the pci cable and also from the 40 pin connector via pogo pins. From the pogo ground pin, and another pogo pin, I measured 5 volts DC. I measured 5 volts DC with a third pogo pin and the ground pin. On a fourth pogo pin and the ground pin, I measured 3.3 volts DC. On a fifth pogo pin and the ground pin, I measured 0 volts. Power is getting from the 40 pin connector to the dual M.2 hat.
I have yet to find a fix for this. I have been able to use a USB 3.0 connected SSD drive with the Hailo-8 accelerator on the R2000. This setup would work for AI work. I still prefer to be able to use the second M.2 slot for a NVME SSD as this is more compact and faster, but so far without any luck.
I don’t have a solution. Will update if I find one. But thanks for sharing your findings. I am experiencing exactly this same issue. When I first added the NVMe SSD, it did not show up. After powering down and up multiple times investigating, now it is showing up and appears to be working normally, but I have the same issues reported in dmesg you post above.
I found a solution that worked for me! From this topic which I can’t directly link to (perhaps you can search) hailo-8-c-processor-not-detected-by-hailortcli-etc-when-on-a-geekworm-pcie-switch-for-rpi5 which linked to issue 6206 on raspberry linux’s GitHub,
The suggestion was to add a device tree overlay to the config:
dtoverlay=pciex1-compat-pi5,no-mip
In my case, I was running HassOS so I needed to take the SD card to a different machine, mount the volume, and I could edit the config.txt which in that case was at the root of the mounted folder (I’ve also seen /boot/config.txt referenced and in another raspberry pi os it was /boot/firmware/config.txt). There are lots of lines in that file and I just needed to add the above to that file.
I wanted to get a raspberry pi OS working as well but found that the Hailo was showing up but the SSD was not.
I compared my working HassOS config and cmdline with the raspberry pi and found that the raspberry pi included several PCIe mitigation/management scripts where HassOS did not. I updated my raspberry pi configs to match and all appeared to work successfully.
I kept this in the config.txt:
dtoverlay=pciex1-compat-pi5,no-mip
Though I noted from the Seeed Studio wiki for the M.2 Hat that they recommend: dtoverlay=pciex1-compat-pi5,no-mip,mmio-hi
And I removed these from the Raspberry PI OS’s default command line (/boot/firmware/cmdline.txt) – as in – they were present out of the box but I removed them so they are no longer there:
Removed Parameters:
pci=pcie_bus_safe - Forces PCIe bus configuration to use safe/conservative settings
PJ 