M key (vs B+M key) in QNAP not recognized (perhaps pci lane downshifting?)

[Nechaken]

I seem to have a problem getting a QNAP NAS to recognize a Hailo-8.

Hardware – I have a QNAP TS-464 which has 2 M.2 slots, each supports only x1 lane. The TS-464 also has a single Gen3 PCIe x4 slot with x2 lanes supported.

While QNAP lists the B+M Key version as compatible, for easier availability, I picked up the M key version, product # is: HM218B1C2FAE. I understand the M key supports support 4 lanes, BUT should downshift to use x2 lanes, and possibly even x1 lane.

But it’s not detected. I’ve gone through the following to verify everything possible.

1. Insert the Hailo-8 (M key) into the first M.2 slot on the QNAP. Does NOT show up in lspci.

2. Insert the Hailo-8 into the other M.2 slot on the QNAP. Does NOT show up in lspci.

3. Insert the Hailo-8 into an M.2 slot on my laptop. It works on the laptop. It is detected so I ran the integration tool and it passes the tests (laptop’s M.2 supports x2 lanes, so I know the Hailo-8 will negotiate down to 2 lanes).

4. Thinking I possibly have a problem with the M.2 slots on the QNAP, I test them by putting a 1TB Samsung NVMe in each slot. The SSD works fine in either slot, although it is downgraded to x1 lane.

5. Now I’m thinking possibly the Hailo-8 won’t downshift to use only 1 lane of the QNAP’s M.2 slots. So, I put a PCIe to M.2 adapter (Startech PEX4M2E1 ) into the QNAP’s PCIe x4 slot. Still does NOT show up in lspci! The red light is on, so I know it’s getting power.

6. Just to make sure the Startech adapter is working, I put a NVMe drive into it as well. Works fine! So it’s not the adapter!

QNAP lists the HM218B1C2LAE as being compatible & supported on QTS > 5.1. I understand this product # is the B+M keyed version that supposes 2 PCIe lanes. But, the HM218B1C2FAE clearly can downshift to use x2 lanes.

Other than they keying, are there any differences between the M-Key Hailo-8 (HM218B1C2FAE) and the B+M Key version (HM218B1C2LAE) ? Power requirements? Different kernel driver between the M Key and B+M Key?

But, even if the kernel driver was different between the -LAE and -FAE versions, the device should still enumerate and be visible in lspci, right ?

I’m somewhat at a loss here… any further suggestions ?

[KlausK]

Welcome to the Hailo Community!

@Nechaken

Other than they keying, are there any differences between the M-Key Hailo-8 (HM218B1C2FAE) and the B+M Key version (HM218B1C2LAE) ?

The modules may use different external components. However they should not make a difference. The Hailo-8 chip itself is the same on both modules.

@Nechaken

Different kernel driver between the M Key and B+M Key?

No. The chip is the same.

@Nechaken

the device should still enumerate and be visible in lspci, right ?

Yes, I have used the Hailo-8 M.2 module key M on Raspberry Pi (single PCIe lane) with many different HATs and PCIe adapters without issues. So, single PCIe lane should not be a problem.

Do you have a setup with 4 PCIe lanes? Just to make sure all lanes work. You have already tested many configurations.

[Nechaken]

This is on W10, in an X4 slot…

I’m using the HailoRT with the 4.18 firmware, since that’s the driver version QNAP includes…

[Nechaken]

Oh, and that is through the StarTech PCIe → M.2 adapter, so that confirms the adapter will work with x4 lanes. Very bizarre it doesn’t work on the qnap.

[Nechaken]

Digging a little further – I don’t believe anything in starting up would affect anything

[~] # cat /proc/cmdline
consoleblank=0 console=ttyS0,115200n8 intel_iommu=on,igfx_off i915.enable_hd_vgaarb=1 iommu=pt i915.disable_power_well=0 i915.enable_rc6=0 i915.enable_guc=2 memmap=2M$0x8000000 ramoops.mem_address=0x8000000 ramoops.mem_size=0x200000 ramoops.console_size=0x100000 usbcore.autosuspend=-1 zswap.enabled=1 zswap.compressor=lz4 BOOT_IMAGE=/boot/bzImage root=/dev/ram0 rw

The iommu settings should only affect passthrough to a VM, not whether the host system could see the card on the PCIe bus.

Given the problem is the failure of the card to enumerate, could this be something with power states or failure of the qnap to provide REFCLK? Would it be worth tying CLKREQ low to see if that has any effect ? I’m kinda grasping at straws, but this is weird….

[0samaBinSmokin]

Get a QNAP-branded pcie adadpter - QNAP is extremely anal when it comes to anything pcie-slottable. Weird that m.2 don’t work.. I’m running 2x Hailo-8 key M each in its own m.2 slot in a TVS-h1688x - works like a charm - my each slot bandwidth matches that of a single module tho.. I’ll add that I’m running QuTS>5.2 and did update firmware on Hailo’s. Also ran em successfully in a qnap twin m.2 pcie adapter.

[Nechaken]

Well, glad to hear they’re working on your setup, at least!

And yeah, this is bizarre. I believe the advantage of the QNAP QM2 series cards is their use of an onboard PCIe switch – which is why they can do 2 NICs + 2 NVMes or 4x NVMe, etc. from a single PCIe x4 slot. For a single M.2 card, a switch shouldn’t be necessary, and I’ve tested a single NVMe in the PCIe adapter.

In any event, I’m waiting for a 2-lane B+M version of the Hailo to arrive – I’ll see if it works in either the M.2 slot or with a PCIe adapter.

[0samaBinSmokin]

Yeah, pretty much. I have the QM2-4P-384 and it’s an x8 unit tho. Wait, the B+M key version was the one u tested? Or am I confused?

[Nechaken]

So everything I’ve written above has been about the M key (4 lane) Hailo-8. It should and (as confirmed in other machines) DOES down train to use 2 lanes – so I shouldn’t need to use an adapter card with a PCIe switch to get it to work in the TS-464.

To try to pin this down, I just ordered a B+M Key version of the Hailo, and I’m waiting for that to arrive to test.

On a whim, I threw a graphics card into the 464’s PCIe slot – it is detected and runs at x2 PCIe lanes:

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV620 LE [Radeon HD 3450] (prog-if 00 [VGA controller])
Subsystem: ASUSTeK Computer Inc. Device 01f4
Physical Slot: 4

	LnkCap:	Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
		ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
	LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
		ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
	LnkSta:	Speed 2.5GT/s, Width x2 (downgraded)
		TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

[Nechaken]

So this is insane! I finally have it working, BUT it appears to be some weird mechanical / electrical issue between the Hailo and the QNAP.

Inserting the card normally doesn’t work. But, with the power on, if I lift the module beyond a ~ 45 degree angle, the card is detected, the driver probes, the red LED goes out, and the Hailo works. I’ve confirmed this with both the full Hailo-8 M-key as well as a Hailo-8L B+M key.

In the photo below, the card is probably at a ~ 30 degree angle. Actually have to lift it further OUT of the slot for it to be recognized, then place it back down. This isn’t persistent across reboot. Even on a warm reboot, I have to repeat this process. Something very very strange going on between the Hailo and the QNAP. Measured the thickness of both Hailo cards, and they’re within .03 mm of a NVME that works fine.

[omria]

Hey @Nechaken,
Thanks for bringing this up - it’s quite an interesting case!
I’ve passed it along to our Hardware team to take a look, as they might have some useful insights.

[Nechaken]

Right? Lifting it that far up has to be disconnecting the top (odd) row. Perhaps something weird with pin 69 (config / pedet) ?