Hailo 8L Disables ASPM (at least L0s)

General
1

I am wondering if the Hailo 8L (or Hailo 8 in General) Device just doesn’t support ASPM at all.

I am using an old ASUS P9D WS Motherboard and I already had to Build a Custom Kernel in order to disable some Quirks, patch ACPI Tables, but still even after all of this, the Hailo 8L Devices still seem to force ASPM to be turned off completely.

Even though the Message only says that ASPM L0s is Disabled. So ASPM L1 should (in theory) still work. Except it doesn’t.

Relevant output from dmesg | grep -i aspm:

[    0.904580] acpi PNP0A08:00: _OSC: OS supports [ASPM ClockPM Segments MSI EDR HPX-Type3]
[    0.905576] acpi PNP0A08:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI]
[   23.759698] hailo 0000:01:00.0: Disabling ASPM L0s 
[   23.764345] hailo 0000:01:00.0: Successfully disabled ASPM L0s 
[   23.936500] hailo 0000:02:00.0: Disabling ASPM L0s 
[   23.937272] hailo 0000:02:00.0: Successfully disabled ASPM L0s 
[   24.089774] hailo 0000:08:00.0: Disabling ASPM L0s 
[   24.090571] hailo 0000:08:00.0: Successfully disabled ASPM L0s

And powertop Reports only Package C-States of C3/C2 and higher.

Nothing like Package C6 or Package C8 which are needed for much lower Power Consumption …

Any Idea if this is a Hardware Limitation or if there are some Kernel Parameters to optimize it ?

2

Hailo-8 supports ASPM L0s and L1. You can check this by running sudo lspci -vvv

You should see something like the following under the Hailo-8 Capabilities:

LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
3

Thank you for your Reply.

Unfortunately it doesn’t look like that … Or, even if it would, it doesn’t seem to make a Difference :frowning_face:.

As I said, this is what appears in dmesg, even though I forced pcie_aspm=force:

[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-6.8.12-6-pve root=ZFS=/ROOT/debian ro "acpi_osi=Windows 2006" systemd.unified_cgroup_hierarchy=1 default_hugepagesz=2M hugepagesz=1G hugepages=16 transparent_hugepage=never intel_iommu=on iommu=pt pcie_aspm=force root=ZFS=rpool/ROOT/debian zfs_force=1 boot=zfs "acpi_osi=Windows 2006" systemd.unified_cgroup_hierarchy=1 default_hugepagesz=2M hugepagesz=1G hugepages=16 transparent_hugepage=never intel_iommu=on iommu=pt pcie_aspm=force root=ZFS=rpool/ROOT/debian zfs_force=1 boot=zfs
[    0.074463] Kernel command line: BOOT_IMAGE=/vmlinuz-6.8.12-6-pve root=ZFS=/ROOT/debian ro "acpi_osi=Windows 2006" systemd.unified_cgroup_hierarchy=1 default_hugepagesz=2M hugepagesz=1G hugepages=16 transparent_hugepage=never intel_iommu=on iommu=pt pcie_aspm=force root=ZFS=rpool/ROOT/debian zfs_force=1 boot=zfs "acpi_osi=Windows 2006" systemd.unified_cgroup_hierarchy=1 default_hugepagesz=2M hugepagesz=1G hugepages=16 transparent_hugepage=never intel_iommu=on iommu=pt pcie_aspm=force root=ZFS=rpool/ROOT/debian zfs_force=1 boot=zfs
[    0.074662] PCIe ASPM is forcibly enabled
[    0.074818] PCIe ASPM is forcibly enabled
[    0.335953] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[   22.048698] hailo 0000:01:00.0: Disabling ASPM L0s 
[   22.048756] hailo 0000:01:00.0: Successfully disabled ASPM L0s 
[   22.180487] hailo 0000:02:00.0: Disabling ASPM L0s 
[   22.180559] hailo 0000:02:00.0: Successfully disabled ASPM L0s 
[   22.312762] hailo 0000:08:00.0: Disabling ASPM L0s 
[   22.312827] hailo 0000:08:00.0: Successfully disabled ASPM L0s

And Output of lspci for all 3 different Hailo-8L:

01:00.0 Co-processor: Hailo Technologies Ltd. Hailo-8 AI Processor (rev 01)
	Subsystem: Hailo Technologies Ltd. Hailo-8 AI Processor
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 42
	IOMMU group: 2
	Region 0: Memory at f0204000 (64-bit, prefetchable) [size=16K]
	Region 2: Memory at f0208000 (64-bit, prefetchable) [size=4K]
	Region 4: Memory at f0200000 (64-bit, prefetchable) [size=16K]
	Capabilities: [80] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75W
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x2 (downgraded)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- TPHComp- ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled,
			 AtomicOpsCtl: ReqEn-
		LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
			 EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [e0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee003d8  Data: 0000
	Capabilities: [f8] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [100 v1] Vendor Specific Information: ID=1556 Rev=1 Len=008 <?>
	Capabilities: [108 v1] Latency Tolerance Reporting
		Max snoop latency: 71680ns
		Max no snoop latency: 71680ns
	Capabilities: [110 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Capabilities: [128 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [200 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 04000001 00000003 01070000 2fef015f
	Capabilities: [300 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Kernel driver in use: hailo
	Kernel modules: hailo_pci




02:00.0 Co-processor: Hailo Technologies Ltd. Hailo-8 AI Processor (rev 01)
	Subsystem: Hailo Technologies Ltd. Hailo-8 AI Processor
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 43
	IOMMU group: 2
	Region 0: Memory at f0104000 (64-bit, prefetchable) [size=16K]
	Region 2: Memory at f0108000 (64-bit, prefetchable) [size=4K]
	Region 4: Memory at f0100000 (64-bit, prefetchable) [size=16K]
	Capabilities: [80] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75W
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x1 (downgraded)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- TPHComp- ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled,
			 AtomicOpsCtl: ReqEn-
		LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
			 EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [e0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee003f8  Data: 0000
	Capabilities: [f8] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [100 v1] Vendor Specific Information: ID=1556 Rev=1 Len=008 <?>
	Capabilities: [108 v1] Latency Tolerance Reporting
		Max snoop latency: 71680ns
		Max no snoop latency: 71680ns
	Capabilities: [110 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Capabilities: [128 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [200 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 04000001 00000003 02070000 c140b44d
	Capabilities: [300 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Kernel driver in use: hailo
	Kernel modules: hailo_pci



08:00.0 Co-processor: Hailo Technologies Ltd. Hailo-8 AI Processor (rev 01)
	Subsystem: Hailo Technologies Ltd. Hailo-8 AI Processor
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 44
	IOMMU group: 15
	Region 0: Memory at f0004000 (64-bit, prefetchable) [size=16K]
	Region 2: Memory at f0008000 (64-bit, prefetchable) [size=4K]
	Region 4: Memory at f0000000 (64-bit, prefetchable) [size=16K]
	Capabilities: [80] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 25W
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s (downgraded), Width x2 (downgraded)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- TPHComp- ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled,
			 AtomicOpsCtl: ReqEn-
		LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [e0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00518  Data: 0000
	Capabilities: [f8] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [100 v1] Vendor Specific Information: ID=1556 Rev=1 Len=008 <?>
	Capabilities: [108 v1] Latency Tolerance Reporting
		Max snoop latency: 71680ns
		Max no snoop latency: 71680ns
	Capabilities: [110 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Capabilities: [128 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [200 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout+ AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [300 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Kernel driver in use: hailo
	Kernel modules: hailo_pci

So the relevant Bits are:

DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited

And:

LnkCtl:	ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+

And:

LnkCap:	Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+

So, in Summary:

  • According to dmesg, the hailo_pci Driver explicitely disables ASPM L0s. This can be verified in the hailort-drivers/linux/pcie/src/pcie.c File and the static int hailo_pcie_disable_aspm(struct hailo_pcie_board *board, u16 state, bool locked) Function
  • According to lspci, the Link Supports L0s and L1, but the Device itself supports only L0s with Latency Latency L0s, while L1 is with Latency L1 unlimited, i.e. it will never be able to Resume

What I tried to do (and it appears that this particular Motherboard is COMPLETELY broken with Regards to PCIe ASPM):

  • Set Kernel Command Line pcie_aspm=force
  • Set all relevant Options in the BIOS
  • Patch the ACPI Tables in InitramFS (fixes ACPI FADT declares the system doesn't support PCIe ASPM, so disable Error)
  • Disassemble BIOS with UEFITool and ifrextractor, then set some Hidden Options using setup_var.efi Tool from EFI Shell using a .nsh Script
  • Edit hailort-drivers/linux/pcie/src/pcie.c File and set a return 0; at the Beginning of hailo_pcie_disable_aspm(struct hailo_pcie_board *board, u16 state, bool locked) Function to prevent it from executing, so that the Hailo-8L will not force ASPM L0s to be turned off
  • Custom Kernel Built to disable Linux Quirk for ASMedia ASM1083/ASM1085 PCIe-to-PCI Devices (which DISABLE ASPM because the PCIe-to-PCI Bridge does NOT Support it)
  • Kill/Remove the PCIe Root Port using echo 1 > /sys/.../remove for the PCIe Root Port associated with the ASMedia ASM1083/ASM1085 Device
  • Automatically force ASPM to be enabled L0s + L1 on all Slots using a Script based on GitHub - notthebee/AutoASPM: A script that automatically activates ASPM for all supported devices on Linux

When I applied some Hidden BIOS Settings, the hailo Device in the latest PCIe Address (which was then most likely renamed from 08:00.0 to 05:00.0) started throwing out some DMAR Errors in dmesg and I could also see that its Inference Time in frigate was approx. 5x Times higher than the other Hailo Devices.

And even after all of this, it’s STILL stuck in Package PC2 State :roll_eyes:.

I managed to Undervolt by 90mV the Vcore which according to turbostat seems to have brought down the Package Power Consumption quite a bit (average around 10W).

But the Package C-States will NOT work, no matter what I do unfortunately :disappointed_relieved: .

4

Module Information

filename:       /lib/modules/6.8.12-6-pve/updates/dkms/hailo_pci.ko
version:        4.19.0
license:        GPL v2
description:    Hailo PCIe driver
author:         Hailo Technologies Ltd.
import_ns:      DMA_BUF
srcversion:     CAF39E55E7AD85E23673874
alias:          pci:v00001E60d000043A2sv*sd*bc*sc*i*
alias:          pci:v00001E60d000045C4sv*sd*bc*sc*i*
alias:          pci:v00001E60d00002864sv*sd*bc*sc*i*
depends:        
retpoline:      Y
name:           hailo_pci
vermagic:       6.8.12-6-pve SMP preempt mod_unload modversions 
....
parm:           o_dbg:int
parm:           no_power_mode:Disables automatic D0->D3 PCIe transactions (invbool)
parm:           force_allocation_from_driver:Determines whether to force buffer allocation from driver or userspace (int)
parm:           force_desc_page_size:Determines the maximum DMA descriptor page size (must be a power of 2) (int)
parm:           force_hailo15_legacy_mode:Forces work with Hailo15 in legacy mode(relevant for emulators) (bool)
parm:           force_boot_linux_from_eemc:Boot the linux image from eemc (Requires special Image) (bool)