Blackwell RTX5090 compatibility issues

General Discussion
,
1

GPU Compatibility Issue Report (Blackwell RTX 5090 + CUDA 12.5 + cuDNN 9.10 + Hailo DFC)

Subject: GPU compatibility issue on RTX 5090 (Blackwell) — CUDA_ERROR_INVALID_HANDLE with Hailo Dataflow Compiler

Hello,

I am experiencing a GPU initialization failure when running the Hailo Dataflow Compiler on a system equipped with an NVIDIA GeForce RTX 5090 (Blackwell architecture).

Even though CUDA, cuDNN, and TensorFlow are successfully installed and detected, the CLI fails immediately upon importing hailo_model_optimization, with the following error:

'cuLaunchKernel(...) failed with CUDA_ERROR_INVALID_HANDLE'
[Op:AddV2]

System Information

  • GPU: NVIDIA GeForce RTX 5090 (Blackwell architecture)

  • Driver: 470.195.03

  • CUDA toolkit: 12.5 (nvcc reports V12.5.82)

  • cuDNN: 9.10

  • OS: Ubuntu 24.04

  • Python: 3.12

  • Hailo DFC version: 3.33.0

TensorFlow GPU visibility test also fails with similar symptoms unless I force CPU execution (CUDA_VISIBLE_DEVICES=-1), in which case Hailo runs correctly.

Question

Is the Hailo Dataflow Compiler currently compatible with NVIDIA’s new Blackwell architecture (SM_120)?

If not, is there an expected timeline for support, or any recommended workaround (specific CUDA/cuDNN version, patched TensorFlow build, etc.)?

I would like to utilize GPU-based hardware emulation but will temporarily continue with CPU fallback if necessary.

Any guidance would be greatly appreciated.

Thank you.

2

Hey @gordon_koo,

Thanks for bringing this up — happy to help clarify the Blackwell architecture situation.

Here’s what’s going on:

  1. NVIDIA Driver Version: You’re currently on 470.195.03, which is quite a bit behind. To use CUDA 12.5.1 (required by the Hailo Dataflow Compiler), you’ll need to upgrade to driver version 555 or later.

  2. Blackwell GPU Support: Officially, Blackwell (SM_120) isn’t supported yet. That said, it can work with the right setup — it’s just not guaranteed.

What I’d recommend:

  • Update your NVIDIA driver to 555+
  • Double-check your CUDA and cuDNN versions: You’ll want CUDA 12.5.1 and cuDNN 9.10 to match DFC 3.32+/3.33.0 and TensorFlow 2.18.0
  • Let DFC manage its own TensorFlow: It’s best to let DFC handle TF within its own virtual environment to avoid version conflicts

A heads-up:

Even with everything updated, support for Blackwell GPUs isn’t official yet, so errors like CUDA_ERROR_INVALID_HANDLE might still pop up. We’re not ignoring it — it’s just that full support isn’t in place yet.

What you’re doing now (CPU fallback) with CUDA_VISIBLE_DEVICES=-1 is actually the most reliable workaround for the moment. It’s fully supported and stable, just slower compared to GPU execution.

I’ll check in internally and will update this thread once there’s more clarity on official Blackwell support.

Let me know if anything’s unclear or if you hit other issues!

1 Like