Hey @user252 — sorry, I should’ve been clearer in my one-liner, because the trick is the opposite of what you’re trying to do, and your diagnosis is actually 100% correct.
The key thing: the stub does NOT run DFC on the 5090. It makes DFC fall back to CPU and never touch the GPU at all. That’s why the PTX error goes away — there’s no PTX-JIT happening anymore.
You nailed the root cause: the bundled TF 2.18.0 in the DFC env only ships PTX/cubins up to an older arch, and there’s no ptxas/cubin for sm_120 in there. So the moment TF tries to launch a kernel on the 5090, the driver’s JIT chokes (CUDA_ERROR_INVALID_PTX). You can’t fix that from the outside without rebuilding TF against a CUDA 12.6+ toolchain that knows sm_120 — not worth it. So I just don’t let it use the GPU.
What “use cuda stubs” actually means in my setup: I LD_PRELOAD a tiny hand-built libcudart.so.12 (and a matching libcuda.so.1) that exports the CUDA Runtime symbols as no-ops, with cudaGetDeviceCount() returning 0 devices (and success). TF then prints No suitable GPU found, falling back to CPU and the whole pipeline — model parse, full optimization (Adaround/QFT), and compile — runs on CPU to completion. Slower than a GPU optimize, but it finishes and the HEF is byte-identical to what a supported-GPU box produces.
So to directly answer your questions:
1. Which stub / where / order / LD_PRELOAD?
It’s not the CUDA toolkit’s stubs/ dir (those forward to the real driver — that’s what’s biting you). It’s a custom ~16 KB no-op .so. I LD_PRELOAD it (not LD_LIBRARY_PATH), so it wins over the real cudart that ships in the DFC conda env. Build your own in 30 seconds:
c
// cudart_stub.c — no-op CUDA runtime, reports 0 GPUs
#include <stddef.h>
int cudaGetDeviceCount(int *n){ if(n) *n = 0; return 0; } // 0 = cudaSuccess, 0 devices
int cudaGetDevice(int *d){ if(d) *d = 0; return 0; }
int cudaSetDevice(int d){ (void)d; return 0; }
int cudaGetLastError(void){ return 0; }
int cudaPeekAtLastError(void){ return 0; }
const char* cudaGetErrorString(int e){ (void)e; return "no cuda (stub)"; }
// registration hooks TF calls at load — make them harmless
void __cudaRegisterFunction(){} void __cudaRegisterVar(){}
void* __cudaRegisterFatBinary(void *x){ return x; }
void __cudaRegisterFatBinaryEnd(){} void __cudaUnregisterFatBinary(){}
bash
gcc -shared -fPIC -o libcudart.so.12 cudart_stub.c
# then, when invoking the DFC python:
LD_PRELOAD=$PWD/libcudart.so.12 python your_compile_script.py
If your TF build dlopen’s libcuda.so.1 directly and crashes before cudart even loads, make a second copy named libcuda.so.1 with the same symbols and add it to the preload (LD_PRELOAD="…/libcudart.so.12 …/libcuda.so.1").
2. Versions:
DFC 3.33.1, bundled TF 2.18.0, NVIDIA driver 570.x. I did not upgrade cudart/cudnn/nvrtc/ptxas in the env — the whole point is to avoid the GPU code path, not fix it.
3. Env vars:
None of the JIT/cache vars. If you’d rather not build a stub, try the trivial version first — just force CPU and don’t enable the GPU:
bash
export CUDA_VISIBLE_DEVICES=-1 # (you had it set to 0 — that's the problem)
On a lot of setups that alone is enough to get TF to fall back to CPU cleanly. The LD_PRELOAD stub is the belt-and-suspenders version for when TF still probes the driver and dies during init.
4. Did a full optimize run?
Yes — full Adaround/QFT to completion on CPU, not just parsing. First run has the usual TF graph-build delay but no JIT delay (no GPU). It’s CPU-speed, so a big model’s optimize is slow, but it does not fail.
5. Bare metal or container?
Bare metal (a local venv/conda DFC env), no container.
One more thing to be clear on: this is a compile-time / DFC-optimize problem, not a runtime one. Inference on the Hailo NPU never uses the GPU — a compiled HEF runs via HailoRT with zero CUDA. So this only ever affects compiling/optimizing a model on the 5090 box, never running one.
TL;DR: the 5090 is fine as a box, but DFC’s TF 2.18 can’t target sm_120, so the move is to stub CUDA out and optimize on CPU — set CUDA_VISIBLE_DEVICES=-1 (try first), and if TF still tries the driver, LD_PRELOAD the no-op cudart above so it sees 0 GPUs. Hope that unblocks you.
Separately — if you’re looking at custom compilation, or converting a PyTorch/Python model into a smaller embedded edge model (we run a lighter-weight owned stack, AxonML, for exactly this kind of thing), that’s something we do at AutomataNexus. Feel free to reach me at devops@automatanexus.com and I’m happy to help.