DFC 3.29 fails to work with GPU (WSL2, Ubuntu 22.04)

General
1

Hello, we have been working with DFC 3.23 in WSL2 with Ubuntu 20.04 environment for a while. But since we require our in-house learnt model of YOLOv8 to be compiled with DFC, it seems that 3.23 does not support in model script to call nms_postprocess() with the meta_arch arguement of yolov8.

So, we decided to install another virtual machine, with the Ubuntu-22.04, CUDA 11.8 and cuDNN of 8.9. DFC version is up to date with 3.29.

the thing is, everytime we try to compile our custom yolov8 model, following error message appears on the log and it automatically operates with the device of CPU. Here is the part of the logs that mention the situation.
image

and here’s the part of the logs showing it’s working with CPU only

image
image1207×113 7.21 KB

We can verify that CUDA driver and GPU are working just fine by following methods

  1. verify it with tensorflow python API
    from tensorflow.python.client import device_lib
    print(device_lib.list_local_devices())
    print("Num GPUs Available: ", len(tf.config.list_physical_devices(‘GPU’)))
    image
    image1205×904 37.7 KB

as you can see, GPU is recognized by tensorflow

  1. by building the CUDA sample app of deviceQuery and run it
    image
    image1153×1036 37.1 KB

device is queried

  1. by learning the simple network with Tensorflow and see the GPU is utilized

→ succeeded

  1. WSL2 of Ubuntu-20.04, lower version of CUDA, CUDNN and DFC of 3.23

→ CUDA and CUDNN is loaded and working fine with the DFC

Right now, we are on a sort of test-the-water phase for the YOLOv8 so, it does not require us to run the optimization level other than 0. But, if it goes higher, not utilizing GPU would be a problem.

is this known nature of the 3.29? To be more specific, with the combination with the WSL2 environment? If so, could you give us any pointer to solve the problem?

3

Oh, I forgot to mention 1 other thing.
We already have tried to run Docker Container on the same environment, with the NVIDIA GPU emulation capability, with -gpus all option on the docker script.

nvidia-smi recognize the GPU and the CUDA, but when we actually run it,

tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:1279] could not retrieve CUDA device count: CUDA_ERROR_NOT_INITIALIZED: initialization error

shows up also on the Docker.

4

Hi @ksj,
DCF on WSL2, is a usecase that is outside of the defined product guidelines. We know that it works for the most part, but actually, the GPU integration is the hard part that was not solved. I hope that we could find time to work on that.

5

Thank you very much for making it clear.
Thought I might have done something wrong on constructing the environment and I have been installing/uninstalling same CUDA and cuDNN from different repos for several days. Glad that it was not on me. :slight_smile:

Have a great day.