Error when creating multiple instances of HailoAsyncInference

General
1

Hi,
When I tried to create multiple instances of HailoAsyncInference, and load a model in each instance and run it in a separate thread, the first instance was created without any issue, but it reports error for the second instance like this:
[HailoRT] [error] CHECK failed - Failed to create vdevice. there are not enough free devices. requested: 1, found: 0.
Does it mean one Hailo device can only load and run one model at any time (my model size is about 2 MB), or there is some other way to run multiple models on one Hailo device?

Thanks!

2

Hey @TT2024,

If you’re looking to run multiple models, we recommend using the scheduler rather than creating multiple vdevice instances (HailoAsyncInference). The scheduler is specifically designed for managing multiple model executions efficiently.

3

Thanks for your suggestion! Do you have some simple examples that I can start with?

4

Never mind, I will check out this one first:

5

Hi @omria , I would like to do switching/loading models manually. How can I check if the hailo device is busy (running another model) before I load a new model?
Thanks for your help!

6

Hey @TT2024 ,

This is the basic flow :

  1. Create one HailoDevice.
  2. Load and run the first model.
  3. When needed, unload the model (call .release()).
  4. Load the new model and continue inference.
7

Hi @omria , thank you very much! (BTW, I believe by “HailoDevice”, you meant VDevice of hailo_platform library.)