Recreate model with different batch sizes fails with error about "internal_queue_size"

General
1

I’m getting this error when trying to load my model twice. First with batch size 1, and then with batch size 2:

Trying to configure a model with a batch=8 bigger than internal_queue_size=4, which is not supported. Try using a smaller batch.

Pseudo code that gives the error:
device = CreateDevice()
model = CreateModel(batchsize=1)
model.Run()
model.Close()
model = CreateModel(batchsize=2) // … error produced

Now the following code flow works, and does not produce the error. But having to do this adds additional complexity:
device = CreateDevice()
model = CreateModel(batchsize=1)
model.Run()
model.Close()
device.Close()

device = CreateDevice()
model = CreateModel(batchsize=2) // this works

So bottom line is that I need to close and reopen the device if I want to change the batch size of my model. This seems like a bug in the driver?

I’m doing this from C++. Sorry for not producing a repro, but my code is a few layers above a raw C++ test file.

2

Hey @rogojin

The batch size issue arises because the Hailo hardware has specific requirements for efficiently managing its internal resources. When a model is loaded onto the device, it configures the internal queues based on the specified batch size during compilation. These queues are optimized for optimal performance and resource utilization. However, attempting to load another model with a different batch size without re-initializing the device causes a mismatch between the existing queue configuration and the new model’s requirements. The Hailo hardware enforces these constraints to maintain predictable operation and guarantee effective resource allocation, preventing unpredictable behavior and potential resource conflicts.

To resolve this issue, you can:

  1. Close the device and reopen it as you suggested.

  2. Utilize the Hailo Model Scheduler or compile multiple configurations together, enabling seamless switching between different model configurations with varying batch sizes without device reinitialization.

3

Thanks for the detailed response.

I’ll take a look at the Model Scheduler, but for now I’ve found a simple solution that works for me:

  1. Create Device
  2. Create all models that I need upfront
  3. Run the models

I’m guessing the resource allocator is delayed until first model run.

1 Like