Help regarding: Python Inference Tutorial - Multi Process Service and Model Scheduler

General
, , , ,
1

Following my last problem on context switching to run 2 models, i’ve decided to start anew after seeing the HailoRT Documentation .
so i started the new attempt by understanding and editing the documetation script:
HailoRT Multi Process Service and Model Scheduler

My problem is regarding how the context switch is working in the script, we have these 2 main function:

  1. def create_vdevice_and_infer(hef_path):
  2. def infer(network_group, input_vstreams_params, output_vstreams_params, input_data):

the “create_vdevice_and_infer” function, which is in a python file called “infer_multi_model.py” sets up the models and call the “infer” which is function that was in the same “infer_multi_model.py” file, runs the inferences. All of this at firts happened in the same python file.

The problem i have, started when I tried to put the infer function on a different file and run the main script, i get this annoying error:

(hrt_env) admin@raspberrypi:~/hrt/hailo_multi $ python infer_multi_model.py
OPERATION START, TIME STARTED: 1747664390.406578

  • Serving Flask app ‘infer_multi_model’
  • Debug mode: off
    2025-05-19 15:19:51,666 - INFO - WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
  • Running on all addresses (0.0.0.0)
  • Running on http://127.0.0.1:5001
    2025-05-19 15:19:51,667 - INFO - Press CTRL+C to quit
    Exception in thread Thread-1:
    Traceback (most recent call last):
    File “/home/admin/.pyenv/versions/3.8.20/lib/python3.8/threading.py”, line 932, in _bootstrap_inner
    self.run()
    File “/home/admin/.pyenv/versions/3.8.20/lib/python3.8/threading.py”, line 870, in run
    self._target(*self._args, **self._kwargs)
    File “infer_multi_model.py”, line 272, in run_stream_watcher
    streaming_frame(hefs)
    File “infer_multi_model.py”, line 94, in streaming_frame
    cropped_roi = infer_crop(infer_params[0], infer_params[2], infer_params[3], input_data, img, prep_img)
    File “/home/admin/hrt/hailo_multi/utils.py”, line 67, in infer_crop
    with InferVStreams(network_group, input_vstreams_params, output_vstreams_params) as infer_pipeline:
    File “/home/admin/hrt_env/lib/python3.8/site-packages/hailo_platform/pyhailort/pyhailort.py”, line 879, in init
    self._net_group_name = configured_net_group.name
    File “/home/admin/hrt_env/lib/python3.8/site-packages/hailo_platform/pyhailort/pyhailort.py”, line 606, in name
    return self._configured_network.get_name()
    hailo_platform.pyhailort._pyhailort.HailoRTStatusException: 8
    ^C2025-05-19 15:21:40,830 - INFO - Shutdown complete

but when i run both the infer and create_vdevice_and_infer functions on the python file, it all runs well. the infer function gets called but when reaches the context manager line it wont go through, this is the infer function as per the documentation:

# Define the function to run inference on the model
def infer(network_group, input_vstreams_params, output_vstreams_params, input_data):
    rep_count = 100
    with InferVStreams(network_group, input_vstreams_params, output_vstreams_params) as infer_pipeline:
        for i in range(rep_count):
            infer_results = infer_pipeline.infer(input_data)

if i put a print after rep_count=100, it will print, but after or inside the context manager it wont.
I would i appreciate some help on this. Thank you for reading.

2

Hey @Claver_Barreto ,

This error happens because ConfiguredNetworkGroup and InferVStreams aren’t safe to pass between threads or module boundaries. When you move the infer() function to another file and call it from infer_multi_model.py, the network_group object might look valid in Python, but it’s actually lost context at the C++ backend — leading to HailoRTStatusException: 8 (internal failure).

:white_check_mark: Fix It Like This:

Keep network_group creation and usage in the same file and thread.
Either:

  • Don’t move infer() to another file, or
  • Let infer() handle network_group creation internally — don’t pass it in.

:test_tube: Quick Test:

If you move everything (VDevice, network group, streams, inference) into one place — even temporarily — and it works, that confirms it’s a scope/lifetime issue.

Also, if you’re using Model Scheduler or Multi-Process Service, make sure VDeviceParams(scheduler=True, multi_process_service=True) is set.

Hope this fixes the issue!