Inquiry About Obtaining Tensor Values from a Specific Layer Using HailoRT C API

General
1

Hello,

I am a developer working on implementing model inference using Hailo NPU.

The NPU model I am testing is the stdc1.hef file, and the output layer name for this model is output_layer1. Since the tensor values in output_layer1 have already been processed through an argmax layer, I am unable to handle lower-confidence classes properly.

Is there a way to specify an output layer using a specific layer name with the HailoRT C API and obtain the corresponding tensor values after inference?

For example, before output_layer1, there is a layer called resize7, and I would like to retrieve the tensor values corresponding to the resize7 layer. If you have an example code snippet, it would be extremely helpful.

Thank you.

2

Hi @Inki_Dae
Welcome to the Hailo community. Is this a custom model you compiled to .hef? If so, during compilation you can specify the end_node name as the layer you want and directly work with that output. For example, you can run your own argmax that provides the additional information you need to handle lower confidence classes.

3

Hi @shashi,

As you suggested, I downloaded the stdc1-sim.onnx model file from the Hailo Model Zoo and tried to compile it. However, I encountered the following error. The error message instructs me to refer to the user guide. Could you please provide guidance on this?

Thank you for your response.

(virtualenv) daeinki@linux:~/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo$ hailo optimize stdc1-sim.har --use-random-calib-set
[info] Current Time: 09:15:51, 03/05/25
[info] CPU: Architecture: x86_64, Model: 11th Gen Intel(R) Core™ i7-11700 @ 2.50GHz, Number Of Cores: 16, Utilization: 0.6%
[info] Memory: Total: 31GB, Available: 22GB
[info] System info: OS: Linux, Kernel: 5.15.0-131-generic
[info] Hailo DFC Version: 3.29.0
[info] HailoRT Version: Not Installed
[info] PCIe: No Hailo PCIe device was found
[info] Running hailo optimize stdc1-sim.har --use-random-calib-set
[info] Found model with 3 input channels, using real RGB images for calibration instead of sampling random data.
[info] Starting Model Optimization
[warning] Reducing optimization level to 0 (the accuracy won’t be optimized and compression won’t be used) because there’s no available GPU
[warning] Running model optimization with zero level of optimization is not recommended for production use and might lead to suboptimal accuracy results
[info] Model received quantization params from the hn
[info] Starting Mixed Precision
[info] Mixed Precision is done (completion time is 00:00:00.37)
[info] LayerNorm Decomposition skipped
[info] Starting Statistics Collector
[info] Using dataset with 64 entries for calibration
Calibration: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 64/64 [01:17<00:00, 1.22s/entries]
[info] Statistics Collector is done (completion time is 00:01:18.74)
[info] Starting Fix zp_comp Encoding
[info] Fix zp_comp Encoding is done (completion time is 00:00:00.00)
[info] Matmul Equalization skipped
[info] No shifts available for layer stdc1-sim/conv30/conv_op, using max shift instead. delta=0.0568
[info] No shifts available for layer stdc1-sim/conv30/conv_op, using max shift instead. delta=0.0284
Traceback (most recent call last):
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/bin/hailo”, line 8, in
sys.exit(main())
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_sdk_client/tools/cmd_utils/main.py”, line 111, in main
ret_val = client_command_runner.run()
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_sdk_client/tools/cmd_utils/base_utils.py”, line 68, in run
return self._run(argv)
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_sdk_client/tools/cmd_utils/base_utils.py”, line 89, in _run
return args.func(args)
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_sdk_client/tools/optimize_cli.py”, line 120, in run
self._runner.optimize(dataset, work_dir=args.work_dir)
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_sdk_common/states/states.py”, line 16, in wrapped_func
return func(self, args, kwargs)
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_sdk_client/runner/client_runner.py”, line 2093, in optimize
self._optimize(calib_data, data_type=data_type, work_dir=work_dir)
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_sdk_common/states/states.py”, line 16, in wrapped_func
return func(self, *args, kwargs)
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_sdk_client/runner/client_runner.py”, line 1935, in _optimize
self._sdk_backend.full_quantization(calib_data, data_type=data_type, work_dir=work_dir)
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py”, line 1045, in full_quantization
self._full_acceleras_run(self.calibration_data, data_type)
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py”, line 1229, in _full_acceleras_run
optimization_flow.run()
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_model_optimization/tools/orchestator.py”, line 306, in wrapper
return func(self, args, *kwargs)
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 326, in run
step_func()
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(args, kwargs)
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_model_optimization/tools/subprocess_wrapper.py”, line 124, in parent_wrapper
func(self, *args, kwargs)
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 345, in step1
self.core_quantization()
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(args, **kwargs)
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 403, in core_quantization
self._create_hw_params()
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(args, **kwargs)
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 451, in _create_hw_params
create_hw_params.run()
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_model_optimization/algorithms/optimization_algorithm.py”, line 50, in run
return super().run()
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_model_optimization/algorithms/algorithm_base.py”, line 150, in run
self._run_int()
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_model_optimization/algorithms/create_hw_params/create_hw_params.py”, line 340, in _run_int
comp_to_retry = self._create_hw_params_component(matching_component_group)
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_model_optimization/algorithms/create_hw_params/create_hw_params.py”, line 204, in _create_hw_params_component
layer.create_hw_params(layer_clip_cfg, hw_shifts=hw_shifts)
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_model_optimization/acceleras/hailo_layers/hailo_avgpool_v2.py”, line 301, in create_hw_params
self.avgpool_op.create_hw_params(max_final_accumulator_by_channel, hw_shifts=hw_shifts)
File “/home/daeinki/project/s5pc210/public/tizen_6.0/github/hailo/hailo_model_zoo/virtualenv/lib/python3.8/site-packages/hailo_model_optimization/acceleras/atomic_ops/avgpool_op.py”, line 104, in create_hw_params
raise AccelerasNumerizationError(
hailo_model_optimization.acceleras.utils.acceleras_exceptions.AccelerasNumerizationError: Shift delta in stdc1-sim/avgpool3/avgpool_op is larger than 2 (2.47), cannot quantize. A possible solution is to use a pre-quantization model script command to reduce global average-pool spatial dimensions, please refer to the user guide for more info.

4

Hi @Inki_Dae
So, this is a model from the model zoo itself. Before digging into compiling the model (as it requires looking at more settings), can you explain what your issue is with the original model output? You mentioned last layer being argmax preventing you from handling low confidence classes. Can you please elaborate on this? Is your final segmentation mask very far from expected output?

5

Hi @shashi,

The output value obtained through the Argmax layer is the index value of the class with the highest confidence for each pixel. The problem is that the Argmax layer outputs the index value of the class with the highest confidence among 19 classes, so even if the confidence is low, it is reflected in the final result. As a result, there are cases where each pixel contains an inaccurate class id. If it is possible to use the resize7 layer, which is the layer before the argmax layer, as the final output value, I would like to consider that detection does not occur if the value with the highest confidence among 19 classes does not exceed a given threshold value. In this case, output value of the pixel is 255.

For example, in pseudo code (threshold = 150):

if (max(resize7[h][x]) > 150)
    output[h][x] = class_index
else
    output[h][x] = 255

Thank you for your response.

6

Hi @Inki_Dae
You can take a look at the alls file that you need to use to compile: hailo_model_zoo/hailo_model_zoo/cfg/alls/generic/stdc1.alls at master · hailo-ai/hailo_model_zoo

7

Hi @shashi

I was able to compile the output layer by specifying it using the model script file you shared. However, there are multiple stdc1.alls files, and I would like to inquire about which file to use for Hailo-8L. Which model script should I use when compiling for Hailo-8L NPU: hailo8/stdc1.alls or generic/stdc1.alls?

I tested both files, and in both cases, I had to modify the model script instead of using it as is. In my case, I made the following modifications:

normalization1 = normalization([123.675, 116.28, 103.53], [58.395, 57.12, 57.375]) ← removed stdc1 prefix pre_quantization_optimization(global_avgpool_reduction, layers=avgpool3, division_factors=[32, 60]) ← added new line due to shift delta issue

Thanks for your response.

8

@Inki_Dae
Great to hear you were able to compile. Let us hope the output is as you expected.

9

Hi @shashi,

The conversion was successful, but the accuracy is too low. It seems that the model I compiled may not be normal. Could you share the model script file and commands you used when converting the stdc1-sim.onnx model file to the model one for Hailo-8L NPU?

The model script file I used is as follows: hailo_model_zoo/cfg/alls/generic/stdc1.alls

normalization1 = normalization([123.675, 116.28, 103.53], [58.395, 57.12, 57.375])
model_optimization_config(calibration, batch_size=1, calibset_size=64)
pre_quantization_optimization(global_avgpool_reduction, layers=avgpool3, division_factors=[32, 60])
pre_quantization_optimization(global_avgpool_reduction, layers=avgpool7, division_factors=[128, 1])
post_quantization_optimization(finetune, policy=enabled, dataset_size=512, epochs=6, loss_layer_names=[conv33], loss_types=[l2])
allocator_param(timeout=25m)

The commands I used during compilation are as follows:
#hailo parser onnx stdc1-sim.onnx --start-node-name input --end-node-names Resize_148 --hw-arch hailo8l
#hailo optimize --hw-arch hailo8l --use-random-calib-set --model-script stdc1.alls stdc1-sim.har
#hailo compiler stdc1-sim_optimized.har

Thanks for your response.

10

Hi @Inki_Dae
We are using the pre-compiled model from the zoo. We did not compile this model. Perhaps someone from the Hailo team can help.