Is there a way to use SAM 2 model just released by Meta in RPI + Hailo8L ?
https://github.com/facebookresearch/segment-anything-2
Hi @dario.ravarro,
SAM 2 is currently not formally supported. You can export it to ONNX or tflite and check using the Hailo Dataflow Compiler tool if it’s supported by the SW.
In any case, we have a fast_sam supported model You can check it’s usage in this Python example:
Regards,
Hi, I was trying fast_sam but unfortunately it doesn’t run in RPI5 + Hail8l.
I am actually trying to re-compile it using DFC 3.27, but facing a few issues.
I took the onnx file of fast_sam and was able to create the .har file using CLI. I couldn’t create the har file using the jupyer notebook tutorial as this part of the code
runner = ClientRunner(hw_arch=chosen_hw_arch)
hn, npz = runner.translate_onnx_model(
onnx_path,
onnx_model_name,
)
gave some errors
ParsingWithRecommendationException Traceback (most recent call last)
File ~/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:220, in Parser.translate_onnx_model(self, model, net_name, start_node_names, end_node_names, net_input_shapes, augmented_path, disable_shape_inference, disable_rt_metadata_extraction, net_input_format, **kwargs)
219 try:
→ 220 parsing_results = self._parse_onnx_model_to_hn(
221 onnx_model=onnx_model,
222 net_name=valid_net_name,
223 start_node_names=start_node_names,
224 end_node_names=end_node_names,
225 net_input_shapes=net_input_shapes,
226 disable_shape_inference=disable_shape_inference,
227 net_input_format=net_input_format,
228 )
230 except Exception as e:
File ~/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:300, in Parser._parse_onnx_model_to_hn(self, onnx_model, net_name, start_node_names, end_node_names, net_input_shapes, disable_shape_inference, net_input_format, **kwargs)
298 self._logger.warning(f"ONNX shape inference failed: {e!s}")
→ 300 return self.parse_model_to_hn(
301 onnx_model,
302 None,
303 net_name,
304 start_node_names,
305 end_node_names,
306 nn_framework=NNFramework.ONNX,
307 output_shapes=output_shapes,
308 net_input_format=net_input_format,
309 **kwargs,
310 )
File ~/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:351, in Parser.parse_model_to_hn(self, model, values, net_name, start_node_names, end_node_names, nn_framework, output_shapes, net_input_format, rename_layers_by_blocks)
349 raise BackendRuntimeException(f"Unsupported NN framework {nn_framework}")
→ 351 fuser = HailoNNFuser(converter.convert_model(), net_name, converter.end_node_names)
352 hailo_nn = fuser.convert_model()
File ~/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/model_translator/translator.py:79, in HailoNNConverter.convert_model(self)
78 self._validate_bn_ops_in_training()
—> 79 self._create_layers()
80 self._add_layers_connections()
File ~/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py:34, in EdgeNNConverter._create_layers(self)
33 self._update_vertices_info()
—> 34 self._add_direct_layers()
35 self._validate_processed_vertices()
File ~/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py:187, in EdgeNNConverter._add_direct_layers(self)
186 if start_suggestions or end_suggestions:
→ 187 raise ParsingWithRecommendationException(
188 suggestions_msg,
189 recommended_start_node_names=list(set(start_suggestions)),
190 recommended_end_node_names=list(set(end_suggestions)),
191 parsing_report=self.get_parsing_report(from_error=True),
192 )
ParsingWithRecommendationException: Parsing failed. The errors found in the graph are:
UnsupportedSliceLayerError in op Slice_316: Failed to create slice layer at vertex Slice_316. Slice on axis 1 is not supported
UnsupportedSliceLayerError in op Slice_319: Failed to create slice layer at vertex Slice_319. Slice on axis 1 is not supported
UnsupportedModelError in op Sub_321: In vertex Sub_321_input the contstant value shape (1, 2, 8400) must be broadcastable to the output shape [1, 8400, 2]
UnsupportedModelError in op Add_323: In vertex Add_323_input the contstant value shape (1, 2, 8400) must be broadcastable to the output shape [1, 8400, 2]
Please try to parse the model again, using these end node names: Reshape_305, Mul_184, Mul_318, Concat_221, Mul_315, Sigmoid_331
During handling of the above exception, another exception occurred:
ParsingWithRecommendationException Traceback (most recent call last)
Cell In[8], line 2
1 runner = ClientRunner(hw_arch=chosen_hw_arch)
----> 2 hn, npz = runner.translate_onnx_model(
3 onnx_path,
4 onnx_model_name,
5 )
File ~/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_common/states/states.py:16, in allowed_states..wrap..wrapped_func(self, *args, **kwargs)
12 if self._state not in states:
13 raise InvalidStateException(
14 f"The execution of {func.name} is not available under the state: {self._state.value}",
15 )
—> 16 return func(self, *args, **kwargs)
File ~/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py:1158, in ClientRunner.translate_onnx_model(self, model, net_name, start_node_names, end_node_names, net_input_shapes, augmented_path, disable_shape_inference, disable_rt_metadata_extraction, net_input_format, **kwargs)
1115 “”"
1116 DFC API for parsing an ONNX model. This creates a runner with loaded HN (model) and
1117 parameters.
(…)
1155
1156 “”"
1157 parser = Parser()
→ 1158 parser.translate_onnx_model(
1159 model=model,
1160 net_name=net_name,
1161 start_node_names=start_node_names,
1162 end_node_names=end_node_names,
1163 net_input_shapes=net_input_shapes,
1164 augmented_path=augmented_path,
1165 disable_shape_inference=disable_shape_inference,
1166 disable_rt_metadata_extraction=disable_rt_metadata_extraction,
1167 net_input_format=net_input_format,
1168 **kwargs,
1169 )
1171 return self._finalize_parsing(parser.return_data)
File ~/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:260, in Parser.translate_onnx_model(self, model, net_name, start_node_names, end_node_names, net_input_shapes, augmented_path, disable_shape_inference, disable_rt_metadata_extraction, net_input_format, **kwargs)
257 milestone = self._format_time_milestone(start_time)
258 self._logger.info(f"Simplified ONNX model for a parsing retry attempt (completion time: {milestone})“)
→ 260 parsing_results = self._parse_onnx_model_to_hn(
261 onnx_model=simplified_model,
262 net_name=valid_net_name,
263 start_node_names=start_node_names,
264 end_node_names=end_node_names,
265 net_input_shapes=net_input_shapes,
266 disable_shape_inference=disable_shape_inference,
267 net_input_format=net_input_format,
268 **kwargs,
269 )
271 milestone = self._format_time_milestone(start_time)
272 self._logger.info(f"Translation completed on ONNX model {valid_net_name} (completion time: {milestone})”)
File ~/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:300, in Parser._parse_onnx_model_to_hn(self, onnx_model, net_name, start_node_names, end_node_names, net_input_shapes, disable_shape_inference, net_input_format, **kwargs)
297 except Exception as e:
298 self._logger.warning(f"ONNX shape inference failed: {e!s}")
→ 300 return self.parse_model_to_hn(
301 onnx_model,
302 None,
303 net_name,
304 start_node_names,
305 end_node_names,
306 nn_framework=NNFramework.ONNX,
307 output_shapes=output_shapes,
308 net_input_format=net_input_format,
309 **kwargs,
310 )
File ~/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:351, in Parser.parse_model_to_hn(self, model, values, net_name, start_node_names, end_node_names, nn_framework, output_shapes, net_input_format, rename_layers_by_blocks)
348 else:
349 raise BackendRuntimeException(f"Unsupported NN framework {nn_framework}")
→ 351 fuser = HailoNNFuser(converter.convert_model(), net_name, converter.end_node_names)
352 hailo_nn = fuser.convert_model()
353 hailo_nn.validate_stage(HnStage.HN)
File ~/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/model_translator/translator.py:79, in HailoNNConverter.convert_model(self)
77 self._validate_model_params()
78 self._validate_bn_ops_in_training()
—> 79 self._create_layers()
80 self._add_layers_connections()
81 self._layers_graph.set_names_and_indices()
File ~/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py:34, in EdgeNNConverter._create_layers(self)
32 self._add_input_layers()
33 self._update_vertices_info()
—> 34 self._add_direct_layers()
35 self._validate_processed_vertices()
File ~/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py:187, in EdgeNNConverter._add_direct_layers(self)
185 suggestions_msg = “More possible end nodes found, consider using them for more precise parsing”
186 if start_suggestions or end_suggestions:
→ 187 raise ParsingWithRecommendationException(
188 suggestions_msg,
189 recommended_start_node_names=list(set(start_suggestions)),
190 recommended_end_node_names=list(set(end_suggestions)),
191 parsing_report=self.get_parsing_report(from_error=True),
192 )
ParsingWithRecommendationException: Parsing failed. The errors found in the graph are:
UnsupportedSliceLayerError in op Slice_316: Failed to create slice layer at vertex Slice_316. Slice on axis 1 is not supported
UnsupportedSliceLayerError in op Slice_319: Failed to create slice layer at vertex Slice_319. Slice on axis 1 is not supported
UnsupportedModelError in op Sub_321: In vertex Sub_321_input the contstant value shape (1, 2, 8400) must be broadcastable to the output shape [1, 8400, 2]
UnsupportedModelError in op Add_323: In vertex Add_323_input the contstant value shape (1, 2, 8400) must be broadcastable to the output shape [1, 8400, 2]
Please try to parse the model again, using these end node names: Mul_184, Sigmoid_331, Concat_221, Reshape_305
update on this:
- downloaded hailo_model_zoo
- hailo optimize fast_sam_s.har --hw-arch hailo8l --use-random-calib-set --model-script hailo_model_zoo-2.12/hailo_model_zoo/cfg/alls/generic/fast_sam_s.alls
the optimization results are
[info] Current Time: 14:03:28, 08/19/24
[info] CPU: Architecture: x86_64, Model: 12th Gen Intel(R) Core™ i7-12700H, Number Of Cores: 8, Utilization: 0.0%
[info] Memory: Total: 11GB, Available: 8GB
[info] System info: OS: Linux, Kernel: 5.15.153.1-microsoft-standard-WSL2
[info] Hailo DFC Version: 3.28.0
[info] HailoRT Version: Not Installed
[info] PCIe: No Hailo PCIe device was found
[info] Running hailo optimize fast_sam_s.har --hw-arch hailo8l --use-random-calib-set --model-script hailo_model_zoo-2.12/hailo_model_zoo/cfg/alls/generic/fast_sam_s.alls
[info] Loading model script commands to fast_sam_s from hailo_model_zoo-2.12/hailo_model_zoo/cfg/alls/generic/fast_sam_s.alls
[info] Found model with 3 input channels, using real RGB images for calibration instead of sampling random data.
[info] Starting Model Optimization
[warning] Reducing optimization level to 0 (the accuracy won’t be optimized and compression won’t be used) because there’s no available GPU
[warning] Running model optimization with zero level of optimization is not recommended for production use and might lead to suboptimal accuracy results
[info] Model received quantization params from the hn
[info] Starting Mixed Precision
[info] Mixed Precision is done (completion time is 00:00:00.46)
[info] Layer Norm Decomposition skipped
[info] Starting Stats Collector
[info] Using dataset with 64 entries for calibration
Calibration: 100%| 64/64 [00:31<00:00, 2.03entries/s]
[info] Stats Collector is done (completion time is 00:00:32.75)
[info] Starting Fix zp_comp Encoding
[info] Fix zp_comp Encoding is done (completion time is 00:00:00.00)
[info] matmul_equalization skipped
[info] No shifts available for layer fast_sam_s/conv1/conv_op, using max shift instead. delta=1.3022
[info] No shifts available for layer fast_sam_s/conv1/conv_op, using max shift instead. delta=0.6511
[info] No shifts available for layer fast_sam_s/conv47/conv_op, using max shift instead. delta=0.6172
[info] No shifts available for layer fast_sam_s/conv47/conv_op, using max shift instead. delta=0.3086
[info] activation fitting started for fast_sam_s/ew_sub_softmax1/act_op
[info] activation fitting started for fast_sam_s/reduce_sum_softmax1/act_op
[info] Finetune encoding skipped
[info] Bias Correction skipped
[info] Adaround skipped
[info] Fine Tune skipped
[info] Layer Noise Analysis skipped
[warning] The expected calibration set should be [0, 255] when using an in-net normalization layer, but the range received is [(0.0, 1.0), (0.0, 1.0), (0.0, 1.0)].
[info] Model Optimization is done
[info] Saved HAR to: /home/dario/hailo/fast_sam_s_optimized.har
Then launched compilation
hailo compiler fast_sam_s_optimized.har --hw-arch hailo8l --model-script hailo_model_zoo-2.12/hailo_model_zoo/cfg/alls/generic/fast_sam_s.alls
output of the compilation
[info] Current Time: 14:05:49, 08/19/24
[info] CPU: Architecture: x86_64, Model: 12th Gen Intel(R) Core™ i7-12700H, Number Of Cores: 8, Utilization: 0.0%
[info] Memory: Total: 11GB, Available: 8GB
[info] System info: OS: Linux, Kernel: 5.15.153.1-microsoft-standard-WSL2
[info] Hailo DFC Version: 3.28.0
[info] HailoRT Version: Not Installed
[info] PCIe: No Hailo PCIe device was found
[info] Running hailo compiler fast_sam_s_optimized.har --hw-arch hailo8l --model-script hailo_model_zoo-2.12/hailo_model_zoo/cfg/alls/generic/fast_sam_s.alls
[info] Loading model script commands to fast_sam_s from hailo_model_zoo-2.12/hailo_model_zoo/cfg/alls/generic/fast_sam_s.alls
[info] Compiling network
[info] To achieve optimal performance, set the compiler_optimization_level to “max” by adding performance_param(compiler_optimization_level=max) to the model script. Note that this may increase compilation time.
[info] Loading network parameters
[info] Starting Hailo allocation and compilation flow
[error] Mapping Failed (allocation time: 11s)
No successful assignment for: format_conversion2, concat18, feature_splitter9, shortcut_softmax1, reduce_max_softmax1, ew_sub_softmax1, reduce_sum_softmax1, ew_mult_softmax1, conv76
[error] Failed to produce compiled graph
[error] BackendAllocatorException: Compilation failed: No successful assignment for: format_conversion2, concat18, feature_splitter9, shortcut_softmax1, reduce_max_softmax1, ew_sub_softmax1, reduce_sum_softmax1, ew_mult_softmax1, conv76
Hi @dario.ravarro,
If you use the alls file in the optimization step, it will apply in the compilation step too. No need to apply it twice.
Does this error happens if you compile the model using the Hailo Model Zoo? meaning running hailomz compile fast_sam_s --hw-arch hailo8l?
Regards,
hailomz compile fast_sam_s --hw-arch hailo8l
In file included from /home/dario/miniconda3/envs/hailo/lib/python3.10/site-packages/numpy/core/include/numpy/ndarraytypes.h:1948,
from /home/dario/miniconda3/envs/hailo/lib/python3.10/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /home/dario/miniconda3/envs/hailo/lib/python3.10/site-packages/numpy/core/include/numpy/arrayobject.h:5,
from /home/dario/.pyxbld/temp.linux-x86_64-cpython-310/home/dario/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_zoo/core/postprocessing/cython_utils/cython_nms.c:1246:
/home/dario/miniconda3/envs/hailo/lib/python3.10/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " “#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION” [-Wcpp]
17 | #warning "Using deprecated NumPy API, disable it with "
| ^~~~~~~
Start run for network fast_sam_s …
Initializing the hailo8l runner…
fast_sam_s.zip: 100%|██████████████████████████████████████████████████████| 38.1M/38.1M [00:27<00:00, 1.46MB/s]
[info] Translation started on ONNX model fast_sam_s
[info] Restored ONNX model fast_sam_s (completion time: 00:00:00.13)
[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:01.01)
[info] Start nodes mapped from original model: ‘images’: ‘fast_sam_s/input_layer1’.
[info] End nodes mapped from original model: ‘Conv_261’, ‘Conv_268’, ‘Conv_216’, ‘Conv_246’, ‘Conv_253’, ‘Conv_205’, ‘Conv_231’, ‘Conv_238’, ‘Conv_194’, ‘Mul_184’.
[info] Translation completed on ONNX model fast_sam_s (completion time: 00:00:01.90)
[info] Saved HAR to: /home/dario/hailo/fast_sam_s.har
Traceback (most recent call last):
File “/home/dario/miniconda3/envs/hailo/bin/hailomz”, line 8, in
sys.exit(main())
File “/home/dario/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_zoo/main.py”, line 122, in main
run(args)
File “/home/dario/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_zoo/main.py”, line 111, in run
return handlersargs.command
File “/home/dario/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_zoo/main_driver.py”, line 250, in compile
_ensure_optimized(runner, logger, args, network_info)
File “/home/dario/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_zoo/main_driver.py”, line 84, in _ensure_optimized
model_script = _extract_model_script_path(
File “/home/dario/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_zoo/main_driver.py”, line 55, in _extract_model_script_path
else resolve_alls_path(networks_alls_script, hw_arch=hw_arch, performance=performance)
File “/home/dario/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_zoo/core/main_utils.py”, line 38, in resolve_alls_path
return path_resolver.resolve_alls_path(Path(hw_arch) / Path(“base” if not performance else “performance”) / path)
File “/home/dario/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_zoo/utils/path_resolver.py”, line 29, in resolve_alls_path
raise FileNotFoundError(“no alls found for requested hw_arch”)
Hi @dario.ravarro,
I was able to compile it successfully with the command you used in Hailo Model Zoo 2.12. What version are you using? it seems that the issue here is that it looks for the alls file under hailo_model_zoo/cfg/alls/hailo8l/ but doesn’t find it. I believe that it’s just a matter of defining the path to it.
Regards,
Hi
I have tried to convert SAM2 to onnx, simplified it and parse with Dataflow Compiler Tool. But it shows full of UnsupportedShuffleLayerError. Example below
UnsupportedShuffleLayerError in op /image_encoder/trunk/patch_embed/Transpose: Failed to determine type of layer to create in node /image_encoder/trunk/patch_embed/Transpose
UnsupportedModelError in op /image_encoder/trunk/Add_1: In vertex /image_encoder/trunk/Add_1_input the contstant value shape (256, 96, 256) must be broadcastable to the output shape [256, 256, 96]
UnsupportedShuffleLayerError in op /image_encoder/trunk/blocks.0/Reshape: Failed to determine type of layer to create in node /image_encoder/trunk/blocks.0/Reshape
It recommends to end with /image_encoder/trunk/patch_embed/proj/Conv. However, this node is the second node of my onnx model.
Hi @junsen.yee,
Yes, as I thought since this model is not currently supported by Hailo, the Parsing process fails on unsupported layers. The recommendation here, as you saw, doesn’t help to actually use the model.
Regards,
Thanks for the information. May I know is there any upcoming plan to make Hailo support on SAM2?
Hi @junsen.yee,
I don’t think there’s a current decision to work on supporting it, but I positive that it will be supported in the future.
Regards,
Try to compile with:
hailo compiler fast_sam_s_optimized.har --hw-arch hailo8l --model-script “resources_param(max_utilization=0.5)”
It will lower the utilization, hopefully producing a working result.
Another preferred option is to compile using “performance flow”, which tries many different utilization levels until finding the highest possible one.
hailo compiler fast_sam_s_optimized.har --hw-arch hailo8l --model-script “performance_param(compiler_optimization_level=max)”
note: it will take a lot of time to complete.
if it doesnt work, try uploading the hailo_sdk.core.log and it will help us determine what the problem is.
1 Like
Hi, I am currently trying fastSAM. Is there any tutorial on how can i used the point prompt feature in fastSAM? I am currently using the fastSAM HEF from modelzoo
I also tried to parse the FASTSAM onnx i downloaded from model zoo to HAR.
It shows the error of this:
Parsing failed with recommendations for end node names: [‘Reshape_305’, ‘Mul_184’, ‘Sigmoid_331’, ‘Concat_221’].
so are you able to compile with DFC? Or you use the model from model zoo?
I will try to use the one from model zoo after I upgrade the system with new package 4.18
After installing latest hailo-all succesfully (with hailort 4.18), and downloaded fast_sam_s.hef from modelzoo, any examples on how to test it on rpi camera?
Sure thing! You can find some great examples to get you started right here: Get started with picamera2 examples
you mean that I follow an example like detect.py that loads yolov8s_h8l.hef and I make a new code that loads fast_sam_s.hef ? but how to post process ?