Currently, I want to deploy a fine-tuned CLIP model on RPI5 + Hailo8. Before that, I am testing how to modify the hef file using the model_zoo.
In the existing demo, the model_zoo’s resnet50x4 HEF file (input 288x288) was used, but now I want to try using the vit or tiny_vit HEF files (input 224x224). I put the new HEF file path in clip_application.py and changed the output layer name in clip.cpp, but I was not successful in running it. Also, since the vit series HEF files have an input size of 224x224, the input image resizing inside the code must be adjusted, and I need guidance on how to modify this part. I am wondering if just replacing the HEF file would automatically adjust everything, and I would appreciate advice on my current method.
It would be very helpful if you could teach me how to utilize the model_zoo hef file for the clip model.
When you update the HEF file, make sure to update the pipeline files to match the new inputs and outputs. You can find the relevant pipeline code here:
You’ll also need to add your own post-processing logic to these files. If you want to see an example of how post-processing is implemented, check out the C++ version here: