Model degradation - How to quantify

General
1

Hei sorry the basic question, but how can I evaluate or interpretate the degradtion after optimization of my model?

  • I trained a model and then transform it to hef following mainly (this guide
  • With both models in a RPI, in a random image i get 6 detections with the original model (pytorch) meanwhile i get no detections with the hef model (in the same image.)
  • hailortcli run show 52 FPS meanwhile pytorch within python only 0.8, so im quite happy about that…

Any suggetions are welcome! thanks :slight_smile:

2

Hi @Camilo_Chiang
If it is an object detection model, you can evaluate the mAP of the model in pytorch and after porting it to Hailo device. Since you are saying that you are getting 0 detections with hef model, it is worth checking if you are interpreting the model output correctly. We can help you if you provide more details on the model you are using.

3

Hei, thanks for your comment!

There is no too much space for interpretation for the output (maybe I did something wrong in the pipeline before…). Im using the picamera2 implementation and the return array is empty…

from picamera2.devices import Hailo
.....
	i = 0
	with Hailo('../../model.hef') as hailo:
		while i < 10:
				print(hailo.get_input_shape())
				tmp = hailo.run(img)
				print(tmp)
				i += 1

gives

(1024, 1024, 3)
[array([], shape=(0, 5), dtype=float64)]