Logging device metrics of Hailo 8L on raspberry pi

General
1

Hello there.

I am trying to get some performance metrics of the Hailo 8L chip on how many models can be run simultaneously. I can get metrics using hailortcli monitor but cannot save it to a log file on a continious mode. I need these logs to plot a graphical log to analyse the device capabilities.

Python api supported solution would be good.
Thanks and Regards.

2

Hey @hyperwolf

To continuously monitor and log the performance metrics of the Hailo-8L chip, you can use the HailoRT CLI along with a Python script to save the output to a log file. The script utilizes the subprocess module to run the HailoRT CLI commands and capture the output in real time.

Here’s a Python-based solution:

import subprocess
import time

def run_hailort_monitor(device_id, log_file_path):
    # Enable monitoring by setting the HAILO_MONITOR environment variable
    env_vars = {"HAILO_MONITOR": "1"}
    
    # Command to run the model with the specified device ID
    run_command = ["hailortcli", "run", "model.hef", "--device-id", device_id]
    
    # Command to start the monitor
    monitor_command = ["hailortcli", "monitor"]
    
    # Open the log file for writing
    with open(log_file_path, "w") as log_file:
        # Start the monitor process
        monitor_process = subprocess.Popen(monitor_command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, env=env_vars)
        
        # Continuously run the model and capture data
        while True:
            run_process = subprocess.Popen(run_command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, env=env_vars)
            # Wait for the command to complete and capture the output
            output, errors = run_process.communicate()
            
            # Write the output to the log file
            log_file.write(output.decode())
            log_file.flush()
            
            # Pause before the next run
            time.sleep(1)  # Adjust sleep time as needed

            # Check if the monitor process is still running
            if monitor_process.poll() is not None:
                break  # Exit if the monitor process has stopped

if __name__ == "__main__":
    device_id = "0000:nn:00.0"  # Replace with your actual device ID
    log_file_path = "hailo_performance_log.txt"
    run_hailort_monitor(device_id, log_file_path)

Explanation:

  • Environment Variable: The script sets HAILO_MONITOR=1 to enable performance monitoring.
  • Running the Model: The model is run using hailortcli run with the specified device ID in a loop to generate continuous data.
  • Monitoring: The hailortcli monitor command is started in a subprocess to capture performance metrics.
  • Logging: The output from the model run is continuously written to a log file.
  • Loop and Sleep: The script loops with a sleep interval to periodically run the model, capturing performance data over time.

Make sure to replace "model.hef" with your actual HEF file and update the device_id to match your setup. Adjust the sleep time according to how frequently you want to capture data.

Regards