Hailo 8 not work in Windows 10 + PyTorch

General
1

Hello! I’m a beginner. I received my modules only yesterday. I installed the module in the computer. Installed the drivers.The drivers have been installed successfully. In system i see device -“Hailo-8 AI Accelerator”. I install transformers torch and write little test script. But python script does not see the accelerator. The script runs as if there is no accelerator. What am I doing wrong?

1 Like
2

Hi @hypertechlab,
Did you install the HailoRT driver & lib for Windows from the Hailo website?

Regards,

3

Yes. I install hailort_4.18.0_windows_installer. Hailo module is working. I can run hailortcli.exe etc. In system i see device -“Hailo-8 AI Accelerator”. But python scripts no see the device. Maybe it is necessary to add some additional libraries to the code? Or do I need to install something else?

4

Hi @hypertechlab,
What is the script that you are using? it’s difficult for me to know the source of the issue without it

Regards,

5

The script - Hello world for AI…
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
model_name = “meta-llama/Meta-Llama-3-8B”
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
input_text = “Hello. How much is two plus two?”
inputs = tokenizer(input_text, return_tensors=“pt”)
outputs = model.generate(**inputs)
decoded_output = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(decoded_output)

6

Hi @hypertechlab,
There is no usage of the Hailo API here, which means that there is no usage in the Hailo accelerator.
Please refer to the Dataflow Compiler user guide and the HailoRT user guide to understand better the usage of the Hailo chip.

You can also try running our pre-made examples here:

But please notice that not all of them are relevant for Windows (most Python examples should work and one of the C++ examples).

Regards,

7

I have a short question. You can use halo with PyTorch? Or is interaction with halo possible only with other libraries?

8

Hi @hypertechlab,
The Hailo chip is meant for inference, meaning that the usage of Hailo comes for trained models (ONNX or tflite files), which needs to go through the Hailo Dataflow Compiler pipeline in order be used with the Hailo device.
You can go to the Hailo website under Developer Zone–>Documentation and look for the Dataflow Compiler’s user guide to better understand the needed flow.

Regards,