Correct, but this is community effort. Neither myself nor @TheRubbishRaider are employees. We are just eager users trying different things and hoping for a solution.
Right now, you would need to compile the code provided by @TheRubbishRaider and then use the Python based FastAPI server by me to get this working with tools support.
It does work! I have both services running on a Raspberry Pi 5 with the Qwen2.5 (1.5b) Instruct model and using my Home Assistant with a custom device I wrote for client side chat. I can chat non stop, but an instruct model is too chatty and doesn’t execute commands on my Home Assistant.
@user459 and @jeff.singleton one important thing I discovered based on the tool calling is that it does depend on the model. I have tool/function calling working now but each model needs to be transformed back and forth between the ollama standard json
That is different for each model, for example I have DeepSeek and Qwen-Instruct models working but each one has its own formatting
I have a question. I cannot start the hailort_server, as it is looking for a device called hailo_pci_e, while mine is name hailo0. Do you install the dkms dependency before executing all those steps? Do you have any idea what else could be?
reading the posts better, I noticed that @jeff.singleton had the same issue and it is something else. I used a clean install too, but now I don’t even have the hailo0 device. Could it be the DKMS library, or the python version? I downloaded the 3.13 pci driver
Thanks for contributing to the community and sharing the code!