Custom Personal Assistant on the Home Assistant

Community Projects
, , , ,
1

Hello All…

I received my Hailo10h chip and got everything compiled and running on my Raspberry Pi 5 under regular old RaspianOS. Then I wrote a custom “Hailo Assistant” device for the Home Assistant (based off the Ollama device) to talk to Hailo-Ollama also running on the Pi 5 and using the Hailo10h chip for inference.

With all that said, I am getting really bad responses from all of the current list of models supported by the Hailo-Ollama server. The RPi5 with 8GB memory can easily run larger more accurate models. How exactly can one of the models listed here be integrated into the Hailo-Ollama server: hailo_model_zoo_genai/docs/MODELS.rst at main · hailo-ai/hailo_model_zoo_genai · GitHub

Is there a workflow for creating a proper manifest.json file so that Hailo-Ollama will see and use a new model? Or do I just copy one and modify it?

I’m real close to success on this project. I just need a better model.

Jeff

3 Likes
2

Hi @jeff.singleton ,

That’s an awesome project, and we’re glad you took advantage of using the Hailo-10H for it! We’d love to hear more about your journey:

  • Which models and what types of prompts have you been trying?

  • What kind of models have you been wishing to have supported?

  • If you can elaborate about what use case you are trying to do?

Note - In case it’s relevant, Hailo models can also be accessed via Python API. Here’s an example of such an app within a Home Assistant context: https://youtu.be/8dwVnmcZ9v0?si=Z2AeSxr0CuNgkZKi&t=144

Looking at the currently available list of models (HEFs):

  1. Qwen2-1.5B-Instruct-Function-Calling-v1 can be used with a manifest similar to existing models.

  2. Whisper models are not relevant for Ollama.

  3. VLMs will need additional support in code - we are looking into it.

Thanks,

3

Hi Michael…

Thanks for the response and encouragement. I have tried all models that come with the Hailo-Ollama code and am getting about the same responses across the board. When I use voice commands I get about 25% accuracy, and when I type the commands it goes up to about 50% accuracy.

The main goal here is to completely automate things in my home with the Home Assistant using my Raspberry Pi 5 and Hailo-10h with a model built with tooling support. Apparently being able to use tooling is key to getting better results for controlling things in my home…Lights, TV, Soundbar, Window Shades, etc.

When I was using the CPU only on the Pi5, I had things running with a security platform I was testing, and had good results using standard Ollama with a Phi4 model. Was hoping for a model similar to Phi4 with tooling support.

I’m hooked on this AI stuff :slight_smile: Think of how cool it will be to come home and have the lights, TV come on automatically to welcome me home?

Regards,

Jeff

4

Hi Jeff,
To see how we handle tool usage, you can check out our agent example here:

Since these models have a limited context length, it is important to manage the context window tightly. Additionally, some models are better at generating “formatted” outputs, which makes them more reliable for tool integration.
I completely share your vision and would love to see Hailo accelerating automations. An n8n node would be a great example of this. Please keep us updated on your progress.
We’d be happy to hear your feedback and suggestions.

1 Like
5

Doing the same thing and hitting the same walls…

1 Like
6

Yep. Seems like the hardware outpaced the software again. We just have to wait or like @TheRubbishRaider did use claude to add tools support to the hailort server. A work in progress, but has a lot of promise.

7

Hi with current models I suggest using the “coder” model it give the best structured results for tool usage.
We will keep updating and give more examples. But our resources are limited.
PLEASE DO use your great ideas and guide the army of robots we all got now to get your wish come true.

Dear Community,

We are investing on multiple targets and integrating with HA is one of our main goals. Please use the “like / heart” button to mark topics you think we should give more attention too. I already gave this one a heart :smiling_face_with_three_hearts:

1 Like