Hailo-ollama return type unsuitable for Open WebUI

[Tony]

I think this may be going in a similar direction as

Using hailo-ollama server with ollama client

Testing out the Hailo-10 on the RPI5. The “hailo-ollama” API is working. My RPI5+Hailo-10 is at IP address 10.0.0.108, so the following is working from a separate computer: curl --silent ``http://10.0.0.108:8000/api/chat`` -H ‘Content-Type: application/json’ -d ‘{“model”: “qwen2:1.5b”, “messages”: [{“role”: “user”, “content”: “Who are you?”}]}’ Is this API compatible with the OLLAMA API ? I tried the following, which is not working: export OLLAMA_HOST=10.0.0.108:8000 ollama run qwen2:1.5b…

but since I am not sure I am opening the issue with a more precise description.

The goal is to use H10 as LLM provider for agentic coding using OpenCode.
Since hailo-ollama implements only the REST API and not the full functionality of Ollama
and it is thus not possible to connect OpenCode to it I want to use Open WebUI as intermediary.

Using v5.3.0 in conjunction with OWUI 0.9.5 works, but requires a fix of user input sanitisation
to prevent JSON parsing errors (JSON config - HAILO_INTERNAL_FAILURE(8)).
With the fix I observe no problems interfacing with the LLMs through the web interface.

Now for the problem:

When attempting to submit a simple query (e.g. Write a Hello World script in Python.) from OpenCode via OWUI
I always encounter Server Connection error.
Looking at OWUI debug output I see

Failed to parse error response: 500, message='Attempt to decode JSON with unexpected mimetype: ', url='http://localhost:8000/api/chat'

This indicates that whatever is returned by hailo-ollama to OWUI is not formatted as expected.
This is likely a hailo-ollama-specific issue, because trying the same with an Ollama docker container (on a different system)
works.

My question is thus: Is this issue known and is there a (simple) fix for it?

I originally thought the issue was the use of application/x-ndjson as return type rather than plain application/json, but switching did not help, so that’s probably not the solution

[Michael]

Hi @Tony,

Since you’ve already resolved the newline sanitization issue and direct OWUI chat works, the remaining problem likely comes down to OpenCode (through OWUI) sending Ollama API request fields or parameters that hailo-ollama doesn’t handle or different streaming expectations - which causes hailo-ollama to hit an error path and return a 500 without proper JSON content-type headers, hence the “unexpected mimetype” on OWUI’s side.

You might try inspecting the exact request body that OWUI forwards to hailo-ollama when the error occurs (e.g., by running hailo-ollama with verbose logging or placing a simple logging proxy between OWUI and hailo-ollama on port 8000) to identify which specific field or request shape triggers the failure - once you know that, you could strip or modify those unsupported fields in an OWUI pipe/filter function before the request reaches hailo-ollama.

Identifying the exact triggering field from your logs would be very helpful if you’d like to share it here.

Thanks,

[Tony]

Hi!
By the ancient art of printf-debugging I have previously managed to
narrow down the origin of the error to the non-streaming section of handle_completion.
But that appears to be a red herring.

The server error is indicated in OpenCode immediately after submitting the query. Looking at the same time at the output from hailo-ollama it takes it a while to process the prompts but it appears to get all of them. Which currently suggests that the error may be actually due to the way the request is processed by oatpp.

Since the current implementation uses oatpp::network::Server rather than ouatpp::web::server
it doesn’t seem like I can register request/response interceptors to get in-depth info on what is
being received where (or at least it’ll take a bit more work). For now I’ve dumped the traffic on the hailo-ollama port using ngrep and will have a closer look at that.
The error I’m getting through the capture at run time is:

server=oatpp/1.4.0
code=500
description=Internal Server Error
stacktrace:
  - [oatpp::data::mapping::TreeToObjectMapper::mapString()]: Node is NOT a STRING
  - [ApiController]: Error processing request
  - Error processing request

I’ll get back to you once I’ve had a more in-depth analysis of the received data.

[Tony]

It seems like I was able to somewhat pinpoint the source of the errors: Function calling. Specifically: The JSON request containing a tools array.

I was able to elicit the same error when querying qwen3:1.7b from OWUI. Disabling Reasoning and Stream Chat Response in the controls of the chat and setting Function Calling to Native instead of Default will result in this error.

The packet sent prior to the error consists of a JSON object containing the members model, messages, options, stream, tools, where

1. options just contains num_ctx:null and temperature:0.7
2. stream contains stream:false
3. tools contains an array of objects describing available functions, their parameters etc.

This packet consistently results in the mapping error and thus the server error.

In contrast, turning Function calling to Default instead will result in a packet consisting of all of the above members except tools and will succeed.

How to fix

A provisional fix for OWUI is to disable Builtin Tools as one of the Capabilities of the model in OWUI admin settings of the model. This will disable the tools field even when using the Native option.
This will not fix the issue for external tools that use OWUI as a provider (e.g., OpenCode). They can pass their own set of tools and trigger this problem. To fix the problem also for external tools a filter for the model must be implemented that strips the tools member from the message body.

The simplest way is to go to admin settings>Functions and create a new function with the only action in the inlet being body.pop("tools"):

from pydantic import BaseModel, Field
from typing import Optional


class Filter:
    class Valves(BaseModel):
        pass

    def __init__(self):
        self.valves = self.Valves()
        pass

    async def inlet(self, body: dict, __user__: Optional[dict] = None) -> dict:
        # Remove the tools
        body.pop("tools")

        return body

    async def outlet(self, body: dict, __user__: Optional[dict] = None) -> dict:
        # We do not have to modify what the model returns to us.
        return body

The function must then be enabled and also enabled under “Filters” for each model you want to use it for. In my case I enabled it only for qwen2.5-coder:1.5b so as to avoid breaking other models.
With the filter in place I can now invoke OpenCode without a server connection error.

[Michael]

Thanks @Tony for sharing this solution to the community - I’m sure it will help others!

[Erik_Nauman]

This was the solution I was looking for, thank you! OpenWebUI as of v0.10.0 has tools enabled by default for all models so I was getting this JSON return type error. Setting the models to ‘legacy’ for function calling and removing built in tools from capabilities solved the issue and OpenWebUI interfaces with hailo-ollama correctly now. I’m using a Raspberry Pi 5 (Trixie) with the Raspi AI Hat + 2.

[Petr_Praha]

Hailo-Ollama 0.5.1: Llama 3.2 3B fails with Open WebUI web search, while Qwen2 1.5B works

Summary

I am running Hailo-Ollama 0.5.1 on a Raspberry Pi 5 with an AI HAT+ 2 (Hailo-10H).

Open WebUI web search works correctly with qwen2:1.5b, but the same web-search request fails consistently with llama3.2:3b.

Both models are configured identically in Open WebUI.

The failure occurs after Open WebUI successfully retrieves the web-search results. The error is:

Response payload is not completed:
<TransferEncodingError: 400, message='Not enough data to satisfy transfer length header.'>

Environment

• Raspberry Pi 5

• Raspberry Pi AI HAT+ 2 / Hailo-10H

• Hailo-Ollama: 0.5.1

• Open WebUI: 0.11.0

• Models:

  • qwen2:1.5b

  • llama3.2:3b

• Web search provider: Brave

• Brave LLM Context is enabled

• Open WebUI Function Calling: Legacy

• Builtin Tools: disabled

Hailo-Ollama models

The following models are available:

deepseek_r1_distill_qwen:1.5b
llama3.2:3b
qwen2.5-coder:1.5b
qwen2.5-instruct:1.5b
qwen2:1.5b

curl http://127.0.0.1:8000/api/tags successfully returns all five models.

Reproduction

In Open WebUI, with qwen2:1.5b:

Prompt:

Give me the latest headlines on the stock market

Result:

Retrieved 3 sources

LIVE • LATEST MARKET MOVES
Live updates
Micron Shares: Technical Perspective and Price Movement
Aug 5, 1:31 PM EDT
Stocks Advance as Crude Slides; Dollar Edges Lower
Aug 5, 1:30 PM EDT
Stocks Climb to Records on Robust Earnings; Economic Data Spurs Varied Moves

The request completes successfully.

With llama3.2:3b, using the same Open WebUI configuration and prompt:

Can you search the web for today's date

Open WebUI reports:

Retrieved 3 sources

Response payload is not completed:
<TransferEncodingError: 400, message='Not enough data to satisfy transfer length header.'>

I also tested with 8 search results:

Retrieved 8 sources

Response payload is not completed:
<TransferEncodingError: 400, message='Not enough data to satisfy transfer length header.'>

Therefore the search itself appears to succeed: Open WebUI receives the retrieved sources. The failure occurs while processing/completing the response from llama3.2:3b.

Open WebUI model configuration

The JSON configuration for llama3.2:3b shows:

"capabilities": {
  "file_context": true,
  "vision": true,
  "file_upload": true,
  "web_search": true,
  "image_generation": true,
  "code_interpreter": true,
  "terminal": true,
  "citations": true,
  "status_updates": true,
  "usage": true,
  "memory": true,
  "builtin_tools": false
},
"params": {
  "function_calling": "legacy"
}

So web search is enabled and builtin tools are disabled.

Additional Hailo-Ollama API issue

I also noticed that:

curl http://127.0.0.1:8000/api/tags

works and returns all models.

However:

curl http://127.0.0.1:8000/api/show \
-H "Content-Type: application/json" \
-d '{"name":"qwen2:1.5b"}'

returns:

server=oatpp/1.4.0
code=500
description=Internal Server Error
stacktrace:
  - [oatpp::data::type::String::operator std::string() const]: Error. Null pointer.
  - [ApiController]: Error processing request
  - Error processing request

This also causes the model list in AnythingLLM to remain empty, although inference itself works.

Question

Could this be a Hailo-Ollama 0.5.1 issue specific to the Hailo-compiled llama3.2:3b model, or an issue with how Hailo-Ollama handles the Open WebUI web-search/RAG request or streaming response for this model?

The interesting comparison is:

qwen2:1.5b + Open WebUI web search  → works
llama3.2:3b + Open WebUI web search → TransferEncodingError

Both use the same Hailo-Ollama server, Open WebUI instance, Brave search configuration, and model settings.

Any suggestions for additional diagnostics would be appreciated. I can provide Hailo-Ollama logs or the exact API request/response if useful.