Qwen3-Embedding-0.6B support on Hailo-10H / Raspberry Pi AI HAT+ 2

General Discussion
, , ,
1

Hi Hailo team,

I have seen recent community discussions mentioning that upcoming GenAI Model Zoo releases may include newer Qwen-family models.

Our specific interest is Qwen3-Embedding-0.6B, not the autoregressive Qwen LLM.

We are building a general-purpose, verifiable documentary retrieval system on a Raspberry Pi 5 cluster and would like to benchmark Qwen3-Embedding-0.6B on the Raspberry Pi AI HAT+ 2 / Hailo-10H as a fully local dense embedding backend.

Current setup:

  • Raspberry Pi 5, 8 GB
  • Raspberry Pi AI HAT+ 2 / Hailo-10H
  • ARM64 Linux
  • OpenSearch
  • hybrid BM25 + dense retrieval + RRF
  • documentary provenance graph and deterministic scope traversal
  • current embedding baseline: OpenAI text-embedding-3-large

We already have a frozen, human-validated benchmark on a real corpus of approximately 47,000 document occurrences.

For our first benchmark case, the current OpenAI single-query baseline gives:

  • 114 human-validated relevant documentary components
  • 192 relevant documents
  • Seed Component Recall: 35.1%
  • Post-provenance Component Recall: 44.7%
  • Seed Document Recall: 25.0%
  • Post-provenance Document Recall: 45.8%

The benchmark is fully reproducible, and we can compare Qwen3-Embedding-0.6B against OpenAI while keeping the documents, chunks, BM25, RRF, provenance graph and ground truth identical.

We do not need to re-ingest the corpus: the existing text chunks can simply be re-embedded while preserving document IDs, chunk IDs, hashes, metadata and provenance.

Our target would initially be:

  • Qwen3-Embedding-0.6B
  • 1024-dimensional embeddings
  • text embedding only
  • typical chunk size around 512 tokens

Would it be possible for Hailo to:

  1. provide or compile a Hailo-10H HEF for Qwen3-Embedding-0.6B;
  2. confirm whether this embedding variant can use the same Qwen3 backend/toolchain that is being developed for upcoming Model Zoo support;
  3. or provide guidance on the officially supported compilation path?

If useful, we can provide:

  • the Hugging Face model;
  • ONNX export;
  • tokenizer configuration;
  • representative real-world calibration chunks;
  • benchmark scripts;
  • reproducible OpenAI-vs-Qwen retrieval quality measurements.

We would also be happy to share the resulting throughput, memory usage and retrieval-quality results from Raspberry Pi 5 + Hailo-10H.

If Qwen3 support is already being added internally for newer GenAI Model Zoo releases, it would be very interesting to know whether Qwen3-Embedding-0.6B could be compiled using the same underlying support.

Thank you!