Hello Hailo team,
We are investigating basic EN/FR answer correctness for an offline tutor on Raspberry Pi 5 + Hailo10H, using HailoRT/driver 5.5.0 and the official Qwen2.5-Coder-1.5B-Instruct HEF. Exact artifact hash, environment and reproduction materials are included below.
The program is:
x = 2
y = x + 3
x = 9
print(y)
It prints 5: assigning x=9 does not reassign y.
With a registered greedy profile, our caller and an independently adapted official 5.5 C++ caller produced byte-identical responses: EN claims y remains 11, and FR claims print(y)=12, while both also compute the initial y=5.
We then tested the SDK-returned sampling knobs (temperature 0.7, top_p 0.8, top_k 20, frequency_penalty 1.1), explicitly retaining cap 256 and seed 17. EN reaches SDK stop but starts with output 12 and later explains y remains 5. FR reaches SDK LENGTH and is not semantically scored, despite a visible im_end marker.
Context was empty; exact Qwen default system, structured messages, getters, template and stops were checked. No retrieval or tutoring layers were added. A local supervisor classification defect was subsequently corrected offline; original native captures are unchanged.
The forum permits image attachments only, so exact SDK argument strings, four captured outputs with hashes/typed terminals, environment and a standalone C++ helper are included in the expandable sections below. The helper compiles/links against our installed 5.5 SDK; it has NOT been run natively. Captured outputs come from the prior experiments.
Could you help with:
- Reproducing this exact HEF/input/profile case and identifying the supported reference recipe, including comparison with the uncompiled source model?
- Clarifying frequency_penalty formula, history window and neutral value, and its relation to upstream repetition_penalty?
- Identifying supported server-rendered prompt/token-ID or compiled/upstream parity diagnostics, and typed-stop precedence at the token limit?
We have not isolated a model, compiled-artifact or runtime cause. Thank you.
Save the following sections as reproduce.cpp, REQUESTS.json and OBSERVED.json. On a Hailo-10H Linux test bench with the installed HailoRT 5.5 development headers/library, RapidJSON headers and a C++17 compiler:
g++ -std=c++17 -O2 -Wall -Wextra reproduce.cpp -o reproduce -lhailort -pthread
# Set HEF to your local copy of the exact artifact identified below.
timeout -k 5s 90s ./reproduce "$HEF" REQUESTS.json greedy_en > greedy_en.jsonl
Start with greedy_en: its captured answer is complete and contradictory. Other case IDs are greedy_fr, sampling_en and sampling_fr; use one invocation per case for a fresh model/context. The helper emits metadata (factory/effective getters, template/stops and actual prompt strings) and a result JSON line with raw output and typed SDK terminal. It reads only REQUESTS.json: observed answers and oracle values are never sent to the model.
These support-side commands were not executed for this export. The helper was compiled and linked only. Captured outputs come from the previous independently adapted-reference experiments. Parsed role/content messages match across profiles; original JSON member order differs and is preserved verbatim. No pure isolated-parameter causality is claimed.
Exact REQUESTS.json
{
"schema": "hailo_support.exact_requests/1",
"cases": [
{
"case_id": "greedy_en",
"language": "en",
"parameters": {
"do_sample": false,
"temperature": 1.0,
"top_p": 1.0,
"top_k": 1,
"frequency_penalty": 1.0,
"seed": 17,
"maximum_generated_tokens": 256
},
"prompt_json_strings": [
"{\"content\":\"You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\",\"role\":\"system\"}",
"{\"content\":\"Explain why changing x does or does not change y.\\n```python\\nx = 2\\ny = x + 3\\nx = 9\\nprint(y)\\n```\",\"role\":\"user\"}"
],
"prompt_json_utf8_sha256": [
"5733bd85459e4ea951476e3c739f31219b3f734e69b3b6eb576356c63efcd688",
"a936f2580b199dfaf030de14fbea594c9195fb41a1c8e499bab7ce8e1dc7e28e"
]
},
{
"case_id": "greedy_fr",
"language": "fr",
"parameters": {
"do_sample": false,
"temperature": 1.0,
"top_p": 1.0,
"top_k": 1,
"frequency_penalty": 1.0,
"seed": 17,
"maximum_generated_tokens": 256
},
"prompt_json_strings": [
"{\"content\":\"You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\",\"role\":\"system\"}",
"{\"content\":\"Explique pourquoi modifier x change ou ne change pas y.\\n```python\\nx = 2\\ny = x + 3\\nx = 9\\nprint(y)\\n```\",\"role\":\"user\"}"
],
"prompt_json_utf8_sha256": [
"5733bd85459e4ea951476e3c739f31219b3f734e69b3b6eb576356c63efcd688",
"8bcc1f015b8110848a3ef473d128c9707e7390f3fbb6598cca51088eef5c368b"
]
},
{
"case_id": "sampling_en",
"language": "en",
"parameters": {
"do_sample": true,
"temperature": 0.699999988079071,
"top_p": 0.800000011920929,
"top_k": 20,
"frequency_penalty": 1.100000023841858,
"seed": 17,
"maximum_generated_tokens": 256
},
"prompt_json_strings": [
"{\"role\":\"system\",\"content\":\"You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\"}",
"{\"role\":\"user\",\"content\":\"Explain why changing x does or does not change y.\\n```python\\nx = 2\\ny = x + 3\\nx = 9\\nprint(y)\\n```\"}"
],
"prompt_json_utf8_sha256": [
"a9281f758053e7236f2fe60e1f9402cbdf7e4d771601e651eb98ca41a4d90844",
"9c6a83a80c554650fce0a7294073d029b440f58803cdc32d54422b11f9485d88"
]
},
{
"case_id": "sampling_fr",
"language": "fr",
"parameters": {
"do_sample": true,
"temperature": 0.699999988079071,
"top_p": 0.800000011920929,
"top_k": 20,
"frequency_penalty": 1.100000023841858,
"seed": 17,
"maximum_generated_tokens": 256
},
"prompt_json_strings": [
"{\"role\":\"system\",\"content\":\"You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\"}",
"{\"role\":\"user\",\"content\":\"Explique pourquoi modifier x change ou ne change pas y.\\n```python\\nx = 2\\ny = x + 3\\nx = 9\\nprint(y)\\n```\"}"
],
"prompt_json_utf8_sha256": [
"a9281f758053e7236f2fe60e1f9402cbdf7e4d771601e651eb98ca41a4d90844",
"0e6f5d5595942fdd62f6a302785c578eddf80d9242650b8808796d476f5356d8"
]
}
],
"note": "Only these captured role/content JSON strings are passed to LLM::generate. No oracle or observed answer is an input."
}
Captured OBSERVED.json
{
"schema": "hailo_support.observations/1",
"trusted_code": "x = 2\ny = x + 3\nx = 9\nprint(y)\n",
"expected_stdout": "5\n",
"cases": [
{
"case_id": "greedy_en",
"SDK_terminal": "stop",
"native_state": 2,
"raw": "In the given Python code, the value of `x` is changed from `2` to `9`, but this change does not affect the value of `y`. The value of `y` remains `11` because it is assigned the result of `x + 3` when `x` is `2`, and then `x` is changed to `9` before the value of `y` is updated again.\n\nHere's a step-by-step breakdown of the code execution:\n\n1. `x = 2`\n2. `y = x + 3` (This line calculates `y` to be `5` because `x` is `2` and `3` is added to it)\n3. `x = 9`\n4. `y` remains `11` because the value of `y` is not updated again after `x` is changed to `9`.\n\nThus, changing `x` does not change `y` in this specific case.<|im_end|>",
"raw_bytes": 626,
"raw_sha256": "46b60c88e5146d7b32958d9d85b7ff74ddbbfae19aeba809226ca4a0be943c5d",
"context_before": 0,
"context_after": 269,
"generation_seconds": 24.242425981,
"semantics": "MATERIAL_CONTRADICTION_EXPECTED5",
"source": "Captured adapted-reference run; not generated by supplied support-only reproducer.",
"source_trace_sha256": "8b653ab1426892ec8d8a256eecc1fb99df69583536efdc9dd35ad219b3cc27b2"
},
{
"case_id": "greedy_fr",
"SDK_terminal": "stop",
"native_state": 2,
"raw": "Dans le code Python donné, le résultat de `print(y)` sera `12`. Voici un détaile de chaque étape du processus :\n\n1. `x = 2` : La variable `x` est initialisée avec la valeur `2`.\n2. `y = x + 3` : La variable `y` est initialisée avec la valeur de `x` plus `3`, donc `y` devient `2 + 3 = 5`.\n3. `x = 9` : La variable `x` est réinitialisée avec la valeur `9`.\n4. `print(y)` : La valeur de `y` est `12`, donc le programme imprime `12`.\n\nEn conclusion, le résultat de `print(y)` est `12` car la variable `y` est mise à jour avec la nouvelle valeur de `x` (`9`) plus `3` (`3`) lors de la première étape de l'exécution du code.<|im_end|>",
"raw_bytes": 642,
"raw_sha256": "1c877f90677ad2ac782ae7f53cdc533de06c0dcd04435eabc78e42e57df369c1",
"context_before": 0,
"context_after": 272,
"generation_seconds": 24.592147302,
"semantics": "MATERIAL_CONTRADICTION_EXPECTED5",
"source": "Captured adapted-reference run; not generated by supplied support-only reproducer.",
"source_trace_sha256": "8b653ab1426892ec8d8a256eecc1fb99df69583536efdc9dd35ad219b3cc27b2"
},
{
"case_id": "sampling_en",
"SDK_terminal": "stop",
"native_state": 2,
"raw": "The output of the code will be `12`.\n\nLet's break down the reasoning:\n\n1. The initial value of `x` is set to 2.\n2. The expression `x + 3` calculates the value of `y`, which is initially 5 (because `x` is still 2).\n3. When `x` is changed to 9, the value of `y` remains unchanged at 5 because `x` does not affect the calculation of `y`.\n\nTherefore, changing `x` does not change `y`.<|im_end|>",
"raw_bytes": 390,
"raw_sha256": "99c11e232a0f0f51668671d77e3ce97142e6e0df95d2ec07c64d186351f93412",
"context_before": 0,
"context_after": 179,
"generation_seconds": 13.784991815,
"semantics": "MATERIAL_CONTRADICTION_EXPECTED5",
"source": "Captured adapted-reference run; not generated by supplied support-only reproducer.",
"source_trace_sha256": "652d2aa8a34b7421f8810e390de1cf9f1316a4ac6820cc579743d2cd488171f8"
},
{
"case_id": "sampling_fr",
"SDK_terminal": "length",
"native_state": 1,
"raw": "Le code Python suivant illustre comment x change ou ne change pas y.\n\n```python\nx = 2\ny = x + 3\nx = 9\nprint(y)\n```\n\nEn exécutant ce code, voici les résultats qui sont obtenus :\n\n1. `x` est initialisé à la valeur `2`.\n2. Une nouvelle variable `y` est créée avec la valeur de `x + 3`, donc `y` sera égal à `5` (car `2 + 3 = 5`).\n3. La valeur de `x` est ensuite modifiée pour avoir la valeur `9`.\n\nAprès avoir modifié `x` en `9`, le nouveau `x` vaut `9`. Cette modification n'affecte pas la valeur de `y`, car elle a été initialisée avant que `x` ne soit modifié.\n\nEn conclusion, lorsque vous modifiez une variable (`x`), cette modification n'impacte pas directement la valeur de la variable précédente (`y`). Cependant, si vous utilisez des variables qui dependent de l'historial de valeurs précédentes, cela peut conduire à des comportements indésirables.<|im_end|>",
"raw_bytes": 885,
"raw_sha256": "815a992c94b819f70ebd0f7f993c66bd8382758eb02b1ba693b82e1c7dd99d05",
"context_before": 0,
"context_after": 322,
"generation_seconds": 30.710478371,
"semantics": "NOT_SCORED_INCOMPLETE_LENGTH",
"source": "Captured adapted-reference run; not generated by supplied support-only reproducer.",
"source_trace_sha256": "652d2aa8a34b7421f8810e390de1cf9f1316a4ac6820cc579743d2cd488171f8"
}
],
"caller_comparison": "In the greedy experiment our caller and independently adapted official5.5caller produced identical argument bytes and output bytes in each language.",
"sampling_limit": "Observed factory sampling knobs used with cap256/seed17 instead of literal factorycap1024/randomseed. Seed29not attempted. No penalty-neutrality or equivalence with upstream repetition_penalty inferred.",
"local_incident": "Sampling supervisor incorrectly treated expected LENGTH/BUDGET_EXCEEDED fallback as technical and kept only one driver record. Both native results/chunks were preserved; classifier was subsequently corrected and checked offline on4focused cases. Original captures remain unchanged. No native SDKfault or schema rejection observed in these two results.",
"serialization_note": "Parsed role/content messages match across profiles; captured JSON member order differs and is preserved verbatim. A/B argument-byte equality was established inside the greedy route comparison. This is not a pure isolated-decoder causal test.",
"cause": "Not isolated to SDK,HEF,quantization or intrinsicmodel capacity. This is a minimal finite correctness report."
}
Environment and exact HEF identity
{
"host": "Raspberry Pi5,8GB RAM",
"accelerator": "Hailo10H",
"HailoRT": "5.5.0",
"model_name": "Qwen2.5-Coder-1.5B-Instruct",
"model_bytes": 1666858287,
"model_sha256": "e0206dc88e0179b84cc0d3ea3a3232b8db72d3f14a5eded5fda6a878bcf713a8",
"official_model_URL": "https://dev-public.hailo.ai/v5.5.0/blob/Qwen2.5-Coder-1.5B-Instruct.hef",
"model_source_URL": "https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct",
"official_reference_source": "https://github.com/hailo-ai/hailort/blob/e7727f7482477e448f4c93b33dc62f9e5a30c3e7/hailort/libhailort/examples/genai/chat_example/chat.cpp",
"SDK_header_sha256": "f034f9fc88825fa8f91828e95ea2a0f1c5bc35bfb2a36fb7adb51f61a12ecd3a",
"captured_client_library": {
"file": "libhailort.so.5.5.0",
"sha256": "0b711e4164bb758a429726900c65119348cecdf9ecab133b95fd5dd5a64b555c"
},
"context_capacity": 2048,
"context_before_each_request": 0,
"chat_template_kwargs": {},
"optimize_memory_on_device": false,
"tokenization_location": "server",
"expected_stop_tokens": [
"<|im_end|>",
"<|endoftext|>"
],
"expected_template_file": "EXPECTED_TEMPLATE.txt",
"expected_template_sha256": "cd8e9439f0570856fd70470bf8889ebd8b5d1107207f67a5efb46e342330527f",
"tools": [],
"RAG_or_pedagogical_layers": false,
"scope": "Exact default Qwen system plus one user question. No tools/retrieval/extra educational prompt instructions.",
"model_weights_included": false,
"build_environment_snapshot_date": "2026-11-06",
"architecture": "aarch64",
"OS": {
"PRETTY_NAME": "Debian GNU/Linux 13 (trixie)",
"VERSION_ID": "13",
"ID": "debian"
},
"kernel": "6.12.75+rpt-rpi-2712",
"driver_version_at_build": "5.5.0",
"build_check": "BUILD_CHECK.json",
"firmware_version": "No fresh firmware/device query performed for this support export."
}
Standalone reproduce.cpp — build-checked, not native-executed
// Support-only reproducer derived from the HailoRT 5.5.0 chat example.
// Official reference: e7727f7482477e448f4c93b33dc62f9e5a30c3e7
// Modified transport: exact captured structured JSON strings from REQUESTS.json.
// This file has been build-checked only; it is NOT the caller that produced OBSERVED.json.
#include "hailo/genai/llm/llm.hpp"
#include <rapidjson/document.h>
#include <rapidjson/stringbuffer.h>
#include <rapidjson/writer.h>
#include <algorithm>
#include <chrono>
#include <fstream>
#include <iostream>
#include <iterator>
#include <stdexcept>
#include <string>
#include <vector>
using rapidjson::Document;
using rapidjson::Value;
using Alloc = Document::AllocatorType;
using Clock = std::chrono::steady_clock;
static Value string_value(const std::string &s, Alloc &a) {
return Value(s.data(), static_cast<rapidjson::SizeType>(s.size()), a);
}
static void emit(const Document &d) {
rapidjson::StringBuffer b;
rapidjson::Writer<rapidjson::StringBuffer> w(b);
if (!d.Accept(w)) throw std::runtime_error("JSON output encoding failed");
std::cout << std::string(b.GetString(), b.GetSize()) << std::endl;
}
static void checked(hailo_status status, const char *operation) {
if (status != HAILO_SUCCESS) throw hailort::hailort_error(status, operation);
}
static Value parameters(const hailort::genai::LLMGeneratorParams &p, Alloc &a) {
Value v(rapidjson::kObjectType);
v.AddMember("do_sample", p.do_sample(), a);
v.AddMember("temperature", p.temperature(), a);
v.AddMember("top_p", p.top_p(), a);
v.AddMember("top_k", p.top_k(), a);
v.AddMember("frequency_penalty", p.frequency_penalty(), a);
v.AddMember("seed", p.seed(), a);
v.AddMember("maximum_generated_tokens", p.max_generated_tokens(), a);
return v; // Every nested string/value uses the owning output document allocator.
}
int main(int argc, char **argv) {
try {
if (argc != 4) throw std::runtime_error("Usage: reproduce <hef_path> <REQUESTS.json> <case_id>");
std::ifstream in(argv[2], std::ios::binary);
if (!in) throw std::runtime_error("Cannot open requests file");
const std::string bytes((std::istreambuf_iterator<char>(in)), {});
Document requests;
requests.Parse<rapidjson::kParseValidateEncodingFlag>(bytes.data(), bytes.size());
if (requests.HasParseError() || !requests.IsObject() || !requests.HasMember("cases") || !requests["cases"].IsArray())
throw std::runtime_error("Invalid requests JSON");
const Value *selected = nullptr;
for (const auto &c : requests["cases"].GetArray()) {
if (std::string(c["case_id"].GetString()) == argv[3]) {
if (selected) throw std::runtime_error("Duplicate case_id");
selected = &c;
}
}
if (!selected) throw std::runtime_error("Unknown case_id");
const Value &c = *selected;
std::vector<std::string> prompts;
for (const auto &s : c["prompt_json_strings"].GetArray()) {
if (!s.IsString()) throw std::runtime_error("Prompt must be a role/content JSON string");
prompts.emplace_back(s.GetString(), s.GetStringLength());
}
if (prompts.size() != 2) throw std::runtime_error("Expected two structured messages");
const Value &wanted = c["parameters"];
auto device = hailort::VDevice::create_shared().expect("create VDevice");
auto model_params = hailort::genai::LLMParams(argv[1]);
model_params.set_optimize_memory_on_device(false);
auto llm = hailort::genai::LLM::create(device, model_params).expect("create LLM");
checked(llm.set_chat_template_kwargs("{}"), "set kwargs");
checked(llm.clear_context(), "clear context");
const auto before = llm.get_context_usage_size().expect("context before");
const auto capacity = llm.max_context_capacity().expect("context capacity");
if (before != 0 || capacity != 2048) throw std::runtime_error("Expected empty context and capacity 2048");
auto p = llm.create_generator_params().expect("factory parameters");
Document meta;
meta.SetObject();
auto &ma = meta.GetAllocator();
meta.AddMember("kind", "metadata", ma);
meta.AddMember("case_id", string_value(argv[3], ma), ma);
meta.AddMember("factory_parameters", parameters(p, ma), ma);
checked(p.set_do_sample(wanted["do_sample"].GetBool()), "do_sample");
checked(p.set_temperature(static_cast<float>(wanted["temperature"].GetDouble())), "temperature");
checked(p.set_top_p(static_cast<float>(wanted["top_p"].GetDouble())), "top_p");
checked(p.set_top_k(wanted["top_k"].GetUint()), "top_k");
checked(p.set_frequency_penalty(static_cast<float>(wanted["frequency_penalty"].GetDouble())), "frequency_penalty");
checked(p.set_seed(wanted["seed"].GetUint()), "seed");
checked(p.set_max_generated_tokens(wanted["maximum_generated_tokens"].GetUint()), "maximum_generated_tokens");
meta.AddMember("effective_parameters", parameters(p, ma), ma);
meta.AddMember("context_before", static_cast<uint64_t>(before), ma);
meta.AddMember("context_capacity", static_cast<uint64_t>(capacity), ma);
meta.AddMember("optimize_memory_on_device", model_params.optimize_memory_on_device(), ma);
meta.AddMember("prompt_template", string_value(llm.prompt_template().expect("template"), ma), ma);
meta.AddMember("chat_template_kwargs", string_value(llm.get_chat_template_kwargs().expect("kwargs"), ma), ma);
Value stops(rapidjson::kArrayType);
for (const auto &s : llm.get_stop_tokens().expect("stop tokens")) stops.PushBack(string_value(s, ma), ma);
meta.AddMember("stop_tokens", stops, ma);
Value args(rapidjson::kArrayType);
for (const auto &s : prompts) args.PushBack(string_value(s, ma), ma);
meta.AddMember("prompt_json_strings", args, ma);
emit(meta);
// Only REQUESTS.json messages are sent. OBSERVED.json and oracle values are never read here.
const auto start = Clock::now();
const auto deadline = start + std::chrono::seconds(45);
auto completion = llm.generate(p, prompts, {}).expect("generate");
std::string raw;
bool deadline_abort = false;
while (completion.generation_status() == hailort::genai::LLMGeneratorCompletion::Status::GENERATING) {
const auto remaining = std::chrono::duration_cast<std::chrono::milliseconds>(deadline - Clock::now());
if (remaining.count() <= 0) {
deadline_abort = true;
checked(completion.abort(), "abort deadline");
break;
}
auto next = completion.read(std::chrono::milliseconds(std::max<int64_t>(1, remaining.count())));
if (!next && next.status() == HAILO_TIMEOUT) {
deadline_abort = true;
checked(completion.abort(), "abort read timeout");
break;
}
raw += std::move(next).expect("read");
}
const auto status = completion.generation_status();
using Status = hailort::genai::LLMGeneratorCompletion::Status;
const char *terminal = status == Status::LOGICAL_END_OF_GENERATION ? "stop" :
status == Status::MAX_TOKENS_REACHED ? "length" : status == Status::ABORTED ? "cancelled" : "unknown";
Document output;
output.SetObject();
auto &a = output.GetAllocator();
output.AddMember("kind", "result", a);
output.AddMember("case_id", string_value(argv[3], a), a);
output.AddMember("SDK_terminal", string_value(terminal, a), a);
output.AddMember("native_state", static_cast<int>(status), a);
output.AddMember("raw", string_value(raw, a), a);
output.AddMember("deadline_abort", deadline_abort, a);
output.AddMember("generation_seconds", std::chrono::duration<double>(Clock::now() - start).count(), a);
output.AddMember("context_after", static_cast<uint64_t>(llm.get_context_usage_size().expect("context after")), a);
emit(output);
if (deadline_abort || status == Status::ABORTED) return 3;
return 0;
} catch (const hailort::hailort_error &e) {
std::cerr << "HailoRT error " << e.status() << ": " << e.what() << std::endl;
return 2;
} catch (const std::exception &e) {
std::cerr << e.what() << std::endl;
return 2;
}
}
HailoRT example license
The MIT License (MIT)
Copyright (c) 2019-2026 Hailo Technologies Ltd.
All rights reserved.
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.