YOLOv8 bbox decoding: DFL softmax overflows in float32 (no max-subtraction), producing NaN boxes

General Discussion
, ,
1

Summary

The CPU YOLOv8 post-process (nms_postprocess(..., meta_arch=yolov8, engine=cpu), also with bbox_decoding_only=True) decodes each box side with a softmax over the 16 DFL bins. The softmax computes exp(x) in float32 without subtracting the max first. When a dequantized regression logit exceeds about 88.72 (= ln FLT_MAX), exp overflows to +inf, the normalisation becomes inf / inf, and the decoded box coordinates are NaN.

The only guard is a hasNaN() check on the exp result. It doesn’t catch +inf, so the NaN only appears later, in the box outputs.

Whether this happens depends only on the calibrated output range of the box-regression layers. A HEF that works can start producing NaN boxes after a recompile with different calibration data, with no change to the model or the runtime.

Where

  • yolov8_post_process.hpp, get_bbox() (L45–L63): each dequantized 16-element DFL vector is passed to SoftmaxPostProcessOp::softmax() (line 63).

  • softmax_post_process.cpp, SoftmaxPostProcessOp::softmax() (L39–L52):

    • L45 src_eigen = src_eigen.array().exp();: float32 exp, no max-subtraction;

    • L47 CHECK(!src_eigen.hasNaN(), ...): +inf passes this check;

    • L49 float sum_exp = src_eigen.sum();: becomes +inf;

    • L52 dst_eigen = src_eigen / sum_exp;: inf / inf = NaN.

  • The file is byte-identical in v5.3.0, v5.4.0 and master (f51959034a; blob df8a6d11).

Environment

  • Hailo-10H, HailoRT 5.3.0 / FW 5.3.0 (Raspberry Pi 5)

  • Dataflow Compiler 5.3.0

  • A custom YOLOv8s detector, compiled at optimization level 4 with a16_w16 on the six head convs, plus nms_postprocess("<decode>.json", meta_arch=yolov8, engine=cpu, bbox_decoding_only=True)

What we observed

  • Same model and recipe, two calibration sets:

    • Compile A: calibrated range of the stride-8 box-regression output [-142.3, 88.50]. No NaN in our tests, but only 0.2 below the overflow point.

    • Compile B: the same output calibrated to [-177.0, 111.93]. The box outputs contained NaN: 13 and 18 anchors on two of six test frames, all in the stride-8 box rows (columns 0/2); the class scores were finite. Our acceptance harness aborted with Hailo BDO output contains NaN or infinity.

  • The on-chip outputs are finite. The NaN comes from the host-side decode.

Minimal reproduction of the arithmetic

Any 16-bin DFL vector with one element ≥ 88.73 (float32):

import numpy as np
x = np.full(16, -10.0, np.float32); x[5] = 90.0
e = np.exp(x)              # e[5] = inf (overflow)
print(e / e.sum())         # [0, ..., nan, ...]  -> NaN box side

Several bins close to the max overflow the sum even earlier: with 16 saturated bins, the safe bound is 88.72 − ln 16 ≈ 85.95.

Suggested fix

Use the numerically stable softmax (subtract the per-vector max before exp). The result is mathematically identical, and it cannot overflow:

// per 16-bin DFL vector
// in SoftmaxPostProcessOp::softmax(), before the exp at L45
const float max_val = src_eigen.maxCoeff();
src_eigen = (src_eigen.array() - max_val).exp();   // every term in (0, 1], sum in [1, 16]
float sum_exp = src_eigen.sum();
dst_eigen = src_eigen / sum_exp;

Optionally, also replace the hasNaN() check with allFinite() on the input, so that a non-finite input fails loudly instead of propagating.

Current workaround

Clamp the output range of each box-regression layer below the multi-bin bound in the model script, e.g.

quantization_param(conv41, force_range_out=[-177.0, 80.0])

With the clamp, compile B had 0 NaN rows and passed our accuracy acceptance. However, this has to be checked for every box head after every recompile, because the calibrated maximum is a statistic, not a runtime bound.

Search

We searched this forum (NaN / inf boxes, softmax, yolov8 postprocess, bbox decoding) and found no existing report. Issues are disabled on hailo-ai/hailort, so we’re reporting it here.