tddfa_mobilenet_v1.hef from the Hailo Model Zoo

General Discussion
1

With some help from you guys I was able to get a working face detector using scrfd_2.5g.hef. My next step is to do 68 point landmark detection on the detected face(s).

To that end, I found tddfa_mobilenet_v1.hef on the hailo model zoo ready to use for the hailo8l, and I modified my script to crop the face that was detected and preprocess it (resize to 120x120, etc) and run the inference. I got back 2 outputs: one that’s got a shape of [120,120,3] which I assume is the original cropped face(?) and one that’s got a shape of [62,].

I need to write a post processor for this now, and I found this in the model zoo source, but when I tried rewriting that into my code it didn’t work well. I ran it through chatgpt hoping it would point out where I was making a stupid mistake and it came back and told me (after much frustration!) that I need to know two values in order to post process this model: mean_face and pca_basis. What are these values and where can I find them (if I even need them?)? Sorry, I’m very confused on what to do with the outputs from this model.

Thanks in advance!

2

I’m still stuck on this. I found the yaml file in the model zoo for face landmark detection and it points to: GitHub - cleardusk/3DDFA_V2: The official PyTorch implementation of Towards Fast, Accurate and Stable 3D Dense Face Alignment, ECCV 2020.. That archive has pkl files that contain the mean_face and pca_basis but those values didn’t seem to work with the hef I have. Are they the right ones and I’m just doing something really wrong or am I missing the right values? I have the outputs from the model and I dequantized them with the scale and zero_points, but I can’t get the landmarks to look correct.

Help, please! :slight_smile:

3

For reference, here is my sample script to do the scrfd detection and then tddfa landmark detection.

import cv2
import numpy as np
import queue
from pose_utils import HailoAsyncInference, load_images_opencv
from scrfd_numpy_bbox_postproc import SCRFDBBoxPostProc

# -----------------------------
# Resize helpers
# -----------------------------
def resize_image(image, target_size=(640, 640)):
    return cv2.resize(image, target_size, interpolation=cv2.INTER_LINEAR)

def resize_face_for_tddfa(face_img, target_size=(120, 120)):
    return cv2.resize(face_img, target_size, interpolation=cv2.INTER_LINEAR)

def draw_bboxes(image, boxes, scores, conf_thresh=0.35):
    for box, score in zip(boxes, scores):
        if score < conf_thresh:
            continue
        x1, y1, x2, y2 = box.astype(int)
        cv2.rectangle(image, (x1, y1), (x2, y2), (0, 255, 0), 2)
        cv2.putText(image, f"{score:.2f}", (x1, max(0, y1-4)),
                    cv2.FONT_HERSHEY_SIMPLEX, 0.4, (0,255,0), 1, cv2.LINE_AA)
    return image

def draw_landmarks(image, landmarks):
    for x, y in landmarks:
        cv2.circle(image, (int(x), int(y)), 1, (0,0,255), -1)

# -----------------------------
# TDDFA Postprocessor
# -----------------------------
class TDDFAProcessor:
    def __init__(self, scale, zero_point, configs_dir="configs"):
        self.scale = scale
        self.zero_point = zero_point

        self.mean_shape = np.load(f"{configs_dir}/bfm_mean_shape.npy")
        self.shape_basis = np.load(f"{configs_dir}/bfm_shape_basis.npy")
        self.exp_basis   = np.load(f"{configs_dir}/bfm_exp_basis.npy")
        self.keypoints   = np.load(f"{configs_dir}/bfm_keypoints.npy")
        self.triangles   = np.load(f"{configs_dir}/bfm_triangles.npy")

        self.img_width = None
        self.img_height = None

    def dequantize(self, output_uint8):
        return (output_uint8.astype(np.float32) - self.zero_point) * self.scale

    def reconstruct_vertices(self, coeffs):
        shape_coeffs = coeffs[:self.shape_basis.shape[1]]
        exp_coeffs   = coeffs[self.shape_basis.shape[1]:self.shape_basis.shape[1] + self.exp_basis.shape[1]]
        vertices = self.mean_shape.flatten() + self.shape_basis @ shape_coeffs + self.exp_basis @ exp_coeffs
        return vertices.reshape(-1, 3)

    def project_to_2d(self, vertices, face_box):
        if self.img_width is None or self.img_height is None:
            raise ValueError("Set img_width and img_height before calling postprocess")

        valid_keypoints = self.keypoints[self.keypoints < vertices.shape[0]]
        landmarks_3d = vertices[valid_keypoints]

        x1, y1, x2, y2 = face_box
        w, h = x2 - x1, y2 - y1

        # Scale to actual face crop
        landmarks_2d = np.zeros((len(valid_keypoints), 2), dtype=np.float32)
        landmarks_2d[:,0] = landmarks_3d[:,0] * w + x1
        landmarks_2d[:,1] = landmarks_3d[:,1] * h + y1

        # Clip
        landmarks_2d[:,0] = np.clip(landmarks_2d[:,0], 0, self.img_width-1)
        landmarks_2d[:,1] = np.clip(landmarks_2d[:,1], 0, self.img_height-1)

        return landmarks_2d

    def postprocess(self, coeffs, face_box):
        vertices = self.reconstruct_vertices(coeffs)
        return self.project_to_2d(vertices, face_box)

# -----------------------------
# SCRFD inference
# -----------------------------
scrfd_input_q = queue.Queue()
scrfd_output_q = queue.Queue()
hef_scrfd = "models/scrfd_2.5g.hef"

hailo_scrfd = HailoAsyncInference(
    hef_path=hef_scrfd,
    input_queue=scrfd_input_q,
    output_queue=scrfd_output_q,
    batch_size=1,
    input_type='UINT8',
    send_original_frame=True
)

img_list = load_images_opencv("person.jpg")
if not img_list:
    raise FileNotFoundError("Could not load 'person.jpg'.")
img = img_list[0]
resized_img = resize_image(img, (640, 640))

scrfd_input_q.put((resized_img, [resized_img]))
scrfd_input_q.put(None)
hailo_scrfd.run()
original_frame, scrfd_outputs = scrfd_output_q.get()

qp = {
    "scrfd_2_5g/conv43": (0.01984906569123268, 4.0),
    "scrfd_2_5g/conv42": (0.003921568859368563, 0.0),
    "scrfd_2_5g/conv50": (0.031789254397153854, 0.0),
    "scrfd_2_5g/conv49": (0.003921568859368563, 0.0),
    "scrfd_2_5g/conv56": (0.029924657195806503, 1.0),
    "scrfd_2_5g/conv55": (0.003921568859368563, 0.0),
}

deq_scrfd = {}
for k, arr in scrfd_outputs.items():
    if k not in qp:
        continue
    scale, zp = qp[k]
    deq_scrfd[k] = (arr.astype(np.float32) - zp) * scale

post_scrfd = SCRFDBBoxPostProc(conf_thresh=0.35, nms_thresh=0.4, max_dets=300)
boxes, scores = post_scrfd.postprocess(deq_scrfd, img_shape=(640, 640))
print(f"SCRFD Detections: {len(boxes)}")

vis = draw_bboxes(resized_img.copy(), boxes, scores, conf_thresh=0.35)
del hailo_scrfd

# -----------------------------
# TDDFA inference
# -----------------------------
hef_tddfa = "models/tddfa_mobilenet_v1.hef"
tddfa_input_q = queue.Queue()
tddfa_output_q = queue.Queue()

hailo_tddfa = HailoAsyncInference(
    hef_path=hef_tddfa,
    input_queue=tddfa_input_q,
    output_queue=tddfa_output_q,
    batch_size=1,
    input_type='UINT8',
    send_original_frame=False
)

tddfa_processor = TDDFAProcessor(scale=0.08367272466421127, zero_point=146.0)
tddfa_processor.img_width = resized_img.shape[1]
tddfa_processor.img_height = resized_img.shape[0]

# --- Queue face crops ---
for box in boxes:
    x1, y1, x2, y2 = box.astype(int)
    x1, y1 = max(0, x1), max(0, y1)
    x2, y2 = min(resized_img.shape[1], x2), min(resized_img.shape[0], y2)
    face_crop = resized_img[y1:y2, x1:x2]
    if face_crop.size == 0:
        continue
    face_input = resize_face_for_tddfa(face_crop)
    tddfa_input_q.put([face_input])

tddfa_input_q.put(None)
hailo_tddfa.run()

# --- Collect outputs ---
num_shape = tddfa_processor.shape_basis.shape[1]
num_exp   = tddfa_processor.exp_basis.shape[1]

for idx, box in enumerate(boxes):
    tddfa_output = tddfa_output_q.get()
    landmarks_raw = np.array(tddfa_output[0]).flatten()
    coeffs_for_reconstruction = landmarks_raw[:num_shape + num_exp]
    # Dequantize after slicing
    coeffs_for_reconstruction = tddfa_processor.dequantize(coeffs_for_reconstruction)
    landmarks = tddfa_processor.postprocess(coeffs_for_reconstruction, box)
    draw_landmarks(vis, landmarks)

cv2.imshow("SCRFD + TDDFA Landmarks", vis)
cv2.waitKey(0)
cv2.destroyAllWindows()
1 Like
4

Hi Marc, did you get anywhere with this? — building eye-open/closed detection on Hailo-8L (baby monitor project) and need TDDFA’s landmarks for it. Did you ever crack the mean_face/pca_basis reconstruction?

5

Hi @theetails :waving_hand: (and @Marc_Jasner, since this is your thread) — I was stuck on the same thing some time ago, so I think I can help a little bit.

Short version: the [62] vector is not landmarks, and it is not [shape | exp] either. It is [12 pose | 40 shape | 10 exp], and the numbers come out normalised. So two steps are missing in the rewrite: de-normalise the 62 values first, then project the 3D vertices with the 12 pose numbers (a 3×4 R | offset matrix). Without the pose block and the projection there is no way to get pixels out of it.

The two things ChatGPT told you about are actually two different files, and one of them you don’t need to look for:

  • The rescale mean/std (62-d) — this is what de-normalises the raw output. It is hardcoded in Hailo’s own model-zoo postproc as TDDFA_RESCALE_PARAMS (hailo_model_zoo/core/postprocessing/face_landmarks_3d_postprocessing.py). I copy it below so you don’t have to search for it.
  • The BFM basis bfm_noneck_v3.pkl (the mean_face = u and the pca_basis = w_shp/w_exp). This one is license-gated: you sign the 3DDFA license and download it manually (Hailo’s code raises exactly that message). The .npy files you were loading are not the same basis, that is why the reconstruction looked wrong.

Here is a TDDFAProcessor that follows the model-zoo postproc step by step, you can paste it as it is:

import numpy as np, pickle

# 62-d rescale params, copied verbatim from Hailo's model-zoo postproc
# (face_landmarks_3d_postprocessing.py -> TDDFA_RESCALE_PARAMS)
PARAM_MEAN = np.array([3.4926363e-04,2.5279013e-07,-6.8751979e-07,6.0167957e01,-6.2955132e-07,5.7572004e-04,-5.0853912e-05,7.4278198e01,5.4009172e-07,6.5741384e-05,3.4420125e-04,-6.6671577e01,-3.4660369e05,-6.7468234e04,4.6822266e04,-1.5262047e04,4.3505889e03,-5.4261453e04,-1.8328033e04,-1.5843289e03,-8.4566344e04,3.8359607e03,-2.0811361e04,3.8094930e04,-1.9967855e04,-9.2413701e03,-1.9600715e04,1.3168090e04,-5.2591440e03,1.8486478e03,-1.3030662e04,-2.4355562e03,-2.2542065e03,-1.4396562e04,-6.1763291e03,-2.5621920e04,2.2639447e02,-6.3261235e03,-1.0867251e04,8.6846509e02,-5.8311479e03,2.7051238e03,-3.6294177e03,2.0439901e03,-2.4466162e03,3.6586970e03,-7.6459897e03,-6.6744526e03,1.1638839e02,7.1855972e03,-1.4294868e03,2.6173665e03,-1.2070955e00,6.6907924e-01,-1.7760828e-01,5.6725528e-02,3.9678156e-02,-1.3586316e-01,-9.2239931e-02,-1.7260718e-01,-1.5804484e-02,-1.4168486e-01], dtype=np.float32)
PARAM_STD  = np.array([1.76321526e-04,6.73794348e-05,4.47084894e-04,2.65502319e01,1.23137695e-04,4.49302170e-05,7.92367064e-05,6.98256302e00,4.35044407e-04,1.23148900e-04,1.74000015e-04,2.08030396e01,5.75421125e05,2.77649062e05,2.58336844e05,2.55163125e05,1.50994375e05,1.60086109e05,1.11277305e05,9.73117812e04,1.17198453e05,8.93173672e04,8.84935547e04,7.22299297e04,7.10802109e04,5.00139531e04,5.59685820e04,4.75255039e04,4.95150664e04,3.81614805e04,4.48720586e04,4.62732383e04,3.81167695e04,2.81911621e04,3.21914375e04,3.60061719e04,3.25598926e04,2.55511172e04,2.42675098e04,2.75213984e04,2.31665312e04,2.11015762e04,1.94123242e04,1.94522031e04,1.74549844e04,2.25376230e04,1.61742812e04,1.46716406e04,1.51156885e04,1.38700732e04,1.37463125e04,1.26631338e04,1.58708346e00,1.50770092e00,5.88135779e-01,5.88974476e-01,2.13278517e-01,2.63020128e-01,2.79642940e-01,3.80302161e-01,1.61628410e-01,2.55969286e-01], dtype=np.float32)

class TDDFAProcessor:
    def __init__(self, scale, zero_point, bfm_path="bfm_noneck_v3.pkl"):
        self.scale, self.zp = scale, zero_point
        bfm = pickle.load(open(bfm_path, "rb"))
        u     = bfm["u"].astype(np.float32)              # (3N, 1)
        w_shp = bfm["w_shp"].astype(np.float32)[:, :40]  # keep 40 shape dims
        w_exp = bfm["w_exp"].astype(np.float32)[:, :10]  # keep 10 exp dims
        kpt   = bfm["keypoints"].astype(int)             # the 68-landmark subset (204 idx)
        self.u_base     = u[kpt].reshape(-1, 1)          # (204, 1)
        self.w_shp_base = w_shp[kpt]                      # (204, 40)
        self.w_exp_base = w_exp[kpt]                      # (204, 10)

    def __call__(self, out62_uint8, roi_box, size=120):
        # 1) int8 -> float, your own output quant (scale / zero_point)
        p = (np.asarray(out62_uint8, np.float32).ravel()[:62] - self.zp) * self.scale
        # 2) de-normalise -- THIS is the step that was missing
        p = p * PARAM_STD + PARAM_MEAN
        # 3) split: 12 pose (R | offset) | 40 shape | 10 exp  (NOT [shape | exp])
        R      = p[:12].reshape(3, 4)[:, :3]
        offset = p[:12].reshape(3, 4)[:, 3:]
        shp    = p[12:52].reshape(-1, 1)
        exp    = p[52:62].reshape(-1, 1)
        # 4) reconstruct the 68 vertices and project them
        pts = R @ (self.u_base + self.w_shp_base @ shp + self.w_exp_base @ exp).reshape(3, -1, order="F") + offset
        # 5) similar_transform: flip y and map the 120 crop back to roi_box (image coords)
        pts[0] -= 1; pts[2] -= 1; pts[1] = size - pts[1]
        sx, sy, ex, ey = roi_box
        pts[0] = pts[0] * (ex - sx) / size + sx
        pts[1] = pts[1] * (ey - sy) / size + sy
        return pts[:2].T   # (68, 2) landmarks in image coordinates

roi_box is the face box in the same coordinate frame you draw on — in your script that is the SCRFD box [x1, y1, x2, y2] in the 640 image, not [0,0,w,h]. Also, 3DDFA works better when the crop is square and a bit padded, so making the SCRFD box square and expanding it a little before the 120 resize gives cleaner landmarks.

Where the old code went wrong, concretely: it sliced raw[:num_shape + num_exp] = raw[:50] and read it as [40 shape | 10 exp], but the first 12 numbers are the pose, so everything was shifted by 12. There was no de-normalisation, and project_to_2d scaled the vertices by the box w/h as if they were in [0,1] — but after the projection they are already in 120-crop pixels, so what you need is the flip-y + similar_transform above, not a w/h scale.

On the outputs: the HEF gives a single [62] tensor. That [120,120,3] you were seeing is not landmarks (it looks like the input frame coming back through the async wrapper) — just decode the [62].

I ran the model-zoo ONNX (tddfa_mobilenet_v1_120x120.onnx) with onnxruntime to check this: the output is one [1,62] vector and it comes out normalised, the first 12 raw values look like [0.0, 2.06, -0.15, 1.12, ...] and say nothing by themselves. After p*STD + MEAN the pose block becomes a reasonable affine — R at ~5e-4 scale and offset ≈ (90, 59) inside the 120 crop. The decode is host-side maths, so it is the same whether the 62 numbers come from the ONNX or from your dequantised HEF output.

For eye open/closed you mostly need the 6 eye points per side (indices 36–41 and 42–47 in the 68-point layout), so the EAR ratio on those is enough once the landmarks are in image coordinates. Hope this helps, and good luck with the baby monitor ..

1 Like