Multiple embeddings for a single face?

General
1

I am using the HailoGallery via json file for face recognition. At the moment the Tappas save_faces.sh script only creates one embedding per face. Is it possible to add multiple embeddings for a person’s face (for example different face angles etc) to maximize recognition is it only possible to have one embedding per face?

These are the recognition and gallery pipelines that I am using.

 recognition_pipeline = (
            f"hailonet hef-path={self.recognition_hef} scheduling-algorithm=1 vdevice-key={self.vdevice_key} ! "
           +QUEUE("recognition_pre_agg_q", max_size_buffers=30, leaky="no")
           +f"hailofilter function-name={self.recognition_function_name} so-path={self.recognition_so} name=face_recognition_hailofilter qos=false ! "
           +QUEUE("recognition_post_agg_q", max_size_buffers=30, leaky="no")
           +"agg2. agg2. ! "
        )

        gallery_pipeline = (
            f"hailogallery gallery-file-path={self.local_gallery_file} "
            +"load-local-gallery=true similarity-thr=.4 gallery-queue-size=20 class-id=-1 ! "
        )

Thank you.

2

Hey @M_S

To implement multiple embeddings for a single face in HailoGallery:

  1. Capture multiple images of the same person in different conditions.

  2. Modify the save_faces.sh script to generate embeddings for each image:

person_images=("face_1.jpg" "face_2.jpg" "face_3.jpg")
for img in "${person_images[@]}"; do
    ./generate_embedding --image $img --class-id 1 --output embedding_output.json
done
  1. Update the JSON gallery file to include multiple embeddings per person:
[
  {
    "class-id": 1,
    "embeddings": [0.12, 0.34, 0.56, ...],
    "image-path": "face_1.jpg"
  },
  {
    "class-id": 1,
    "embeddings": [0.22, 0.45, 0.67, ...],
    "image-path": "face_2.jpg"
  }
]

  1. Load the updated gallery file in your pipeline setup.

  2. Run face recognition as usual. The system will compare incoming faces against all embeddings for each person, improving recognition across various conditions.

This approach enhances recognition accuracy for individuals with varying facial appearances.

Would you like me to elaborate on any specific part of this process?

3

I would appreciate your input in regard to how to tell hailogallery to append new embeddings to a given person name instead of replacing them. Thank you.

4

The reason I ask this is because it seems like hailogallery itself is what populates the json file not the developer via python manually. I could be wrong about this though. Thanks.

5

Hey @M_S

Appending Embeddings for the Same Person in HailoGallery

HailoGallery doesn’t natively support appending new embeddings for the same person. However, you can implement this functionality by manually managing the gallery file. Here’s how:

Approach

  1. Modify your embedding generation script to append new embeddings instead of overwriting.
  2. Manually manage the gallery file (JSON format) to support multiple embeddings per person.

Implementation

Python Function to Append Embeddings

import json

def append_embeddings_to_gallery(new_embeddings, image_path, class_id, gallery_file):
    with open(gallery_file, "r") as f:
        gallery_data = json.load(f)

    person_found = False
    for person in gallery_data:
        if person["class-id"] == class_id:
            person["embeddings"].append(new_embeddings)
            person["image-paths"].append(image_path)
            person_found = True
            break

    if not person_found:
        gallery_data.append({
            "class-id": class_id,
            "embeddings": [new_embeddings],
            "image-paths": [image_path]
        })

    with open(gallery_file, "w") as f:
        json.dump(gallery_data, f, indent=4)

# Usage example
append_embeddings_to_gallery([0.12, 0.34, 0.56, 0.78], "face_angle_3.jpg", 1, "gallery.json")

Resulting JSON Structure

[
  {
    "class-id": 1,
    "embeddings": [
      [0.12, 0.34, 0.56, ...],
      [0.22, 0.45, 0.67, ...],
      [0.32, 0.55, 0.77, ...]
    ],
    "image-paths": [
      "face_angle_1.jpg",
      "face_angle_2.jpg",
      "face_angle_3.jpg"
    ]
  }
]

Integration with HailoGallery

Load the updated gallery file into your pipeline:

gallery_pipeline = (
    f"hailogallery gallery-file-path={self.local_gallery_file} "
    "load-local-gallery=true similarity-thr=.4 gallery-queue-size=20 class-id=-1 ! "
)

Considerations

  • Test face recognition performance with the new multi-embedding structure.

This approach allows you to store and utilize multiple embeddings per individual, potentially improving recognition accuracy across various conditions.

Regards

6

Thank you I will try this. Can you tell me why you said: “Ensure HailoGallery supports the updated JSON structure with multiple embeddings per person.” ?

I am confused since you just told me how the JSON file should look.

Am I misunderstanding something? Thanks a lot.

7

Hey @M_S,

My apologies for the confusion. To clarify: please ensure you’re using the correct JSON structure I provided earlier for multiple persons. This format is crucial for handling data on multiple individuals. If you need me to review the structure or have any questions, just ask.

8

Thank you I will give that format a try. Can you help me understand why the format you are suggesting is so different than the default format from the examples?

The format from the examples looks like this:

[{
    "FaceRecognition": {
        "Name": "Johny",
        "Embeddings": [
            {
                "HailoMatrix": {
                    "width": 1,
                    "height": 1,
                    "features": 512,
                    "data": [
                        -0.048001375049352649,
                        -0.048001375049352649,
                        -0.04061654582619667,
                        .....rest of embedding.....
                        ]
                   }
              }
            ]
         }
     }
]

Another question I have is, why does the format that you suggest require the path for the original pictures?

Thank you.

9

Hey @M_S,

Thanks for bringing up those differences in the JSON formats. You’re right, and I should have been clearer. Let me break it down:

  1. JSON Structure:
    My initial suggestion was a simplified approach for manually managing multiple embeddings per person. It included “image-path” to track which images generated each embedding. This can be helpful if you’re handling various conditions or angles manually.

But here’s the thing - if you’re using HailoGallery’s internal processes, stick with their format. It’s designed for better compatibility with their system.

  1. Picture Path:
    I included “image-path” in my initial suggestion to help track which image corresponds to each embedding. It’s useful when dealing with different face angles or conditions manually.

However, since HailoGallery manages embeddings internally, you don’t need to include the image path. You can safely omit it if you’re relying on the system’s built-in embedding management.

  1. Embedding Structure:
    The “HailoMatrix” structure you showed is HailoGallery’s format for representing embeddings. It includes crucial info like dimensions (width, height, features) and the actual embedding data. This format is essential for HailoGallery to process embeddings correctly.

Bottom line:

  • For manual embedding management, you could use the simpler format I initially suggested.
  • For full compatibility with HailoGallery, use the “HailoMatrix” structure in your JSON (like in your example) and drop unnecessary fields like “image-path”.

Hope this clears things up! Let me know if you need any more info.

10

Thank you. This is what I have done. Please let me know if it is a bad idea or whether I am doing the right thing.

RECOGNITION_PIPELINE = (
        f'hailocropper so-path={CROPPER_SO} function-name=face_recognition internal-offset=true name=cropper2 '
        f'hailoaggregator name=agg2 '
        'cropper2. ! '
            'queue name=bypess2_q leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! '
        'agg2. '

        'cropper2. ! '
            'queue name=pre_face_align_q leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! '
            f'hailofilter so-path={FACE_ALIGN_SO} name=face_align_hailofilter use-gst-buffer=true qos=false ! '
            'queue name=detector_pos_face_align_q leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! '
            f'hailonet hef-path={recognition_hef} scheduling-algorithm=1 vdevice-key={vdevice_key} ! ' 
            f'hailofilter function-name={recognition_post} so-path={RECOGNITION_POST_SO} name=face_recognition_hailofilter qos=false ! '
            'queue name=recognition_pre_agg_q leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! '
            'tee name=recognition_tee '
        
        # First branch of the tee
        'recognition_tee. ! '
            'queue name=recognition_post_q leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! '
        'agg2. '

        # Second branch of the tee
        'recognition_tee. ! '
            'hailopython module=post_k.py '

        'agg2. '
    )

Basically as soon as I get an embedding I use 'tee name=recognition_tee ' to send that embedding into my post_k.py custom post processing. There I manually perform cosine similarity to find whether the face matches one in my custom faces.json file.

By the way, I am running this on a Raspberry Pi 5 with libcamerasrc using the Pi camera. I have noticed that I cannot achieve 1280 x 720 resolution for sustained periods of time or I will randomly get segmentation faults. Not sure why but 800x600 seems to be more stable.

Thanks for any insight you can provide.

11

Sorry me reply above was for you but I forgot to click Reply! Thanks

12

Any input you can provide will be greatly appreciated

13

Hi @omria

Is there something wrong with my followup questions? Haven’t heard from you in a while.

Thank you

14

Hey @M_S

Here’s my analysis of your pipeline design and suggestions for addressing your concerns:

Your approach using tee to split the pipeline and handle embeddings in post_k.py is definitely valid and efficient. It provides good flexibility for implementing cosine similarity with your custom faces.json file.

A few key points to ensure optimal performance:

  1. For your embedding comparisons:
  • Remember to normalize embeddings before calculating cosine similarity
  • When dealing with multiple embeddings per person in faces.json, compare against all embeddings and take the highest match score
  • Consider batching similarity checks if you’re processing many detections
  1. Regarding your segmentation faults at 1280x720 resolution: The issues you’re seeing on the Raspberry Pi 5 are likely related to memory constraints. Here are some solutions:
  • Adjust GStreamer’s buffer settings for larger frames
  • Utilize GPU-accelerated elements where possible
  • Monitor memory usage with tools like htop
  • If needed, you could reduce the frame rate while keeping the higher resolution
  • Double-check your Pi camera drivers are up-to-date

Your current workaround of using 800x600 resolution is perfectly reasonable if it meets your requirements. The pipeline design using tee is solid - it allows you to branch the pipeline without duplicating work and keeps your recognition system modular.

Let me know if you need more specific guidance on any of these points!