Vector order is different between parse-hef and pipeline.get_input_vstreams()

General
,
1

Hello.

I’ve noticed discrepancy in the order of input/output vectors between the output of hailoclirt parse-hef <model>.hef and of hailort::InferVStreams.get_input_vstreams().

For example, on the tiny_yolov4_license_plates.hef model I get this from hailortcli parse-hef

Architecture HEF was compiled for: HAILO8
Network group name: tiny_yolov4_license_plates, Single Context
    Network name: tiny_yolov4_license_plates/tiny_yolov4_license_plates
        VStream infos:
            Input  tiny_yolov4_license_plates/input_layer1 UINT8, NHWC(416x416x3)
            Output tiny_yolov4_license_plates/conv19_centers UINT8, NHWC(13x13x6)
            Output tiny_yolov4_license_plates/conv19_scales UINT8, NHWC(13x13x6)
            Output tiny_yolov4_license_plates/conv19_obj UINT8, NHWC(13x13x3)
            Output tiny_yolov4_license_plates/conv19_probs UINT8, NHWC(13x13x3)
            Output tiny_yolov4_license_plates/conv21_centers UINT8, NHWC(26x26x6)
            Output tiny_yolov4_license_plates/conv21_scales UINT8, NHWC(26x26x6)
            Output tiny_yolov4_license_plates/conv21_obj UINT8, NHWC(26x26x3)
            Output tiny_yolov4_license_plates/conv21_probs UINT8, NHWC(26x26x3)

and from the following example code

for (const auto &input_vstream : (*pipeline).get_input_vstreams())
{
	auto frame_size = input_vstream.get().get_frame_size();
	uint8_t *ptr = func(frame_size); //allocate memory
	input_data_mem_views.emplace(input_vstream.get().name(), hailort::MemoryView((void *)ptr, frame_size));

	std::cout << "Adding input layer " << input_vstream.get().name() << " of size " << frame_size << std::endl;

	current_input_vector++;
}

for (const auto &output_vstream : (*pipeline).get_output_vstreams())
{
	auto frame_size = output_vstream .get().get_frame_size();
	uint8_t *ptr = func(frame_size); //allocate memory
	input_data_mem_views.emplace(output_vstream.get().name(), hailort::MemoryView((void *)ptr, frame_size));

	std::cout << "Adding input layer " << output_vstream .get().name() << " of size " << frame_size << std::endl;

	current_input_vector++;
}

I get

Adding input layer tiny_yolov4_license_plates/input_layer1 of size 519168
Adding output layer tiny_yolov4_license_plates/conv19_centers of size 1014
Adding output layer tiny_yolov4_license_plates/conv19_obj of size 507
Adding output layer tiny_yolov4_license_plates/conv19_probs of size 507
Adding output layer tiny_yolov4_license_plates/conv19_scales of size 1014
Adding output layer tiny_yolov4_license_plates/conv21_centers of size 4056
Adding output layer tiny_yolov4_license_plates/conv21_obj of size 2028
Adding output layer tiny_yolov4_license_plates/conv21_probs of size 2028
Adding output layer tiny_yolov4_license_plates/conv21_scales of size 4056

This is causing problems in my application because I have one process using hailortcli to figure how the total space needed for the input/output vectors, allocating their memory and passing pointers to those vectors to the process that actually posesses the vDevice and performs the inference. This obviously causes issues has the inference process goes out of bound, in this case on the last vector.

My original approach was to temporarily create a vDevice to use get_output_vstreams to load the vectors dimensions, but I can’t do that as the HAILO8 device is already taken by the inference process. I don’t have the hailort-service, as I really dont need it, I only have one inference process.

Why is there this discrepancy between hailortcli parse-hef and how the vectors are ordered in the std::map of MemoryView?

Thanks.

2

Hey @Stefano_Nicolis ,

The ordering difference you’re seeing is because these two methods use completely different sorting logic:

hailortcli parse-hef shows the VStreams in the exact order they’re defined in the HEF metadata. For your model that’s:

conv19_centers, conv19_scales, conv19_obj, conv19_probs,
conv21_centers, conv21_scales, conv21_obj, conv21_probs

get_output_vstreams() in the C++ API returns them in alphabetical order because internally it’s using a std::map<std::string, hailo_vstream_params_t> which automatically sorts keys lexicographically. So you get:

conv19_centers, conv19_obj, conv19_probs, conv19_scales,
conv21_centers, conv21_obj, conv21_probs, conv21_scales

If you need HEF order in your code, use:

auto infos = hef.get_output_vstream_infos();  // returns vector in HEF order

Or grab the names from there first, then look them up in your pipeline results to reorder as needed.

1 Like