Forgive me if this is due to a misunderstanding of how the Hailo compilation works under the hood, but I’ve noticed that when I attempt to re-compile the same model (or a similar one) several times, it starts the context search from 1 each time. If we know (from our prior experience) that it will not fit in any less than 4 partitions, for example, shouldn’t there be an option to start at a certain number of contexts for the search? Perhaps as one of the Allocator Parameters?
This would save a lot of time during the compilation process. However, there might be a reason that this cannot be done, such as the next context number requiring information from the previous.
Thanks, and please keep me updated if you add an option for this.
Please have a look at the Hailo Dataflow Compiler User Guide section Automatic Model Script.
After the compilation process, in addition to the binary .hef file, the compiled HAR (Hailo ARchive) file is created. This HAR file contains the final compilation results, as well as the automatic model script (.auto.alls) file, that contains the exact instructions for the compiler for creating the same binary file (for the specific Dataflow Compiler version).
This model script can be used to compile the model again (from the corresponding quantized HAR file), for a quick compilation.
If I understand correctly, the auto.alls can help if you want to avoid the cost of re-allocation, since it stores the compilation and allocation decisions. Is it correct then to assume that if I retrain a model with the same architecture, but different weights, that I could re-use the auto.alls file from the previous run to speed up the allocation of this run?
However, this still doesn’t address the other problem of experimenting with similar, but not identical, architectures (such as changing the input resolution, for example). You may still know that it will not fit into a smaller number of contexts, but will be unable to re-use the existing auto.alls file from a previous run (I assume). Thus you’re stuck waiting for it to check that the lower context numbers will not work, before finding a valid partition.
When you change the architecture of a model, the Hailo Dataflow Compiler needs to search for a new solution for allocating the model across the available resources of the Hailo device. The compiler is designed to take many factors into account to maximize performance, without requiring any special commands.
If you are not concerned about the performance of the resulting HEF file and simply want the compiler to find a feasible solution as quickly as possible, you can enable context switching and reduce the optimization level to 0. This instructs the compiler to return the first feasible solution it finds.