Recommendations for PC Specs for Training AI Models Compatible with Hailo-8, Jetson, or Similar Hardware (Computer Vision & Signal Classification)

General
1

Hey everyone,

I’m looking to build or buy a PC tailored specifically for training AI models for Computer Vision and Signal Classification that will eventually be deployed on edge hardware like the Hailo-8, NVIDIA Jetson, or similar accelerators. My goal is to create an efficient setup that balances cost and performance while ensuring smooth training and compatibility with these devices.

Details About My Needs

  • Model Training: I’ll be training deep learning models (e.g., CNNs, RNNs) using frameworks like TensorFlow, PyTorch, HuggingFace, and ONNX.
  • Edge Device Constraints: The edge devices I’m targeting have limited resources, so my workflow might includes model optimization techniques like quantization and pruning.
  • Inference Testing: I plan to experiment with real-time inference tests on Hailo-8 or Jetson hardware during the development phase.
  • Use Case: My primary application involves object detection (for work) and, at a later stage, signal classification. For both cases, recall is our highest priority (missed true positives are fatal). Precision is also important (We don’t, want false alarms, but better having some false alarms then missing an event)

Questions for Recommendations

  1. CPU: What’s the ideal number of cores, and which models would be most suitable?
  2. GPU: Suggestions for GPUs with sufficient VRAM and CUDA support for training large models?
  3. RAM: How much memory is optimal for this type of work?
  4. Storage: What NVMe SSD sizes and additional HDD/SSD options would you recommend for data storage?
  5. Motherboard & Other Components: Any advice on compatibility with Hailo-8 or considerations for future upgrades?
  6. Additional Tips: Any recommendations for OS, cooling, or other peripherals that might improve efficiency?

If you’ve worked on similar projects or have experience training models for deployment on these devices, I’d love to hear your thoughts and recommendations!

Thanks in advance for your help!

2

Your question is exactly what I wanted to ask once mine has been answered on my topic. (How many Hailo modules can be used simultaneously on a PC & how to do that.)

I will try to help to the best of my knowledge. Even if I find that you are rushing things ^^.

For the moment I have not yet seen a PC working fully under Hailo.
From what I understand on a PC you can only run 1 M.2 or USB card. (I have not yet seen an assembly with more, but I am still reading the entire forum.)

On Arduino/Raspberry modules i have seen assemblies of 4 Hailo and theoretically we could do 6. (From what I understand)

So I hope/dream to be able to do this:
-1 multi Gpu motherboard 4 gpu. At 5 pcie slots minimum. = 1200€
-A cpu capable of running such a motherboard = 1000€
200 GB of cpu ram > 2000€
Pcie:
-1 recovery graphics card in order to have an image 50€
-4 pcie>M.2 cards with 4 M.2 slots 400€
16 hailo in the M.2 slots (16*200= 3200€)

~ 6600€/$ =
16 hailo8 x 26 = 416 TFlops (no information on fp 16 or 32)
For 50 watts + pc.

Hailo modules will use the cpu ram as Vram (if i did not make any error), so 200 cpu ram ≈ 100-150 vram (It’s less efficient from what I understand, but I don’t know by how much).

Otherwise another system that really works at the moment, while remaining on a contained price =
4 7900XTX at 1000€

It works very well under linux and from 6-8 months under windows.
But i must admit that some models such as flux voluntarily boycott this company. ( I did not know why)

7400€ the pc = (fp 32 461= 240 TFlops) ( fp 16 4122= 480 TFlops.
For 500-800 watt4 = 2000-3200watt.
And under 244 = 96 GB of VRAM.

The 4090s do the same but are at ~2000.
Which brings the pc to 11,000€
But they are better compatible with Windows.

For NVIDIA jetsones… I don’t know enough about the subject.

And the legendary H100 at 20,000€ each.
200 TFlops (fp16) (techpowerup. Com my source)
50 TFlops (fp32)
And 80 GB of VRAM each.

The LLama 405B model requires 230 GB of vram.
So only 3 H100 can make it work…
But some motherboards can do 300gb of cpu ram… So Hailo could theoretically work on it too.
(If one day a motherboard finally ends up being and supports 4 parallel gpu & 300gb of ram)

Here is the state of the market according to my current little knowledge (imperfect).
If this could help you ^^.