Adaround loss type

General
1

Hi, is it possible to run adaround with a loss function different from l2? I’m trying to optimize a clip based model, which outputs a shape of [1, 197, 768] and I believe cosine similarity would be a more appropriate loss function to use. Unfortunately in the DFC manual I didn’t see it as a parameter

2

ok solved :slight_smile: l2 is actually fine

1 Like
3

Hey @user155,

We are happy for you that it did work without the change.

This is how you do it:

Yes, you can change the loss function in Adaround to use cosine similarity instead of the default L2 loss.

To use cosine similarity in Adaround:

post_quantization_optimization(adaround, policy=enabled, def_loss_type=cosine)

Supported loss functions include:

  • cosine - Cosine similarity
  • ce - Cross-entropy
  • l2 - Mean squared error
  • l2rel - Relative L2 loss (default)

For layer-specific adjustments:

post_quantization_optimization(adaround, layers=["conv1", "fc1"], loss_types=[cosine, cosine])
4

Hi @omria , I didn’t check for adaround, but in the last docker you released (Jan 2025) the cosine loss is missing for the finetuning algorithm, even if it’s mentioned in the DFC documentation. I personally checked the source code and added it myself, and it’s now working. Just wanted to let you know…

If you tell me that it is available for adaround then I will try :slight_smile:

5

Hey @user155,

First of all, thank you for bringing this to our attention. I’ll move this to R&D and we’ll get it fixed.

Yes, from what I know, the cosine functionality is available for adaround.

6

Hi @omria ,

I get this when trying to run adaround with cosine loss, as you suggested.

ValidationError: 1 validation error for ModelOptimizationConfig
adaround -> def_loss_type
  extra fields not permitted; def_loss_type from line 3; command was post_quantization_optimization(adaround,policy=enabled,def_loss_type=cosine) (type=value_error.extra)

Also, the documentation doesn’t show the possibility of changing loss for adaround

7

Hey @user155,

You’re correct. I’ve discussed both of your issues with our R&D team, and they confirmed that both should be supported.
I’ve opened a Jira ticket and they will update the DFC to work with both configurations.
Hopefully in the next release.

Thanks for your feedback - we really appreciate it. Please feel free to share any other issues you encounter.