Sparity-Adjusted DARE

[MonsterAzi]

It's as simple as making the density used in the fill value equal to the desired density divided by the current density.

[linux-leo]

@MonsterAzi I took a look at the code and I think get the Idea. But you calculate the sparsity of the vectors like so:

s = (tensor.count_nonzero() / tensor.numel()).item()

This is before any sort of sparsity is applied to the vectors, right? So how likely is it that there will be values in the tensor that are exactly zero? Wouldn't it make more sense to count the number of values below a certain threshold when taking their absolute value divided by the number of elements?

[MonsterAzi]

It's actually highly likely that there will be values that are exactly zero. The models that result from sparsification (all DARE and TIES models) have some amount of sparsification leftover (values set to exactly zero).