Feature Request: More Spotify like volume normalization

[JasonLG1979]

Librespot already has volume normalization which I would assume (hopefully) follows the ReplayGain spec since that's what Spotify uses. But unlike Spotify it seems to use gain reduction as it's clipping prevention method whereas Spotify uses limiting. There also seems to be nothing in the librespot docs about how to currently approximate Spotify's 3 different volume normalisation options.

From what I can tell to approximate the 3 Spotify volume options the args are:

Loud
--enable-volume-normalisation --normalisation-pregain 6

Normal (Default)
--enable-volume-normalisation --normalisation-pregain 3

Quiet
--enable-volume-normalisation --normalisation-pregain -5

The problem is that with gain reduction as the clipping prevention method setting a positive pregain value basically breaks volume normalization. A drop in the pregain of a track that would clip can possibly make for a huge drop in perceived volume compared to other tracks.

What I would like to see is a choice of clipping prevention methods one being a limiter like what Spotify uses (Threshold -1dB, Attack 5ms, Release 100ms [for bonus points you could make it a look-ahead limiter so the Attack would be 0]) and the other being the current gain reduction method.

It would also be nice to have a set of args that would directly map to the 3 Spotify presets, applying the appropriate pregain and using the limiter.

For reference:

Here is the ReplayGain spec:
http://wiki.hydrogenaud.io/index.php?title=ReplayGain_specification

This explains Spotify's definition of volume normalization and the specs of their limiter:
https://artists.spotify.com/faq/mastering-and-loudness#what-is-loudness-normalization-and-why-is-it-used

This explains the volume normalization options in the official clients:
https://artists.spotify.com/faq/mastering-and-loudness#can-users-adjust-the-levels-of-my-music

[JasonLG1979]

I'm also curious if the audio processing is currently done in 16 bit? I ask because I notice that the output of librespot is 16 bit. Lowering the gain of 16 bit audio by several dB with ReplayGain throws away bits. It would be advantageous audio quality wise to do audio processing in 24 or 32 bit mode and if the sound card will accept it just give it to them or if not truncate it to 16 bit.

[JasonLG1979]

Not accounting for noise shaping and other tricks 16 bits gets you a theoretical dynamic range of 96.33dB and 24 bit gets you 144.49dB so that would mean if you did processing in 24 bit mode you could lower the gain by up to 48.16dB before you had to start throwing away bits.

[JasonLG1979]

Another option of course is to do the gain reduction in hardware for sound cards that have a hardware volume control. Because that's basically all that your implementation of volume normalization does. Turn the volume up and down. The replay spec mentions that as an option. Doing it in hardware at least doesn't throw bits away.

[sashahilton00]

@JasonLG1979 iirc the audio Is processed in 16 bits, since the Spotify files are 44,100, 16bit. If you want to examine the processing logic and potentially change it to 24/32 then that could potentially be worth having. My concern is that it would probably want to have a usage flag as I imagine that 32 bit processing will put a strain on some of the more memory constrained devices that librespot supports.

Hardware based normalisation would be good to have, ideally we would just offload to the hardware where possible, otherwise fallback to a software implementation

[JasonLG1979]

iirc the audio Is processed in 16 bits, since the Spotify files are 44,100, 16bit.

That's not how lossy audio like vorbis works. The source file may have been 16 bit but in the process of converting it it was transformed into the frequency domain sorta like converting PCM to PWM. Lossy audio does not have a bit depth. The bit depth of the resulting PCM is decided by the decoder. I would think the decoder does it's work internally in at least 32 bit float if it's worth a crap anyway.

My concern is that it would probably want to have a usage flag as I imagine that 32 bit processing will put a strain on some of the more memory constrained devices that librespot supports.

It would use more memory but no more CPU really. All you're doing is bit shifting. If the decoder won't output anything but 16 bit basically you just pad the bottom 8 or 16 bits with zeros and then do your gain adjustment just like before, Except now you're not throwing away bits.

Hardware based normalisation would be good to have, ideally we would just offload to the hardware where possible, otherwise fallback to a software implementation

That would also have to imply fixed or softvol volume, as in librespot is the only thing that should be turning the hardware volume up or down.

[JasonLG1979]

Only outputting 16 bit also affects the quality of librespot's software volume implementation. The same thing happens when you turn the volume down in 16 bit mode. You're throwing away bits. It would be nice to have "lossless" software volume control also.

Turning S16_LE to S32_LE would be trivial I would think since i32 is a native rust data type. It should give you more than enough space to lower the volume to below the physical noise floor of a device before you have to start throwing away bits even with gain adjustment. S24_LE and S24_3LE might be a little tricky though.

[roderickvd]

I'm also curious if the audio processing is currently done in 16 bit? I ask because I notice that the output of librespot is 16 bit. Lowering the gain of 16 bit audio by several dB with ReplayGain throws away bits. It would be advantageous audio quality wise to do audio processing in 24 or 32 bit mode and if the sound card will accept it just give it to them or if not truncate it to 16 bit.

Interested in this point, I dug around the source code.

It seems that gain normalization is applied in 32 bit, then converted to 16 bit output:

librespot/playback/src/player.rs

Line 1098 in 7f705ed

*x = (*x as f32 * normalisation_factor) as i16;

Same for the software volume control:

librespot/playback/src/mixer/softmixer.rs

Line 42 in 7f705ed

*x = (*x as i32 * volume as i32 / 0xFFFF) as i16;

For the ALSA sink this is even done in 64 bit:

librespot/playback/src/mixer/alsamixer.rs

Line 168 in 7f705ed

new_vol = (((cur_vol - self.params.min) as f64 / self.params.range) * 0xFFFF as f64)

So while this does not answer your feature request, at least the volume controls seem to be in HQ order!

[JasonLG1979]

So while this does not answer your feature request, at least the volume controls seem to be in HQ order!

No. The audio is spit out as 16bit 44.1 by the decoder and then processed. Converting a 16bit int into 32bit float then doing some math on it and then converting it back to a 16bit int is in no way HQ and you gain nothing. You're still throwing away bits, Best case you're wasting time converting back and forth, worst case you're introducing rounding errors/distortion converting an int to a float and then back to an int.

The solution is to do the gain normalization in 24 or 32bit (or 64bit or whatever the decoder natively works in) during the decoding process and just leave it 24 or 32bit. That way you can still fit the whole 16bits inside the 24/32bits with room for gain normalization without throwing away bits.

[roderickvd]

You are absolutely right. Blame on me for missing that glaring point at such a late hour. Output should remain at high bit depth after processing, not casted back to 16 bit.

[sashahilton00]

Feel free to create a PR if you want to/have time to. I'd be curious to see if the difference is noticeable or if this ends up as more of a case of 'doing it properly'.

[JasonLG1979]

Feel free to create a PR if you want to/have time to.

It might be a while. I'd need to learn rust.

I'd be curious to see if the difference is noticeable or if this ends up as more of a case of 'doing it properly'.

The difference would certainly be measurable I would think, but with all things audio, depending on the person and/or audio gear it may or may not be preservable? IMHO It never hurts to do things right though.

[roderickvd]

Why 24 bit resolution matters for volume control and normalization is described here: http://archimago.blogspot.com/2019/02/musings-why-bother-with-24-bit-dacs.html

Dialing the volume down to -25 dB in 16 bit decreases dynamic range from 98,9 dBA (CD quality) to 73,7 dBA (3,7 dB higher than vinyl). In comparison, doing the same in 24 bit pretty much maintains CD quality at 96,6 dBA. This is within the 120 dB dynamic range of human hearing and so practically observable.

I am enthusiastic about investing my time in this for the ALSA and Rodio backends. It would mean:

• moving all internal audio operations to i32
• adding a command-line option for 16 (default), 24 or 32 bit output
• casting i32 audio to the configured output bit depth

For 24 bit output, the following looks promising: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=3d233fedc8ed595a1e88e815d23cd009

Is this something of interest?

[ashthespy]

No. The audio is spit out as 16bit 44.1 by the decoder and then processed. Converting a 16bit int into 32bit float then doing some math on it and then converting it back to a 16bit int is in no way HQ and you gain nothing. You're still throwing away bits, Best case you're wasting time converting back and forth, worst case you're introducing rounding errors/distortion converting an int to a float and then back to an int.

I was under the impression that the ogg stream from Spotify was encoded in 16bit 44.1 to begin with? Or do I misunderstand?
I am not well versed with these things -- but now that we use a new version of lewton it lets you prescribe what format you want to read the samples out as. So you could use read_dec_packet_generic instead of read_dec_packet_itl in

librespot/audio/src/lewton_decoder.rs

Lines 32 to 34 in ed20f35

	loop {
	match self.0.read_dec_packet_itl() {
	Ok(Some(packet)) => return Ok(Some(VorbisPacket(packet))),

[JasonLG1979]

I was under the impression that the ogg stream from Spotify was encoded in 16bit 44.1 to begin with?

Like I said before, the source audio maybe was 16/44.1 at some point but once converted to lossy audio it has no bit depth it purely exists in the time dominion. The decoder quantizes it to a bit depth when it's decoded to PCM. Internally the decoder more than likely uses a much higher bit depth (24, 32, 64 or whatever) during the decoding process to take advantage of a much lower digital noise floor for stuff like noise shaping and other tricks to help hide what was thrown away.

Everything else being equal if you don't do any volume/gain adjustments a 16 bit output is fine, but as soon as you start messing with the levels you want to go to at least 24bit for the extra expendable dynamic range.

[roderickvd]

That's true, it's encoded at 16 bit 44,1 kHz so that gives a dynamic range of 96,3 dB at 0 dBFS. Now if you go under 0 dBFS (such as when attenuating volume or applying negative replay gain) you are adjusting the magnitude of the encoded wave. For every 6 dB attenuation you lose 1 bit.

Intuitively: at one point in the signal is encoded at 65535 (maximum amplitude). This is encoded as 1111 1111 1111 1111. Now you halve the volume. The signal should then be 32767 (half amplitude). This is encoded as 0111 1111 1111 1111. You have just lost one bit of information to reconstruct the same signal.

This can be circumvented by taking the 16 bit Ogg Vorbis stream, padding it with 8 or 16 zeros to 24 or 32 bit, then do volume control and normalization on it and keep it at that bit depth. You now have 48 respectively another 96 dB of headroom to do volume control in without losing dynamic range.

Staying with the example, 1111 1111 1111 1111 padded to 32 bit is 1111 1111 1111 1111 0000 0000 0000 0000. Halving the volume makes it 0111 1111 1111 1111 1000 0000 0000 0000. No more information lost.

(This does not really concern the title of this issue, should we open a new one?)

[roderickvd]

I created a fork with 32-bit internal sample storage and 64-bit arithmetic for volume control and normalisation. So far the Rodio backend seems to be working on my Mac. Initial commit here, feedback welcome!

Working on ALSA next as well as a command-line option to specify output depth.

[DeLub]

Works like a charm! Thanks.

[roderickvd]

Right on. Good deal. It's not a super huge deal but not all hardware can do 32bit natively. You might want to have 32bit as an option and by default just truncate to keep the default behavior.

Yes I'll add a command-line option to set output bit depth to either 16, 32 or possibly 24 bit depth. I'm first making everything work on 32-bit, then working back from there.

As far as memory/cpu usage is concerned librespot is pretty light weight. I run it on a PiZero with absolutely no problem, so it will basically run on a potato, although I'm not sure what the weakest target device is? If you'd share your build I love to try it out and compare the footprints.

Great idea. However the RPi 0 is ARMv6 and my RPi 3B+ is ARMv7. I just tried to cross-compile but would need to setup an entire ARMv6 sysroot to link to the ALSA libraries. So if you could compile for yourself? It's really not hard:

sudo apt-get install build-essential libasound2-dev pkg-config git
curl https://sh.rustup.rs -sSf | sh
git clone --branch hi-res-volume-control https://github.com/roderickvd/librespot
cd librespot && cargo build --features alsa-backend

You should now have a working librespot binary in librespot/target/debug. On my RPi 3B+, the debug build is much more prone to suffer from buffer underruns. Compile with --release to get a leaner binary in librespot/target/release.

Looking forward to hearing your findings.

[JasonLG1979]

Yes I'll add a command-line option to set output bit depth to either 16, 32 or possibly 24 bit depth. I'm first making everything work on 32-bit, then working back from there.

32 bit linear (int) is the same as 24 bit in ALSA at least. It's a bit confusing.

Great idea. However the RPi 0 is ARMv6 and my RPi 3B+ is ARMv7. I just tried to cross-compile but would need to setup an entire ARMv6 sysroot to link to the ALSA libraries. So if you could compile for yourself? It's really not hard:

Right on. It takes FOREVER to compile anything on the PiZero (potato),lol!!! I'll check it out this weekend.

[roderickvd]

Yes I'll add a command-line option to set output bit depth to either 16, 32 or possibly 24 bit depth. I'm first making everything work on 32-bit, then working back from there.

32 bit linear (int) is the same as 24 bit in ALSA at least. It's a bit confusing.

That does confuse me. What I meant was setting S16, S32 or S24 as output. Then writing either the 16, 32 or 24 most significant bits. What is this 32 linear == 24 bits you are referring to?

[roderickvd]

Right on. It takes FOREVER to compile anything on the PiZero (potato),lol!!! I'll check it out this weekend.

Have you had the time and get it to compile?

Also in roderickvd@d6121c4 I have added a dynamic limiter! New command line options:

        --normalisation-method NORMALISATION_METHOD
                        Specify the normalisation method to use - [basic,
                        dynamic]. Default is dynamic.
        --normalisation-threshold THRESHOLD
                        Threshold (dBFS) to prevent clipping. Default is -1.0.
        --normalisation-attack ATTACK
                        Attack time (ms) in which the dynamic limiter is
                        reducing gain. Default is 5.
        --normalisation-release RELEASE
                        Release or decay time (ms) in which the dynamic
                        limiter is restoring gain. Default is 100.
        --normalisation-growth GROWTH
                        Growth factor influencing the steepness of the dynamic
                        limiter. Default is 1.0.

For the growth factor, values from 0.5 up to and including 2.0 seem to work well.

As test track I have mostly used Mahler's "Blumine" from "Masterpieces in Miniature" which has huge dynamic range. The dynamic limiter seems to work well for pregains up to around 6. Beyond it doesn't cope as well, hardly a surprise.

For reference listen to the way Spotify's limiter sounds at different pregains:
https://melodiefabriek.com/blog/spotify-loudness-norm/

[roderickvd]

Made some more progress tonight, successfully getting to compile with libvorbis, GStreamer and JACK audio.

Notes:

• Loading a track with libvorbis throws a attempted to zero-initialize type ``vorbisfile_sys::ov_callbacks``, which is invalid runtime error. I have yet to check if this is already the case on dev or is a regression caused by me. No issues with lewton.
• On my RPI 3B+ with Raspian 10, loading a track with Rodio throws a runtime error an error occurred on output stream: A backend-specific error has occurred: ALSA function 'snd_pcm_prepare' failed with error 'EBUSY: Device or resource busy'. Might by that my device is limited to 24 bits, however, using the ALSA backend this is automatically formatted to S24. Again I have to check if this is already the case on dev. No issues on Mac OS 11.2.1.
• I have only tested the Rodio, ALSA, pipe and subprocess backends. Please test the others and let me know.

As said on Gitter, I'll open a PR in a short while once I've gotten the final PulseAudio and PortAudio backends to compile. So I can squash after a few more nights of incremental development. Meanwhile I'll report here.

[Johannesd3]

I saw on crates.io that vorbis was last updated 4 years ago, while this panic was introduced in the recent rust version 1.48, so it's not surprising.

Since it's really outdated, is there anything against dropping libvorbis support?

[roderickvd]

I opened a PR to fix vorbis regardless: tomaka/vorbis-rs#20. Thanks for your earlier fix for libtremor which I basically copied and then updated crates.

[roderickvd]

@JasonLG1979 I think you might like roderickvd/librespot@1037108. I just changed volume normalisation so that is clips specific samples instead of reducing the overall gain.

[roderickvd]

So I've got a working prototype of a dynamic limiter with attack and decay time (not committed yet). And yeah, the sudden gain changes are pretty brutal. Definitely needs a knee. Any functions you can recommend?

[JasonLG1979]

Any functions you can recommend?

When compressing/limiting an entire mix generally you would start with a really soft knee something that looks like a 1/4 of a circle on a graph and if it sounds like it's pumping, making the track duck or sound like it's breathing make the knee harder until it goes away. You still want to catch fast transients though. With a threshold of -1dB and an attack of 5ms it's pretty tight as far as the limiter has a pretty short period of time to be in full effect.

[roderickvd]

I've got experience with filters but not limiters. I understand the convex function, the quarter circle. I assume with on the y axis the gain and on the x axis the time. The function would then look like (1-a) * normal_gain + a * limiter_gain where the factor a goes up from 0 to 1 as the knee becomes harder. Questions I now have:

1. Does the knee come into effect after the attack / decay time? (i.e. are they triggers)?
2. If so, what is a reasonable time for the knee to take full effect?
3. If not, does that mean Spotify goes from 0 to 1 for the hard knee in 5 ms, the back again from 1 to 0 in 100 ms?

[JasonLG1979]

I've got a little recording, mixing and mastering experience so I've used a few hardware and software compressor/limiters but I've never designed one. I'm not sure there's a wrong way to do any of this, I'd bet that different compressor/limiters do it different ways. I think it's going to be a matter of trying different things and comparing the objective effectiveness of the limiter, as in how effective is it at preventing clipping, and it's subjective quality as in how does it sound.

My all time favorite compressor/limiter is the Urei Universal Audio 1176LN. I actually owned one for a while, found it disassembled in a church's A/V room and bought it as a pile of parts on the spot. Took it home, replaced a fuse, a VU lamp, cleaned it up and put it back together. It sounded amazing. I'd say mimic the 1176.

[JasonLG1979]

I mean mimic it's curves, you'll never be able to mimic it's exact sound of course, being as it's an analog device.

[Johannesd3]

Would you like to create a work in progress PR? So it's easier to follow.

You're saying that you don't use read_dec_packet_generic because of interleaving. Are you aware of lewton::samples::InterleavedSamples<f32>?

[roderickvd]

Would you like to create a work in progress PR? So it's easier to follow.

Sure, will do later today. I didn't want to spam you with commits, which at this time are pretty frequent.

You're saying that you don't use read_dec_packet_generic because of interleaving. Are you aware of lewton::samples::InterleavedSamples<f32>?

I wasn't, thanks. It even seems the native type (the i16's are conversions). It has a lot going for it. The one downside being that this provides 24-25 bits of transparency instead of 32 bits. Which means that you're dipping dynamic range below CD quality when attenuating by 25 dB or more. Doing it in i32 maintains dynamic range all the way down.

What do you think?

[JasonLG1979]

Which means that you're dipping dynamic range below CD quality when attenuating by 25 dB or more.

The difference between 24 and 16 bit is 48dB (144 vs 96). 48dB is plenty IMHO and it would save conversion and be more compatible with a wider variety of sound cards. -48dB is pretty quite I'm not sure you'd notice the bits thrown away since they would be well under the physical noise floor any audio equipment.

[JasonLG1979]

Even REALLY good DACs can't do much better than -120dB.

[roderickvd]

Alright I'll refactor to f32.

My -25 dB stems from the fact that the ENOB of a DAC is most always lower than the input bit depth -- see http://archimago.blogspot.com/2019/02/musings-why-bother-with-24-bit-dacs.html for a pretty good discussion. Anyway it makes my idea w.r.t. 32 bit dynamic range moot.

If all goes well I'll open a PR later today or tomorrow. Really getting into this!

[JasonLG1979]

My -25 dB stems from the fact that the ENOB of a DAC is most always lower than the input bit depth

I know. You're lucky to get 20bits with a good DAC, and maybe 21 or 22bits of actual headroom from a really good DAC. That was my point, no current DAC as far as I know can even really actually do 24bits. You're doing really good to actually get 16bits from a cheap dongle type DAC.

It's not that the chips can't play a 32bit file it's that they're running up against a physical noise floor.

[kingosticks]

Maybe it's also worth clarifying with the maintainers if this code would be merged. It's starting to sound like something you'd expect to find in music player software. Does this still belong in a library for using Spotify or some other generic library for applying replay gain?

[roderickvd]

There are two discussions going on here.

First is storing samples in f32 instead of i16 so you don’t lose dynamic range when doing volume control and normalization. This work is basically ready for PR, minus a few enhancements I have in mind. On Gitter @sashahilton00 stated he’s OK to get this into main.

Second and the original topic of this discussion, is changing the existing volume normalization so it better mimics Spotify’s native behavior. This is a separate development and so warrants a separate PR.

The way I look at it, is that normalization and peak limiting is already implemented but in a very basic fashion. Might as well follow Spotify’s published parameters instead. That’s what I am working towards.

While at it, it’s easy enough to make the threshold, attack and decay command-line options for users to play with. But otherwise I won’t work on any special curves — I agree you should then pass through to some other software.

It’s good to get this out in the open.

[roderickvd]

I submitted PR #660, let's continue there.

[giggywithit]

OK let me state for starters that I am not a programmer but am an audio systems designer.
My DAC has hardware volume capability and I prefer it to any other control method.
In fact, if the software does not come with the capacity to control my hardware volume, I don't use it.
I prefer the purest path possible.
With my system I can hear the smallest differences in sound quality.
Players that use softvol and other dithering, multi-level sound processing, show up easily in comparisons.
Once you hear it clear, you will not want to listen any other way.

I realize that this topic is mostly about softvol control, but I'd just like to toss in my request for a true hardware path for those of us that have and would like to use spotify to it's best capacity. My system is end to end designed for best capacity - there are precisely zero capacitors in my signal chain and meticulously designed for zero ground loops. It will be elsewhere for the weakest link.

[roderickvd]

That's good to know, thanks.

[JasonLG1979]

For the sake of a reasonably non-scientific test I did a fresh install, dist-upgrade, installed rust via curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh with the default option, and used this script to configure my audio. I am currently compiling it. Once that's done I'll see how it runs with and without overclocking, if overclocking is even necessary. Without upsampling librespot only really hits the CPU for like 20 or 30% momentarily during song changes if memory serves me. So unless this adds substantial overhead I don't think it will be a problem.

[JasonLG1979]

@roderickvd you pushed a couple commits since I cloned and started to compile. The version I cloned does not work. I get errors about setting the format. ALSA does not seem to support float formats even running though dmix. I see in your commits since then you mention ALSA only working in 16bit linear mode.

aplay --dump-hw-params /usr/share/sounds/alsa/Front_Right.wav tells me that on my system at least that dmix will accept S16_LE S16_BE S24_LE S32_LE S32_BE S24_3LE so basically 16, 24, and 32bit linear.

If you're already doing 16bit is there a reason you can't do 32bit?

[JasonLG1979]

Really though I'd like to see a 24bit option also. Basically all but the cheapest DACs will do 24bit natively a lot more than will do 32bit float that's for sure.

[roderickvd]

Let's continue at #660.

[giggywithit]

No arguments here.... When lossless is just as available, I'll be going that way. For now I am satisfied with spotify. Whatever works the best it's what I'll be using, now and then.

…

On Fri, Mar 12, 2021, 11:14 AM Jason Gray ***@***.***> wrote: You realize that you will get quantization noise anyway because of Spotify's (or actually: Vorbis') compressed nature? If you *really* care about audio fidelity Spotify is not the service for you. Audio fidelity is not Spotify's selling point. It's meant to be "good enough". In my mind this is about matching the behavior of the official clients and making the best of what is available. It would be interesting to have ReplayGain also use the Alsa volume control, if available, instead of doing it in librespot software. It would be trivial but it would make it so that you couldn't adjust the volume without messing up the gain adjustment which would only be suitable for systems that control the volume later in the chain. This not only has a noise floor of -318 dB (which simply cannot be audible) but is also better than 99% of DACs doing volume control in "only" 32-bit hardware. Yep, the max dynamic range of any sound on Earth at sea level is 194 dB. Pushing all quantization noise 124 dB below that makes 64 bit digital volume control superior to any physical volume control in every measurable way. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#608 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIWGVS6ES3JECRH6FIAUKDTTDIVW7ANCNFSM4YDP3QYA> .