Why digital volume control degrades audio quality
Bits and dynamic range
Each bit of digital audio adds approximately 6.02 dB of dynamic range. A 16-bit file has 96 dB of theoretical range; a 24-bit file has 144 dB - more than the human ear can resolve in a single moment.
When you attenuate digitally by 6 dB, you effectively discard the least significant bit. At −12 dB you lose two bits. At −48 dB you have thrown away 8 bits - taking a 16-bit signal down to 8-bit quality. The noise floor rises with every step.
The fix
Use 24-bit or 32-bit output mode at the OS level and keep the digital volume at 100%. Control loudness at the analog stage - the amplifier's volume knob, the preamp, or a DAC with a hardware volume control.
With 24-bit OS output, you can attenuate digitally by 48 dB and still have 16-bit quality remaining. Many high-end DACs implement volume in the analog domain for exactly this reason. If your DAC has a volume knob, use it.