Trellis quantization

Trellis quantization is an algorithm that can improve data compression in DCT-based encoding methods. It is primarily used to optimize motion estimation coefficients in lossy video compression codecs such as Xvid and H.264/AVC. Trellis quantization reduces the size of some DCT coefficients while recovering others to take their place. This process can increase quality because the recovered coefficients were calculated by Trellis to have a larger impact on quality than the ones removed. Trellis quantization effectively finds the optimal quantization for each macroblock to maximize the PSNR relative to bitrate. It has varying effectiveness depending on the input data and compression method.

References

VirtualDub/Xvid guide mentioning Trellis quantization FFMPEGx option documentation