Pinsker's Inequality

Relative Entropy and Total Variation Distance satisfy:

Proof

First suppose and are both Bernoulli with probabilities and respectively. WLOG assume (if not, then swap and ) Let

Fix and consider all . Since , it suffices to show

for all .

Now in the general case Let and consider . Then and . where and . Then by the binary case

Now apply Data Processing Property of the Relative Entropy and note

to find