Pinsker's Inequality

Relative Entropy and Total Variation Distance satisfy:

Proof

First suppose and are both Bernoulli with probabilities and respectively.
WLOG assume (if not, then swap and )
Let

Fix and consider all .
Since , it suffices to show

for all .

Now in the general case
Let and consider .
Then and .
where and .
Then by the binary case

Now apply Data Processing Property of the Relative Entropy and note

to find