Relative Entropy

Let and be two distributions
on the same discrete alphabet .
The relative entropy is:

Additionally, we define

Lemma

For any and we have

with

Proof

Using Log-Sum Inequality:

with equality if and only if .
But then this constant has to be .