Conditional Entropy

For Random Variables and define the conditional entropy

where is Random Probability of Random Variable.
Expanding:

Entropy Chain Rule

Lemma

Joint Entropy satisfies

Proof

Corollary

with equality if and only if and are Independent.

Lemma

Proof

We expand in two different ways:

Thus