Joint Entropy

Let and be Random Variables
taking values in and respectively.
The joint entropy of and is

where is the Information Content.
Expanding:

where is their joint distribution.

Lemma

with equality if and only if and are Independent

Proof

and
, ,
Apply Gibbs’ inequality with distributions and

i.e.

Equality for all and are independent.