Let and be Random Variables
taking values in and respectively.
The joint entropy of and is
where is the Information Content.
Expanding:
where is their joint distribution.
Lemma
with equality if and only if and are Independent
Proof
and
, ,
Apply Gibbs’ inequality with distributions and
i.e.
Equality for all and are independent.