Discrete Memoryless Source

A Source taking values in a discrete set
where all are identically distributed and independent
is called a Discrete Memoryless Source.
Sometimes it is also called a Bernoulli Source.

Theorem

A DMS satisfies Asymptotic Equipartition Property
with entropy equal to Mathematical Entropy

Proof

Let be the distribution of
and distribution of
Then:

Set , the Information Content of
and use Weak Law of Large Numbers for
to find:

Hence satisfies Asymptotic Equipartition Property
with entropy