Stationary Source

A Source is stationary if for each , the distribution of is the same for all .

Theorem

If is a stationary source on a finite alphabet , then the Entropy Rate exists and is equal to

Moreover, we can express this as

Proof

Consider the sequence . It is bounded below by , and it is non increasing:

By the Entropy Chain Rule:

so the averages converge.