Poisson Approximation

Suppose are binary Random Variables
with each and let .
Let

and let be its distribution. Then:

where is the Relative Entropy and is the Mathematical Entropy.

Proof

Let be Independent with each other
following the Poisson Distribution:

and let

Now use Data Processing Property of the Relative Entropy to find:

where

Corollary

If as above are IID with then:

for all , where is Total Variation Distance.

Proof

Apply Pinsker’s Inequality.

Corollary