Poisson Approximation

Suppose are binary Random Variables with each and let . Let

and let be its distribution. Then:

where is the Relative Entropy and is the Mathematical Entropy.

Proof

Let be Independent with each other following the Poisson Distribution:

and let

Now use Data Processing Property of the Relative Entropy to find:

where

Corollary

If as above are IID with then:

for all , where is Total Variation Distance.

Proof

Apply Pinsker’s Inequality.

Corollary

where the supremum is taken over all that are independent Bernoulli with