Characteristic Functions

Introduction

The characteristic function of a random variable corresponds exactly to its Fourier transform. If the random variable has a density function, then the Fourier transform of the density function is the characteristic function of the random variable. It seems simple enough, but turns out to be surprisingly helpful. We will delay demonstrating this fact until we have covered sufficient preliminary results.

Sometimes, the subscript is omitted and we simply write $\phi(t)$ if the random variable is clear from context. Moreover, we can immediately see that the characteristic function depends only on the distribution of $X$ , by the change of variables theorem.

On the surface, the characteristic function closely resembles the moment generating function. However, the addition of the imaginary unit $i = \sqrt{-1}$ gives it different properties. For starters, the characteristic function is always finite, since $|e^{itx}| = 1$ and so $|\phi_X(t)| \leq 1 < \infty$ , in contrast to the moment generating function which may be infinite for any $s \neq 0$ .

Like for moment generating functions, we have $\phi_X(0) = 1$ for any $X$ , and if $X$ and $Y$ are independent, then $\phi_{X+Y}(t) = \phi_{X}(t)\phi_Y(t)$ . Letting $\mu = \mathcal{L}(X)$ , we have

\htmlId{eq-1}{\begin{aligned} |\phi_X(t+h) - \phi_X(t)| &= \left| \int \left( e^{i(t+h)X} - e^{itX}\right) d\mu \right| \\ &\leq \int \left| e^{i(t+h)X} - e^{itX} \right| d\mu \\ &= \int \left| e^{itX} \right| \left| e^{ihX} - 1 \right| d\mu \\ &= \int \left| e^{ihX} - 1 \right| d\mu. \end{aligned}} \tag{1}

Now, as $h \to 0$ the bounded convergence theorem, (given that $\left| e^{ihX} - 1 \right| \leq 2$ ), states that the last expression goes to $0$ . We can therefore conclude that $\phi_X$ is always a (uniformly) continuous function (since the last expression does not depend on $t$ ).

Similarly, we can define the derivatives of $\phi_X(t)$ . It will be analogous to the derivation for the moment generating function, but we’ll need fewer constraints thanks to the facts we just showed.

The Continuity Theorem

The whole point of this chapter is to get a new tool. This tool is the fact that characteristic functions are unique, and you can (usually) back out the corresponding distribution. The implication, which is what has to be proven, is that sequences of characteristic functions correspond to sequences of distributions, and the limits match. In other words, we can work entirely in terms of characteristic functions, compute the limit, and then back out the corresponding distribution.

In order to prove this result, we’ll need some machinery. For the result to hold, we need to know that the function limit actually corresponds to the characteristic of some distribution. For this to happen, the limit of distributions has to be well-behaved, and the probability can’t escape to infinity. In this case, the limit of characteristic functions may exist, but not actually correspond to a distribution. Also, the limiting function has to correspond to a unique distribution. We’ll cover the notion of a “tight” sequence of distributions, Fourier inversion, uniqueness, and eventually the central limit theorem and the method of moments.

First, the following theorem

We need lemmas to prove this:

Proof.

First note by the triangle inequality that

\htmlId{eq-4}{\begin{aligned} \left| \frac{e^{-ita} - e^{-itb}}{it} e^{itx} \right| &= \left| \frac{e^{-ita} - e^{-itb}}{it}\right| = \left| \int_a^b e^{-itr} dr \right| \\ &\leq \int_a^b \left| e^{-itr} \right| dr = b - a. \end{aligned}} \tag{4}

Hence, we have

\htmlId{eq-5}{\begin{aligned} \int_{\mathbf{R}} \int^T_{-T} \left| \frac{e^{-ita} - e^{-itb} }{it} e^{itx} \right| dt \mu(dx) &\leq \int_{\mathbf{R}} \int_{-T}^T (b-a) dt \mu(dx) \\ &= \int_{\mathbf{R}} 2T(b-a) \mu(dx) = 2T(b-a). \end{aligned}} \tag{5}

$\blacksquare$

Moreover,

The proof follows from standard results from complex analysis, namely contour integration.

These results then allow one to prove the inversion theorem purely mechanically by substituting the exponential by its sine and cosine representation. The proof is ommitted for brevity. However, we note an important corollary:

Proof.

The reverse direction is straightforward. If the distributions are equal, then the expected values of all functions are the same.

The forward direction follows from our previous theorem. Basically, $\mathbf{P}(a \leq X \leq b) = \mathbf{P}(a \leq Y \leq b)$ for all $a, b$ as long as $\mathbf{P}(X = a \text{ or } b, \text{ or } Y = a \text{ or } b) = 0$ . This can only be violated for countably many $a, b$ . By taking limits and by the continuity of probabilities, $\mathbf{P}(X \in I) = \mathbf{P}(Y \in I)$ for all intervals $I \subseteq \mathbf{R}$ . Then by the uniqueness of extensions, $\mathcal{L}(X) = \mathcal{L}(Y)$ .

$\blacksquare$

This shows that at least the notion we want to show is well-defined. It’s plausible that the limits in characteristic functions ensure corresponding limits in distribution. However, this does not ensure that the limit actually exists. To prove this, we need a new fact, called the Helly selection principle.

The proof is tedious, so rather than state it formally, we’ll give a coarse statement with the proof idea. If we have a sequence of cumulative distribution functions $\left\{ F_n(x) \right\}$ , then there’s always a subsequence with limit function $F$ . The limit function $F$ is non-decreasing and right-continuous, and points $x$ where $F(x)$ is continuous have $\lim_{k \to \infty} F_{n_k}(x) = F(x)$ . However, the function may not start from $0$ and go to $1$ .

The general idea of the proof is to pick a countable set of support points, say the rationals, where we will define an auxiliary function $G$ . It can be shown by a diagonalization argument that you can find a subsequence such that $F_{n_k}$ converges for all points in the countable set. We call the limit function $G(q)$ for $q \in \mathbf{Q}$ , and then we pick

\htmlId{eq-7}{F(x) = \inf \left\{ G(q) \,:\, q \in \mathbf{Q}, x > q \right\}.} \tag{7}

This function satisfies the desired properties, which the reader should verify.

The additional constraint on the sequence we need is that of “tightness”. As foreshadowed, we have to disallow sequences of distributions where the probability mass escapes to infinity for the continuity theorem to hold.

We then have the following desireable theorem:

The proof consists of showing that the limit of CDFs $F$ is a proper CDF using the tightness condition. Then we can set $\mu((a, b]) = F(b) - F(a)$ which is indeed the weak limit.

To get a proper result for the continuity theorem, we need to know that the sequence of distributions is tight.

We can now finally prove the continuity theorem.

Proof.

First, suppose that $\mu_n \Rightarrow \mu$ . Since $\cos(tx)$ and $\sin(tx)$ are bounded continuous functions, we must have

\htmlId{eq-9}{\begin{aligned} \phi_n(t) &= \int \cos(tx) \mu_n(dx) + i \int \sin(tx) \mu_n (dx) \\ &\to \int \cos(tx) \mu(dx) + i \int \sin(tx) \mu (dx) \\ &= \phi(t). \end{aligned}} \tag{9}

Conversely, suppose that $\phi_n(t) \to \phi(t)$ for all $t \in \mathbf{R}$ . Then, by the previous Lemma, we must have the sequence of distributions be tight. In that case, there’s a subsequence that weakly converges to, say, $\nu$ . From the previous paragraph, we must have $\phi_{n_k}(t) \to \phi_\nu(t)$ . On the other hand, we know that $\phi_n(t) \to \phi(t)$ for all $t$ ; so $\phi_\nu = \phi$ . Fourier uniqueness implies that we must have $\nu = \mu$ .

We’ve therefore shown that $\mu$ is the only possible weak limit of the $\mu_n$ , and so the full sequence must weakly converge to $\mu$ , i.e. $\mu_n \Rightarrow \mu$ .

$\blacksquare$

The Central Limit Theorem

Let’s pause and figure out what we’ve learned and where we should go from here. First, we acquainted ourselves with characteristic functions of random variables. They have some nice properties, like being unique and invertible. Moreover, manipulating a characteristic function is often easier than manipulating the underlying measure.

As a result of these niceties, we’d really like it to be true that a sequence of distributions converges to $\mu$ if and only if the corresponding characteristic functions converge to $\mu$ ‘s characteristic function. In other words, we can work in characteristic function space, and go back at the end. It turns out this is true, as we showed.

With this new tool at hand, we’d do well to apply it. This is what we’ll do in this section, where we use the previous section’s results to prove the central limit theorem. Loosely, it states that a sum of i.i.d. random variables tends to the normal distribution. We’ll want to show that the characteristic functions of the partial sums tend to the characteristic function of the normal. The result then follows by the continuity theorem.

Let’s start by computing the characteristic function of $N(0, 1)$ .

\htmlId{eq-10}{\begin{aligned} \phi_X(t) &= \int_{-\infty}^\infty e^{itx} \frac{1}{\sqrt{2\pi }} e^{- \frac{1}{2}x^2} dx \\ &= \frac{ e^{-\frac{t^2}{2}}}{\sqrt{2\pi}} \int_{-\infty}^\infty e^{- \frac{1}{2} (x - it)^2} dx \\ &= e^{-\frac{t^2}{2}}. \end{aligned}} \tag{10}

We can now prove the central limit theorem:

Proof.

We can assume without loss of generality that $m = 0$ and $v = 1$ by replacing $X_i$ by $\frac{X_i - m}{\sqrt{v}}$ .

We know how to add random variables. Let $\phi_n$ be the characteristic function of $S_n / \sqrt{n}$ , and set $\phi(t) = \mathbf{E}(\exp(it X_1))$ . It suffices to show that $\phi_n \to e^{- \frac{t^2}{2}}$ .

\htmlId{eq-12}{\begin{aligned} \phi_n(t) &= \mathbf{E}\left[\exp\left(it \frac{S_n}{\sqrt{n}}\right)\right] \\ &= \mathbf{E}\left[\exp\left(i \frac{t}{\sqrt{n}} X_1 \right)\right]^n \\ &= \phi\left(\frac{t}{\sqrt{n}}\right)^n \\ &= \left[ 1 + i \mathbf{E}[X_1] \frac{t}{\sqrt{n}} - \frac{1}{2} \mathbf{E}[X_1^2] \frac{t^2}{n} + \cdots \right]^n \\ &= \left[ 1 - v \frac{t^2}{2 n} + o \left( \frac{1}{n} \right) \right]^n \\ &\to e^{- \frac{t^2}{2} + n o(1 / n)} \\ &= e^{- \frac{t^2}{2}}. \end{aligned}} \tag{12}

$\blacksquare$

There are several generalizations to the classical central limit theorem. For example, we can drop the i.i.d. assumption, or get additional results on the rate of convergence. We don’t give the results here, since they are digressions. However, we mention that with additional conditions on the third moment, it is possible to drop the assumption that the variables are identically distributed. See the Lyapunov Central Limit Theorem, or the Lindeberg Central Limit Theorem.

Method of Moments

We saw that we could look at the characteristic functions to check or find limits of distributions. However, there are other ways to do that and prove the central limit theorem. One such method, called the method of moments, looks only at the moments of the distributions.

Some questions arise in this approach, similar to those we had about convergence in the characteristic functions. How do we know that a convergence in the moments yields a unique distribution? When is a distribution uniquely defined by its moments? And, when is the limit and the distribution interchangeable in the moments?

It turns out that not all limits have the right properties. We’ll say that a distribution $\mu$ is determined by its moments if all its moments are finite and no other distribution has identical moments.

The proof is not particularly informative, but since the distributions have finite moments, they are tight. Since some subsequence must converge to some distribution, say $\nu$ , then $\nu = \mu$ since we can identify all the moments of $\nu$ with those of $\mu$ . For this fact, use Skorohod’s theorem.

Finally, we specify which distributions are uniquely defined by their moments.

Exercises

11.5.1

a) No. The probability mass escapes to $+\infty$ . Formally, for every $\epsilon > 0$ , if you pick an $a, b \in \R$ , at some point in the sequence the mass escapes from $[a, b]$ . Specifically, starting at $n = \lfloor b + 1 \rfloor$ .

b) No, no tightness so no theorems hold. The Helly Selection principle says nothing about the distributions converging to anything.

c) Yes, the Helly selection Principle says so. The function is given by

\htmlId{eq-13}{F(x) = 0.} \tag{13}

d) Same, but $F(x) = 1$ .

11.5.6

It must be tight.

Proof.

Suppose that $\mu_n \Rightarrow \mu$ . This also means that $F_n(x) \to F(x)$ . In other words, for every $\epsilon > 0$ , there exists $N$ such that for all $n > N$ ,

\htmlId{eq-14}{|F_n(x) - F(x)| < \epsilon / 2.} \tag{14}

For any $\epsilon >0$ there are $a, b \in \R$ such that

\htmlId{eq-15}{\mu([a, b]) \geq 1 - \epsilon.} \tag{15}

Similarly, there is an $N$ such that $|F_n(a) - F(a)| \leq \epsilon /4$ and $|F_n(b) - F(b)| < \epsilon /4$ for all $n > N$ . Now pick $[a, b]$ such that $\mu([a, b]) \geq 1 - \frac{\epsilon}{2}$ and $N$ as above, then

\htmlId{eq-16}{\begin{aligned} \mu_n([a, b]) &= F_n(b) - F_n(a) \\ & \geq F(b) - \frac{\epsilon}{4} - (F(a) + \frac{\epsilon}{4}) \\ &= \mu([a, b]) - \frac{\epsilon}{2} \\ &\geq 1 - \frac{\epsilon}{2} - \frac{\epsilon}{2} \\ &= 1 - \epsilon, \end{aligned}} \tag{16}

as desired. So the tail is tight, and whatever comes before $N$ is a finite set, and so is also tight. Therefore, the whole sequence is tight.

$\blacksquare$

11.5.9

Yes, it does. Intuitively, we know it’s the Dirac delta distribution at $0$ . Can we show it rigorously? The distributions are tight, that’s straightforward, so the continuity theorem holds.

We know that

\htmlId{eq-17}{\phi_n(t) = e^{-\frac{t^2}{2n}} \to 1.} \tag{17}

What’s the characteristic function of a Dirac delta?

\htmlId{eq-18}{\phi(t) = \int \exp(itx) d \delta_0 = \exp(it(0)) = 1.} \tag{18}

So $N(0, \frac{1}{n}) \Rightarrow \delta_0$ .

11.5.11

In each case we just sum or integrate $e^{itx}$ against the law of $X$ .

a) Binomial: by the binomial theorem,

\htmlId{eq-19}{\phi_X(t) = \sum_{k=0}^n \binom{n}{k} (pe^{it})^k (1-p)^{n-k} = \left( 1 - p + p e^{it} \right)^n.} \tag{19}

Then $\phi_X'(t) = n(1-p+pe^{it})^{n-1} i p e^{it}$ , so $\phi_X'(0) = inp = i \mathbf{E}(X)$ .

b) Poisson: the sum is an exponential series,

\htmlId{eq-20}{\phi_X(t) = e^{-\lambda} \sum_{k=0}^\infty \frac{(\lambda e^{it})^k}{k!} = e^{\lambda (e^{it} - 1)}.} \tag{20}

Then $\phi_X'(t) = i \lambda e^{it} \phi_X(t)$ , so $\phi_X'(0) = i\lambda = i \mathbf{E}(X)$ .

c) Exponential: since $\mathrm{Re}(\lambda - it) = \lambda > 0$ , the integral converges and

\htmlId{eq-21}{\phi_X(t) = \int_0^\infty \lambda e^{-(\lambda - it)x} dx = \frac{\lambda}{\lambda - it}.} \tag{21}

Then $\phi_X'(t) = \frac{i\lambda}{(\lambda - it)^2}$ , so $\phi_X'(0) = \frac{i}{\lambda} = i \mathbf{E}(X)$ .