# Category Archives for **Information theoretic methods**

## Lecture 10. Concentration, information, transportation (2)

Recall the main proposition proved in the previous lecture, which is due to Bobkov and Götze (1999). Proposition. The following are equivalent for : for every and with finite mean. for every probability measure . This result provides a characterization … Continue reading

## Lecture 9. Concentration, information, transportation (1)

The goal of the next two lectures is to explore the connections between concentration of measure, entropy inequalities, and optimal transportation. What is concentration? Roughly speaking, concentration of measure is the idea that nonlinear functions of many random variables often … Continue reading

## Lecture 8. Entropic cone and matroids

This lecture introduces the notion of the entropic cone and its connection with entropy inequalities. Entropic cone Recall that if is a discrete random variable with distribution , the entropy of is defined as Now let be (not … Continue reading

## Lecture 7. Entropic CLT (4)

This lecture completes the proof of the entropic central limit theorem. From Fisher information to entropy (continued) In the previous lecture, we proved the following: Theorem. If are independent random variables with , then (1) where are weights satisfying … Continue reading

## Lecture 6. Entropic CLT (3)

In this lecture, we complete the proof of monotonicity of the Fisher information in the CLT, and begin developing the connection with entropy. The entropic CLT will be completed in the next lecture. Variance drop inequality In the previous lecture, … Continue reading

## Lecture 5. Entropic CLT (2)

The goal of this lecture is to prove monotonicity of Fisher information in the central limit theorem. Next lecture we will connect Fisher information to entropy, completing the proof of the entropic CLT. Two lemmas about the score function Recall … Continue reading

## Lecture 4. Entropic CLT (1)

The subject of the next lectures will be the entropic central limit theorem (entropic CLT) and its proof. Theorem (Entropic CLT). Let be i.i.d. real-valued random variables with mean zero and unit variance. Let If for some , … Continue reading

## Lecture 3. Sanov’s theorem

The goal of this lecture is to prove one of the most basic results in large deviations theory. Our motivations are threefold: It is an example of a probabilistic question where entropy naturally appears. The proof we give uses ideas … Continue reading

## Lecture 2. Basics / law of small numbers

Due to scheduling considerations, we postpone the proof of the entropic central limit theorem. In this lecture, we discuss basic properties of the entropy and illustrate them by proving a simple version of the law of small numbers (Poisson limit … Continue reading

## Lecture 1. Introduction

What is information theory? The first question that we want to address is: “What is information?” Although there are several ways in which we might think of answering this question, the main rationale behind our approach is to distinguish information … Continue reading