Probability Review

@NainAKNain-2022

The study of probability (and most of statistics) revolves around “containers of information” that can hold different contents based on different associated events.

Random variables

Containers of random values. Can be discrete (e.g. coin flip) or continuous (e.g. height of a person). Notation: is a random variable, is a value of .

Independence

Two random variables and are independent if the value of does not affect the value of and the value of does not affect the value of . Notation: and are independent if .

Expectation

The expected value of a random variable is the average value it takes on over all possible values. Notation: .

Variance

The variance of a random variable is the expected value of the square of the difference between and its expected value, i.e., how much the maximum value of the random variable deviates from the expectation in the long run. Notation:

Distribution

A distribution is a function that gives the probability of a random variable taking on a particular value . Notation: is the probability of taking on the value .

Uniform distribution

The uniform distribution is a distribution where all events are equally likely.

Binomial distribution

The binomial distribution is a distribution that models the number of success in binary trials. Notation: , where is the probability of success in each trial.

Normal/Gaussian distribution

The normal distribution is a distribution that models the distribution of random variables that are normally distributed. Notation: , where is the mean, is the standard deviation, and is the variance.

Probability density function

The probability density function is a function that gives the probability of a random variable taking on a particular value .

For a real-valued random variable , the probability density function is:

For a standard normal distribution (), the probability density function is .

Multivariate normal distribution

Multivariate normal distribution is the extension of univariate normal distribution to the case where we deal with a real-valued vector input instead of a single real-valued random variable. Like in the univariate case, the multivariate normal distribution has associated parameters representing the mean vector and representing the covariance matrix.

This is the probability density function for multivariate case:

• Random variable is a -dimensional vector
• Mean is a -dimensional vector$
• Covariance matrix is a matrix

Covariance

Let’s say and are two random variables. The covariance of and is:

i.e., the covariance is difference between the expectation of the product of and and the product of their individual expectations.

Covariance tells us:

1. Covariance gives a sense of how much two random variables as well their scales are linearly related
2. Covariance captures only linear dependence and gives no information about other kind of relationships
3. If the sign of the covariance is positive, then both variables tend to take on relatively high values simultaneously. If the sign of the covariance is negative, then one variable tends to take on a relatively high value at the times that the other takes on a relatively low value and vice versa.
4. If two variables and are independent, then but the reverse is not true. Why? Because covariance doesn’t take into account non-linear relationships

Covariance matrix

If is a -dimensional vector , then the covariance matrix is a matrix where the -th entry is the covariance between the two random variables of the vector and .

Also:

Isotropic Gaussian distribution

An isotropic Gaussian distribution is a distribution where the covariance matrix is:

• the identity matrix
• is the scalar variance

This form is desriable because as the dimensions of a multivariate Gaussian grows, the mean follows a linear growth where the covariance matrix follows quadratic growth in terms of number of parameters. This quadratic growth isn’t computation friendly. A diagonal covariance matrix makes things much easier

A few things to note about isotropic Gaussian:

1. Eq. (4) represents a diagonal matrix multiplied by a scalar variance. This means that the variance along each dimension is equal. Hence an isotropic multivariate Gaussian is circular or spherical.
2. We discussed that doesn’t mean and are independent but if the distribution is multivariate normal and , it implies that and are independent.
3. If the multivariate distribution is isotropic, that means: i. Covariance Matrix is diagonal ii. The distribution can be represented as a product of univariate Gaussians i.e. ..

Conditional distribution