09 - Properties of Expectation

Properties of Expectation

• Proposition: If $P(a\le X\le b)=1$ then $a\le EX\le b$

Expectation of Sums of RV

• X and Y have joint probability mass function (discrete RV) p(x,y)
  • $E[g(X,Y)]={\sum }_y{\sum }_{x}g(x,y)p(x,y)$
• X and Y have joint probability density function (continuous RV) f(x,y)
  • $E[g(X,Y)]={\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }g(x,y)f(x,y)dxdy$
• BSKT:
  • cdf: cumulative distribution function: probability that a random variable (continuous or discrete) take on value $\le$ certain value
• $X\ge Y$ for random var X and Y ⇒ $E[X-Y]\ge 0$ ⇒ $EX\ge EY$
• If $E[X_i]$ is finite for all $i=1,\cdots ,n$ ⇒ $E[X_1+\cdots +X_n]=E[X_1]+\cdots +E[X_n]$ (expectation of all $X_i$ is sum of expectation of each $X_i$)
• The sample mean:
  • $X_i,\cdots ,X_n$ is sequence of independent + identically distributed RV
  • distribution function F, expected value $\mu$
  • sample mean: $\bar{X}={\sum }_{i=1}^n\frac{X_i}{n}$
  • Expectation: $E\bar{X}=\mu$
• Expectation of a binomial random variable
  • X is binomial random variable, parameters $n$ and $p$
  • $X=X_1+\cdots +X_n$, where $X_i=\{\begin{array}{ll}1& \mathrm{if}\mathrm{the}\mathrm{ith}\mathrm{trial}\mathrm{is}\mathrm{a}\mathrm{success}\\ 0& \mathrm{otherwise}\end{array}$
  • $X_i$ is Bernoulli RV (see Bernoulli from lec 06), with $E[X_i]=p$ ⇒ $E[X]=E[X_1]+\cdot \cdot \cdot +E[X_n]=np$

Covariance, Variance of Sums, and Correlations

• BSKT: Expectation calculation
  • DRV: $E[X]={\sum }_{}^{}xp(x)$
  • CRV: $E[X]={\int }_{-\infty }^{\infty }x\cdot f(x)dx$
• X and Y independent → just like product rule, for functions h and g
  • $E[g(X)h(Y)]=E[g(X)].E[h(Y)]$
• Covariance: give info on relationship btw random vars
  • $Cov(X,Y)=E[(X-E[X])(Y-E[Y])]=E[XY]-E[X]E[Y]$
  • If X and Y independent ⇒ $Cov(X,Y)=0$; the converse is not true
• Covariance proposition:
  • $\mathrm{Cov}(X,Y)=Cov(Y,X)$ → cov is equal to itself
  • $\mathrm{Cov}(X,Y)=\mathrm{Var}(X)$ → cov is equal to var of x
  • $\mathrm{Cov}({\sum }_{i=1}^n{X}_i,{\sum }_{j=1}^m{Y}_j)={\sum }_{i=1}^n{\sum }_{j=1}^m\mathrm{Cov}(X_i,Y)$ → cov of sum = sum of cov
• If $Y_j=X_j,j=1,...,n$
  • $\mathrm{Var}\left({\sum }_{i=1}^n{X}_i\right)={\sum }_{i=1}^n\mathrm{Var}\left(X_i\right)+2{\sum }_{}{\sum }_{i<j}\mathrm{Cov}(X_i,X_j)$
• If $X_1,\cdots ,X_n$ are pairwise independent ($X_i$ and $X_j$ independent for $i\ne j$)
  • $Var({\sum }_{i=1}^n{X}_i)={\sum }_{i=1}^{n}Var(X_i)$ → var of sum = sum of var
• Sample variance, variance of sample mean, expectation of sample variance
  • $X_i,\cdots ,X_n$ is sequence of independent + identically distributed RV, expected value $\mu$, variance ${\sigma }^2$, sample mean $\bar{X}$ ⇒ $S^2={\sum }_{i=1}^n\frac{(X_i-\bar{X}{)}^2}{n-1}$
• Variance of Binomial RV, para n and p
  • $X=X_1+\cdots +X_n$, where $X_i$ are independent Bernoulli RV, such that $X_i=\{\begin{array}{ll}1& \mathrm{if}\mathrm{the}\mathrm{ith}\mathrm{trial}\mathrm{is}\mathrm{a}\mathrm{success}\\ 0& \mathrm{otherwise}\end{array}$ ⇒ $\mathrm{Var}(X)=\mathrm{Var}(X_1)+\cdot \cdot \cdot +\mathrm{Var}(X_n)=np(1-p)$ as $Var(X_i)=p-p^2$ for all $i$
• Correlation of 2 RV, $\rho (X,Y)$: ⇒ $\rho (X,Y)=\frac{\mathrm{Cov}(X,Y)}{\sqrt{Var(X)Var(Y)}}$ ⇒ $-1\le \rho (X,Y)\le 1$
  • $\rho (X,Y)=1\mathrm{implies}Y=a+bX,\mathrm{where}b={\sigma }_y/{\sigma }_x>0$
  • $\rho (X,Y)=-1\mathrm{implies}Y=a+bX,b=-{\sigma }_y/{\sigma }_x<0$
  • Correlation coefficient: measure of degree of linearity btw X, Y
    • $\rho (X,Y)near1$ or $\rho (X,Y)near-1$ ⇒ high linearity btw X and Y
    • $\rho (X,Y)near0$ ⇒ absent linearity
    • $\rho (X,Y)=0$ ⇒ uncorrelated
    • $\rho (X,Y)>0$: Y increases when X increases
    • $\rho (X,Y)<0$: Y decreases when X increases
• Conditional Expectation
  • DRV: $E[X|Y=y]={\sum }_{x}xP(X=x|Y=y)$, for all y such that $p_Y(y)>0$
  • CRV: $E[X|Y=y]={\int }_{-\infty }^{\infty }x{f}_{X\mid Y}(x|y)dx$, where $f_{X|Y}(x|y)=\frac{f(x,y)}{f_Y(y)}$
  • BSKT:
    • $f_X(x)={\int }_{-\infty }^{\infty }f(x,y)dy$ → integrate w.r.t $y$
    • $f_Y(y)={\int }_{-\infty }^{\infty }f(x,y)dx$ → integrate w.r.t $x$
  • Conditional expectation satisfy properties of ordinary expectation
    • $E[g(X)|Y=y]=\{\begin{array}{l}\sum _{x}g(x)p_{X|Y}(x|y)\mathrm{in}\mathrm{the}\mathrm{discrete}\mathrm{case}\\ {\int }_{-\infty }^{\infty }g(x)f_{X|Y}(x|y)dx\mathrm{in}\mathrm{the}\mathrm{continuous}\mathrm{case}\end{array}$
    • $E\left[{\sum }_{i=1}^n{X}_i|Y=y\right]={\sum }_{i=1}^{n}E[X_i|Y=y]$
  • Computing Expectations by Conditioning
    • $E[X]=E[E[X|Y]]$
    • If Y is DRV: $E[X]={\sum }_{y}E[X|Y=y]P\{Y=y\}$
    • If Y is CRV, density $f_Y(y)$: $E[X]={\int }_{-\infty }^{\infty }E[X|Y=y]f_Y(y)dy$
• Computing Probabilities by Conditioning
  • A: arbitrary event; Indicator random variable X by $X=\{\begin{array}{ll}1& \mathrm{if}\mathrm{A}\mathrm{occurs}\\ 0& \mathrm{if}\mathrm{A}\mathrm{does}\mathrm{not}\mathrm{occur}\end{array}$ ⇒ $E[X]=P(A)$ ⇒ $E[X]Y=y]=P(A|Y=y)$, for RV Y
  • DRV Y: $P(A)={\sum }_{y}P(A|Y=y)P(Y=y)$
  • CRV Y: $P(A)={\int }_{-\infty }^{\infty }P(A|Y=y)f_Y(y)dy$
• Conditional Variance:
  • $\mathrm{Var}(X|Y)=E[(X-E[X|Y]{)}^2|Y]=E[X^2|Y]-(E[X|Y]{)}^2$
  • $E[Var(X|Y)]=E[X^2]-E[(E[X|Y]{)}^2]$
  • $\mathrm{Var}(E[X|Y])=E[(E[X|Y]{)}^2]-(E[X]{)}^2$
  • $\mathrm{Var}(X)=E[\mathrm{Var}(X|Y)]+\mathrm{Var}(E[X|Y])$