Linear regression

Description:

• Prediction by Regression Model
• We might not know the PDF of $X$ and $Y$, but if we know the mean and standard variance, we can determine the best linear predictor of $Y$ with respect to $X$

Finding line of best fit

• 2 main methods:
  1. Scattergraph method:
     • Draw a line through data points with about an equal number of points above and below the line.
  2. Linear regression:
     • Using least squares method
     • Using correlation:
       • We need to choose a and b to minimize the Mean square error
       • $$\begin{gathered} (E[(Y-(a+bX){)}^2]=E[Y^2]-2aE[Y]-2bE[XY]+a^2+2abE[X]+b^{2}E[X^2] \\ ) \end{gathered}$$
       • Taking partial derivative and set them equal to 0 we have $a$ and $b$ when it is at minimum point
         • Cant have maximum due to the nature of the problem
       • $b=\frac{\text{Cov}(X,Y)}{{\sigma }_X^2}$
       • $a=E[Y]-bE[X]$
       • The best linear predictor (lowest mean square error) is: ${\mu }_y+\frac{\rho {\sigma }_y}{{\sigma }_x}(X-{\mu }_x)$
         • Happens when ${\mu }_y=E[Y],{\mu }_y=E[X]$ and $\rho$ is the Correlation of $X$ and $Y$
         • The Mean square error of this predictor is given by $E[(Y-{\mu }_y-\frac{\rho {\sigma }_y}{{\sigma }_x}(X-{\mu }_x{)}^2]={\sigma }_y^2(1-{\rho }^2)$
     • Using $$\begin{gathered} (y=a+bX \\ ) \end{gathered}$$
       • $$\begin{gathered} (b=\frac{n\sum x_i{y}_i-\sum x_i\sum y_i}{\sqrt{n\sum x_i^2-(\sum x_i{)}^2}} \\ ) \end{gathered}$$
       • $$\begin{gathered} (a=\frac{\sum y}{n}-\frac{b\sum x}{n} \\ ) \end{gathered}$$

Assumptions about error term ϵ in linear regression model:

1. $$\begin{gathered} (E(ϵ)=0 \\ ) \end{gathered}$$. This implies ${\beta }_0$ and ${\beta }_1$​ are constants, and hence $$\begin{gathered} (E(y)=\beta 0+\beta 1x \\ ) \end{gathered}$$
2. The variance of ε, denoted by $$\begin{gathered} ( \\ {\sigma }^2 \\ ) \end{gathered}$$, is the same for all x
3. The values of ε are independent.
4. ε is a normally distributed random variable for all values of x

Testing for significance

• If $$\begin{gathered} ({\beta }_1=0 \\ ) \end{gathered}$$, then $$\begin{gathered} ( \\ E(y)={\beta }_0 \\ ) \end{gathered}$$. In this case, we would conclude that x and y are not linearly related

• If $$\begin{gathered} (\beta 1\ne 0 \\ ) \end{gathered}$$, we would conclude that the two variables are related.

• Thus, to test for a significant regression relationship, we must conduct a hypothesis test to determine whether $$\begin{gathered} ({\beta }_1=0 \\ ) \end{gathered}$$

• The t-test is commonly used.

  • 
  • 
  •