Estimate the expected (average) outcome given predictors.
The expected value of a random variable with a finite number of outcomes is a weighted average of all possible outcomes.
Linear Regression
A constant change in x leads to a constant change in the expected outcome.
Normal linear regression: lm(y~x)
For a fixed \(x_i\), the model describes the distribution of possible outcomes around the expected value.
Probability Distributions
Why probability distributions?
Our models do not fit our data exactly, we predict the average value of the outcome given the predictors.
Probability distributions help us characterize the variation that remains after predicting the average.
Quantify uncertainty in the estimated parameters of the model!
Probability Distributions
A probability distribution describes all possible values a random variable \(Y\) can take and their associated probabilities, given certain parameters.
\[f(y \mid \boldsymbol{\theta})\]
Where:
\(y\) is a specific value
\(\boldsymbol{\theta}\) is a vector of parameters (e.g., \(\mu\), \(\sigma^2\))
\(f(\cdot)\) is the probability function
Discrete vs. Continuous Variables
Continuous random variables (time, weight, temperature, etc.)
Probability Density Function (PDF)
\(f(y \mid \boldsymbol{\theta})\)
The probability density at \(y\) (not a probability!)
\(P(Y = y) = 0\) for any specific value
Discrete vs. Continuous Variables
Discrete random variables (counts, binary outcomes, etc.)
Probability Mass Function (PMF)
\(P(Y = y \mid \boldsymbol{\theta})\)
The probability of observing exactly \(y\)
\(P(Y = 3) = 0.25\) means 25% chance of getting 3
Distributions and linear regression
Expected value (mean): \(E[Y]=\mu\).
The long-run average of \(Y\) under its distribution.
Conditional expected value: \(E[Y\mid X]=\mu(X)\).
The long-run average outcome for a fixed predictor value\(X=x\).
Variance: \(\mathrm{Var}(Y)\) (or, in regression, \(\mathrm{Var}(Y\mid X)\)).
How much outcomes vary around the mean (overall or at a given \(x\)).
In normal linear and generalized linear models we generally include predictors on the mean of the distribution.
Why is it important?
The definitions of expected value and variance are universal, but how we compute them (and what they equal) depends on the distribution’s parameters.
For a Normal distribution \(Y \sim \mathcal{N}(\mu,\sigma^2)\), the parameters are exactly the mean and variance.
Expected value (mean):\(E[Y]=\mu\).
Variance:\(\mathrm{Var}(Y)=\sigma^2\).
Distribution
Support
mean: \(E[Y]\)
variance: \(\mathrm{Var}(Y)\)
Normal: \(f(y \text{ | } \mu,\sigma^2)\)
\(y \in \mathbb{R}\)
\(\mu\)
\(\sigma^2\)
Gamma: \(f(y \text{ | } \alpha,\lambda)\)
\(y \in (0,\infty)\)
\(\alpha/\lambda\)
\(\alpha/\lambda^2\)
Binomial: \(f(y \text{ | } n,p)\)
\(y \in \{0,1,\dots,n\}\)
\(np\)
\(np(1-p)\)
Poisson: \(f(y \text{ | } \lambda)\)
\(y \in \{0,1,2,\dots\}\)
\(\lambda\)
\(\lambda\)
The Normal distribution
Normal distribution
A Normal distribution has parameters \(\mu\) (mean) and \(\sigma^2\) (variance):
# number of students that have passed the examsum(dat_exam$passed) #> [1] 39# proportion of students that have passed the exammean(dat_exam$passed) #> [1] 0.39# study hours and passing the examtapply(dat_exam$studyh, dat_exam$passed, mean)#> 0 1 #> 37.52459 69.05128 #> table(dat_exam$passed, cut(dat_exam$studyh, breaks =4))#> (1.9,26.2] (26.2,50.5] (50.5,74.8] (74.8,99.1]#> 0 23 22 10 6#> 1 2 6 10 21#> tapply(dat_exam$passed, cut(dat_exam$studyh, breaks =4), mean)#> (0,24.8] (24.8,49.5] (49.5,74.2] (74.2,99.1] #> 0.0800000 0.2142857 0.5000000 0.7777778
Visualize data
Fitting a Normal Linear Model
How does this look to you?
Fitting a Generalized Linear Model
The model should consider both the support of the \(y\) variable and the non-linear pattern!
Example: Reaction times
Reaction times on two-choice questions as a function of hours studied.
range(dat_rt$rt)#> 1.341846 119.428719# mean RT as a function of studyhtapply(dat_rt$rt, cut(dat_rt$studyh, breaks =4), mean)#> (0,24.8] (24.8,49.5] (49.5,74.2] (74.2,99.1] #> 56.10084 26.73179 11.39825 6.46455
Visualize data
Fitting a Normal Linear Model
How does this look to you?
Fitting a Generalized Linear Model
The model should consider both the support of the \(y\) variable and the non-linear pattern!
Generalized Linear Models (GLM)
General ideas
using distributions beyond the Normal
modeling non linear functions on the response scale
taking into account mean-variance relationships
The three ingredients of a GLM
Random Component: Choose a distribution
Systematic Component: Linear predictor \(\eta_i = \beta_0 + \beta_1 x_i\)
Link Function: Transform the mean \(g(\mu_i) = \eta_i\)
Random component
The random component specifies a probability model for \(y_i\):
The Bernoulli(p) or the Binomial(n,p) distributions can be used as random component when we have a binary dependent variable or the number of successes over the total number of trials.
Binary (pass/fail), one trial per person; or
Counts of successes out of \(n\) trials (e.g., items correct out of \(n\)).
Bernoulli Distribution
Suppose you collected data on whether each student passed the exam. Let \(K \in \{0, 1\}\) denote the outcome, where 0 = fail and 1 = pass.
\[
P(K = k) =
\begin{cases}
p & \text{if } k = 1 \text{ (pass)}, \\
1-p & \text{if } k = 0 \text{ (fail)}.
\end{cases}
\]
