Ctrl+K

Dimension Reduction

6. Dimension Reduction#

6.1. Dimension Reduction Techniques#

Dimension reduction techniques aim to reduce the number of predictors by transforming them into a lower-dimensional space while preserving most of the variability in the data.

  • Dimension reduction techniques are effective for dealing with multicollinearity and high-dimensional data

  • The goal is constructing M new predictors, which are linear combination of p original predictors, and use the new predictors to make predictions.

    • Let Z1,Z2,...,ZM represent the new predictors with M<P

    • ϕm1,...,ϕmj are some constants $Zm=∑j=1pϕmjXj$

    • Then we fit * i=1,...,n $yi=θ0+∑m=1MθmZim+ei$ image-2.png

  • But may result in less interpretable models compared to subset selection and shrinkage methods.

6.1.1. Principal Component Analysis#

  • PCA transforms the original predictors into a new set of uncorrelated variables called principal components.

  • These components are ordered by the amount of variance they explain, allowing for dimensionality reduction by retaining only the most important components.

image-2.png

  • The first component summarize the most information from these two features

image-2.png

  • Not many information left on the 2nd component

image-2.png

6.1.2. Partial Least Squares (PLS)#

  • PCR assumes the principle components from predictors might have the best performance on dependent variable prediction (unsupervised way)

    • This assumption might wrong

    • We do not use the information from dependent variable to supervise the identification of the principal components.

  • PLS is also a dimension reduction technique

    • But unlike PCR, PLS identify new features that not only approximate the old features well, but also that are related to the target variable

    • Steps:

      1. Standardize p predictors

      2. Construct simple Bivariate linear regression to get θ1,...,θp

      3. Compute the first direction Z1=∑j=1pθ1jXj

        1. PLS places the highest weight on the variables that most strongly related to the response.

Contents