{ "cells": [ { "cell_type": "markdown", "id": "d0db1426-71a9-41f5-80f8-b27e6f8f16ae", "metadata": {}, "source": [ "# Classification\n", "\n", "- Example: Credit Card Default\n", "\n", "\n", "\n", "## Why not Linear regression\n", "\n", "\n", "\n", "- In this case of a **binary outcome**, linear regression does a good job as a classifier, and is equivalent to `linear discriminant analysis`.\n", " - For more than 2 outcome classes, regression will suggests an ordering ($3>2>1$), therefore can not use regression for multi-types classification.\n", "- However, `linear regression` might produce probabilities less than zero or bigger than one. `Logistic regression` is more appropriate.\n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "97c5362c", "metadata": {}, "source": [ "## Logistic Regression\n", "\n", "- **Definition**: `Logistic Regression` is a statistical method used for `binary classification`. It predicts the probability that a given instance belongs to a particular class. The core idea is to model the probability that a target variable `Y` is a particular value ($1$ or $0$) given a set of independent variables `X`.\n", "\n", " $$\n", " p(X)=\\frac{e^{\\beta_0+\\beta_1 x}}{1+e^{\\beta_0+\\beta_1 x}}\n", " $$\n", "\n", "- A bit of rearrangement gives \\* This monotone transformation is called the `log odds` or `logit` transformation of $p(X)$\n", " $$\n", " log(\\frac{p(X)}{1-p(X)}=\\beta_0+\\beta_1 X)\n", " $$\n", "\n", "### The logistic Function\n", "\n", "- The `logistic function`, also known as the `sigmoid` function, is an S-shaped curve that can take any real-valued number and map it into a value between $0$ and $1$, but **never exactly at those limits**. Mathematically, it is represented as:\n", " $$\n", " \\sigma(Z)=\\frac{1}{1+e^{-z}}\n", " $$\n", "- where $z$ is a linear combination of the input features $X$, i.e., $z = β₀ + β₁X₁ + β₂X₂ + ... + βₙXₙ$, where $β₀, β₁, ..., βₙ$ are the parameters of the model.\n" ] }, { "cell_type": "markdown", "id": "1d1d39e8", "metadata": {}, "source": [ "### Estimate Parameters (Maximum Likelihood)\n", "\n", "$$\n", "l(\\beta_0,\\beta)=\\prod_{i:y_i=1}p(x_i)\\prod_{i:y_i=0}(1-p(x_i))\n", "$$\n", "\n", "- The parameters $β₀, β₁, ..., βₙ$ are estimated using the `Maximum Likelihood Estimation` (MLE).\n", "- The idea is to find the set of parameters that maximize the likelihood of the observed data.\n", "- We pick $\\beta_0$ and $\\beta$ to maximize the likelihood of observed data.\n", "\n", "\n", "\n", "### Making Predictions\n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "2470685a", "metadata": {}, "source": [ "### Script Example\n" ] }, { "cell_type": "code", "execution_count": 1, "id": "de6ee197", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.metrics import accuracy_score\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns" ] }, { "cell_type": "code", "execution_count": 2, "id": "7364b8bb", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", " | Age | \n", "Sex | \n", "ChestPain | \n", "RestBP | \n", "Chol | \n", "Fbs | \n", "RestECG | \n", "MaxHR | \n", "ExAng | \n", "Oldpeak | \n", "Slope | \n", "Ca | \n", "Thal | \n", "AHD | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | \n", "63 | \n", "1 | \n", "typical | \n", "145 | \n", "233 | \n", "1 | \n", "2 | \n", "150 | \n", "0 | \n", "2.3 | \n", "3 | \n", "0.0 | \n", "fixed | \n", "No | \n", "
2 | \n", "67 | \n", "1 | \n", "asymptomatic | \n", "160 | \n", "286 | \n", "0 | \n", "2 | \n", "108 | \n", "1 | \n", "1.5 | \n", "2 | \n", "3.0 | \n", "normal | \n", "Yes | \n", "
3 | \n", "67 | \n", "1 | \n", "asymptomatic | \n", "120 | \n", "229 | \n", "0 | \n", "2 | \n", "129 | \n", "1 | \n", "2.6 | \n", "2 | \n", "2.0 | \n", "reversable | \n", "Yes | \n", "
4 | \n", "37 | \n", "1 | \n", "nonanginal | \n", "130 | \n", "250 | \n", "0 | \n", "0 | \n", "187 | \n", "0 | \n", "3.5 | \n", "3 | \n", "0.0 | \n", "normal | \n", "No | \n", "
5 | \n", "41 | \n", "0 | \n", "nontypical | \n", "130 | \n", "204 | \n", "0 | \n", "2 | \n", "172 | \n", "0 | \n", "1.4 | \n", "1 | \n", "0.0 | \n", "normal | \n", "No | \n", "