Data Preprocessing & Regression Analysis

Imagine a spreadsheet of patient records where some rows have no entry for "blood pressure." If you feed that gap straight into a model, it either crashes or silently learns something wrong. You have two main options: imputation (filling in a reasonable substitute) or deletion (removing the incomplete row).

For imputation, the most common approach is replacing the blank with the column's mean or median. Mean works well for normally distributed data, while median is safer when you have outliers — for instance, if most blood pressures are around 120 but one reading is 300, the mean gets pulled up, while the median stays stable. Deletion is simpler but risky: if 30% of your rows are missing that field, you're throwing away a lot of data.

Models do math, so a column like "Color" with values Red, Blue, Green is meaningless to them as text. Label encoding assigns each category an integer (Red=0, Blue=1, Green=2), which is quick but introduces a problem: the model may interpret Blue as "between" Red and Green, or assume Green > Red, when there's no such ordering.

One-hot encoding avoids this by creating a separate binary column for each category, so you'd get three columns (is_Red, is_Blue, is_Green), each containing 0 or 1. The trade-off is that one-hot encoding adds more columns, which can be expensive with high-cardinality features (imagine a "City" column with 500 unique values).

Say you have two features: age (ranging 18–65) and income (ranging 20,000–200,000). A distance-based algorithm like k-nearest neighbours calculates how "close" two data points are. Without scaling, income dominates that distance calculation simply because its numbers are bigger — age differences become basically invisible. Scaling fixes this by putting both features on a comparable range.

Min-Max scaling squeezes everything into 0–1 using the formula . So an age of 18 becomes 0, an age of 65 becomes 1, and everything else falls between. Standardisation (z-score) instead centres the data around 0 with a standard deviation of 1, using . Min-Max is great when you know your data has fixed bounds and no extreme outliers. Standardisation is more robust to outliers and is generally preferred for algorithms like logistic regression or SVMs.

Not all algorithms need feature scaling. Tree-based models (like decision trees and random forests) split on thresholds and don't care about magnitude, so scaling doesn't affect them. It matters most for distance-based and gradient-based algorithms.

You can learn more about cost function here. The core idea remains pretty much the same, only difference is that this formula writes the cost as the sum of all errors, which corresponds to batch gradient descent where you accumulate the total error across every example before making one update. Batch is more stable per step, but each step is expensive because you have to loop through the entire dataset.

means "assign." It computes the expression on the right using the current values, then updates the left side with the result.

This is gradient descent in its simplest form, adjust the weight by stepping in the opposite direction of the slope, using a small learning rate so we don't overshoot. If you want to learn more, check out this post, but just note that it's the same update rule, just applied in a deeper context. Linear regression uses the simplest form of this update, back-propagation extends it to networks with multiple layers.

The goal is to find the partial derivative for our update rule:

We start with our Least Squares Cost Function (summing over all examples):

And our linear hypothesis:

Set up the Explicit Chain Rule

Instead of differentiating everything at once, we split the derivative into two clear parts using Leibniz notation:

We put the derivatives inside the sum because the total gradient is just the sum of the gradients of each individual training example.

Calculate the Two Parts

Part 1: How does the cost change with respect to the prediction? Let . Using the power rule ( where is just with respect to ):

Part 2: How does the prediction change with respect to the weight ? Looking at the hypothesis , if we take the derivative with respect to , all other terms become , leaving only the feature :

Combine the Chain Rule

Now, we multiply Part 1 and Part 2 together and place them back inside our summation:

The Final Update Rules

Substitute the combined derivative back into the original gradient descent update rule:

To get the most elegant form, distribute the negative sign to flip the inner terms to .

For Batch Gradient Descent (All Samples):

For Stochastic Gradient Descent (1 Sample at a time): We simply drop the summation, giving us the exact LMS update rule:

In standard linear regression, when you fit a line, every single training example has an equal vote. A house that is 5,000 sq ft has the exact same influence on the line as a house that is 1,000 sq ft.

LWLR changes this. If you want to predict the price of a house that is 1,500 sq ft (your query point, ), LWLR says: "I only really care about training examples that are similar in size to 1,500 sq ft. The 5,000 sq ft houses shouldn't influence my prediction right now."

To do this, we multiply the error of each training example by a weight, .

Deconstructing the Equation

The weight formula is a Gaussian (bell-shaped) curve:

Here is what each piece is doing:

1. The Distance:
   This is the squared distance between a training example and the specific point you are trying to predict .
   • If the training point is very close to your query point, this value is almost 0.
   • If the training point is very far away, this value is large.
2. The Exponential & Negative Sign:
   Because of the negative sign, we are doing to the power of a negative number.
   • If the distance is roughly (the points are close), we get . The weight is . The model gives this point maximum importance.
   • If the distance is large (the points are far), we get , which approaches 0. The weight becomes . The model completely ignores this point.
3. The Bandwidth Parameter: (Tau)
   dictates how "fat" or "thin" our bell curve is. It controls how quickly the weights fall off to zero as you move away from the query point .
   • Large : The weights fall off very slowly. The model looks at a very wide "neighborhood" of points. If is infinitely large, LWLR just becomes standard linear regression.
   • Small : The weights fall off rapidly. The model only looks at a very tiny, strict neighborhood of points immediately next to .

Why is it called "Non-parametric"?

Standard linear regression is parametric. You look at the data once, calculate your parameters, and then you can throw the training data away forever. Your formula does all the work.

LWLR is non-parametric. Because the weights depend on (the specific thing you are trying to predict right now, you must recalculate from scratch every single time you want to make a new prediction.), you cannot pre-calculate your values. You have to keep the entire training dataset in memory and calculate a brand new set of values for every single prediction you make.

Imagine you are a doctor trying to predict if a tumor is Malignant () or Benign () based on its size ().

If you use standard Linear Regression, you fit a straight line through the data. You might say: "If the line outputs a value greater than 0.5, I will predict 1 (Malignant). If it's less than 0.5, I predict 0 (Benign)."

This works okay until a patient comes in with a massive tumor. Because linear regression is a straight line, it will try to accommodate this massive outlier. The line will tilt upwards heavily. Suddenly, your threshold shifts, and tumors that you used to correctly predict as Malignant are now being predicted as Benign.

Even worse, the straight line might output a value like or . What does a "3.5" or "-1.2" mean when the answer can only be 0 or 1? It's nonsense.

The Solution: The Sigmoid Function

Instead of a straight line, we want a curve that flatlines at and flatlines at . We want the "S-shape" curve.

This is what the Sigmoid function does:

Think of as a "raw confidence score" (which can be any number from to ).

• If the raw score is , the sigmoid turns it into exactly (50% probability).
• If the raw score is a huge positive number (like ), the sigmoid squashes it to (99.99% probability it is Malignant).
• If the raw score is a huge negative number (like ), the sigmoid squashes it to (0.01% probability it is Malignant).

Logistic Regression is just Linear Regression wrapped in a translator that turns "raw scores" into "probabilities."

In Linear Regression, we used the Least-Squares cost function (minimising the squared error).

If we try to use Least-Squares with our new curvy Sigmoid function, the math breaks. It creates a "wavy" 3D landscape with lots of fake valleys (local minima). Gradient descent gets stuck and can't find the bottom.

So, statisticians use a different way to measure error called Maximum Likelihood.

This equation looks terrifying, but it's actually an elegant logical "IF" statement:

• If the actual answer is : The second half of the equation becomes and disappears. We are just left with . We want our prediction to be as close to as possible.
• If the actual answer is : The first half of the equation becomes and disappears. We are just left with . We want our prediction to be as close to as possible.

It essentially heavily punishes the model if it is very confident but wrong. (e.g., If the model predicts a 99% probability of a tumor being Malignant, but it's actually Benign, the cost explodes to infinity).

The "Beautiful Symmetry"

Here is the craziest part of all of this.

If you take that terrifying Log-Likelihood equation, and you do the calculus chain rule to find the Gradient Descent update rule (just like we did for linear regression)... a bunch of math cancels out perfectly.

You are left with this:

It is the EXACT same update rule as standard Linear Regression.

The computer code you write to update the weights for Logistic Regression is identical to the code for Linear Regression. The only difference is that in Logistic Regression, calculates the Sigmoid formula instead of just .