When real world data refuses to fit a clean line, least squares approximation becomes your best ally in linear algebra. You will learn how to handle inconsistent systems, why orthogonal projections matter, and how to build a predictive line from noisy points, useful for anyone working with data, statistics, or machine learning foundations.

Why does an inconsistent system need least squares?

A system Ax = b has no solution when the vector b lives outside the column space of A. No linear combination of the columns of A can build b exactly, so you need a workaround instead of giving up.

The trick is elegant: if you cannot find an x that satisfies Ax = b, you look for an x̂ (x hat) that gets you as close as possible. That closest reachable point is the orthogonal projection of b onto the column space of A, which we call p.

What is an inconsistent system? It is a system of equations with no exact solution because the result vector lies outside the reachable space of the coefficient matrix.

How does the error vector lead to the normal equation?

The error vector e measures the gap between where you want to go (b) and where you can actually go (p), so e = b minus p. Geometrically, it is the shortest line between point b and the column space plane.

The shortest distance happens when e is orthogonal to the column space of A. Since the dot product of orthogonal vectors is zero, you can write A transposed times e equals zero.

From there, replace e with b minus Ax̂, distribute A transposed, and rearrange. You arrive at the normal equation:

• A transposed times A times x̂ equals A transposed times b.

This equation gives you the best possible solution when the original system has none.

How do you apply least squares to a real example?

Imagine three students and you want to model their grade based on study hours using grade = c1 + c2 times hours. The data points are:

• Student 1: 2 hours, grade 70.
• Student 2: 3 hours, grade 90.
• Student 3: 4 hours, grade 80.

This gives three equations with only two unknowns, c1 and c2. That makes it a sobredetermined system, and in this case also inconsistent. A quick note: not every overdetermined system is inconsistent, some have redundant equations and still solve cleanly.

Setting up matrix A and vector b

The coefficient matrix A has a column of ones and a column with 2, 3, 4. Vector b is 70, 90, 80. Vector x holds the unknowns c1 and c2.

What does A transposed times A do? It transforms an unsolvable system into a square, solvable one by projecting the problem into the column space.

Computing A transposed times A gives the matrix [[3, 9], [9, 29]]. Computing A transposed times b gives the vector [240, 730]. Now you have a clean 2x2 system to solve.

Solving with Gaussian elimination

Building the augmented matrix and subtracting three times row one from row two, you get an upper triangular form. Back substitution gives:

• 2c2 = 10, so c2 = 5.
• 3c1 + 45 = 240, so c1 = 65.

Your best fit line becomes grade = 65 + 5 times hours. Each extra hour of study adds five points to the predicted grade, starting from a baseline of 65.

What does the best fit line tell you about the data?

When you plot the three original points and overlay the line, the dotted segments connecting each point to the line represent the projections, the closest reachable values inside the column space.

No line passes through all three points exactly, and that is fine. Least squares does not promise perfection, it promises the smallest possible total error. That is why this method powers regression analysis, trend lines, and predictive modeling across countless real world applications.

Why use orthogonal projection instead of any other point? Because orthogonality guarantees the minimum distance, which means the smallest error between your data and your model.

Now a small challenge for you. If the data points were [1,1], [2,2] and [3,3], would the system Ax = b be inconsistent or not? Drop your answer in the comments and explain your reasoning.