Linear Systems 1

Topics in Digital Heritage:
Numerical Methods for Digital Reconstruction

Jin Woo Lee

KAIST

Fall 2026

Announcements

• Quick Intro
• Course Update
  • Logistics
  • Plan for the semester
• Seminar Schedule
  • Sep. 15 (Tue), 16:00 – 17:30; N25 #3229 백남준홀
    • GSCT Colloquium (Prof. Junghoon Pi)
  • Sep. 17 (Thu), 16:00 – 17:30; N25 #3229 백남준홀
    • GSMV Colloquium (Prof. Jin Woo Lee)

Recap

Linearity

Definition

A function \(f: \mathbb{R}^n \to \mathbb{R}^m\) is linear if it satisfies the following two properties for all \(\mathbf{u}, \mathbf{v} \in \mathbb{R}^n\) and all scalars \(c \in \mathbb{R}\):

1. Additivity: \(f(\mathbf{u} + \mathbf{v}) = f(\mathbf{u}) + f(\mathbf{v})\)
2. Homogeneity: \(f(c\mathbf{u}) = c f(\mathbf{u})\)

Linear System

\[\begin{split} 3x + 2y + 5z &= 0 \\ -4x + 9y - 3z &= -7 \\ 2x -3y -3z &= 1 \end{split}\]

\[ \begin{bmatrix} 3 & 2 & 5 \\ -4 & 9 & -3 \\ 2 & -3 & -3 \end{bmatrix} \begin{bmatrix} x \\ y \\ z \end{bmatrix} = \begin{bmatrix} 0 \\ -7 \\ 1 \end{bmatrix} \]

\[ \mathbf{A}\mathbf{x} = \mathbf{b} \]

If \(\exists \mathbf{A}^{-1}\), s.t. \(\mathbf{A}^{-1}\mathbf{A} = \mathbf{I}\), then \[\mathbf{x} = \mathbf{A}^{-1}\mathbf{b}.\]

Today’s ILO

• Explain the solvability of linear systems and the conditions under which a unique solution exists.
• Explain the concept of Tikhonov regularization and how it can be used to stabilize ill-conditioned systems.
• Apply linear algebra techniques to solve problems in image alignment and deconvolution.

Solvability

\[\begin{split} 3x + 2y &= 6 \\ -4x + y &= 7 \end{split}\]

\[ \begin{bmatrix} 3 & 2 \\ -4 & 1\end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} 6 \\ 7 \end{bmatrix} \]

\[ \mathbf{A}\mathbf{x} = \mathbf{b} \]

Solvability

The solvability of \(\mathbf{A}\mathbf{x} = \mathbf{b}\) must fall into one of three cases:

1. The system may not admit any solutions, as in: \[ \begin{bmatrix} 1 & 0 \\ 1 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} -1 \\ 1 \end{bmatrix} \]

2. The system may admit a single solution.

3. The system may admit infinitely many solutions, e.g., \(0\mathbf{x} = \mathbf{0}\).

Solvability

\[ \mathbf{A}\mathbf{x} = \mathbf{b} \]

1. No wide matrix system admits a unique solution.
2. For every tall matrix \(\mathbf{A}\), there exists a \(\mathbf{b}_0\) such that \(\mathbf{A}\mathbf{x} = \mathbf{b}_0\) is not solvable.

Parametric Regression

Assume data points \((\mathbf{x}^{(k)}, y^{(k)})\) for \(k=1, ..., n\), where \(\mathbf{x}^{(k)} \in \mathbb{R}^n\) and \(y^{(k)} \in \mathbb{R}\), carried out by \(n\) experiments that reveal \(y^{(k)}\equiv f(\mathbf{x}^{(k)})\) for samples \(\mathbf{x}^{(k)}\).

Assume that \(f\) is linear: \[ f(\mathbf{x}) = a_1x_1 + a_2x_2 + \cdots + a_nx_n \] We want to determine the coefficients \(a_1, ..., a_n\).

\[ \begin{split} y^{(1)} &= f(\mathbf{x}^{(1)}) = a_1x_1^{(1)} + a_2x_2^{(1)} + \cdots + a_nx_n^{(1)} \\ y^{(2)} &= f(\mathbf{x}^{(2)}) = a_1x_1^{(2)} + a_2x_2^{(2)} + \cdots + a_nx_n^{(2)} \\ & \qquad\qquad \vdots \\ y^{(n)} &= f(\mathbf{x}^{(n)}) = a_1x_1^{(n)} + a_2x_2^{(n)} + \cdots + a_nx_n^{(n)} \end{split}\]

\[ \begin{bmatrix} - & \mathbf{x}^{(1)\top} & - \\ - & \mathbf{x}^{(2)\top} & - \\ & \vdots & \\ - & \mathbf{x}^{(n)\top} & - \end{bmatrix} \begin{bmatrix} a_1 \\ a_2 \\ \vdots \\ a_n \end{bmatrix} = \begin{bmatrix} y^{(1)} \\ y^{(2)} \\ \vdots \\ y^{(n)} \end{bmatrix} \]

The coefficients \(a_i\) can be determined by solving the linear system \(\mathbf{X}^\top\mathbf{a} = \mathbf{y}\), where \(\mathbf{X}\) is the columns of the data points and \(\mathbf{y}\) is the vector of observed outputs.

Generalization to Nonlinear Functions

Suppose that nonlinear \(f\) is written in a linear combination of basis functions \(f_i(\mathbf{x})\): \[ f(\mathbf{x}) = a_1f_1(\mathbf{x}) + a_2f_2(\mathbf{x}) + \cdots + a_nf_n(\mathbf{x}) \] We want to determine the coefficients \(a_1, ..., a_n\).

\[ \begin{split} y^{(1)} &= f(\mathbf{x}^{(1)}) = a_1f_1(\mathbf{x}^{(1)}) + a_2f_2(\mathbf{x}^{(1)}) + \cdots + a_nf_n(\mathbf{x}^{(1)}) \\ y^{(2)} &= f(\mathbf{x}^{(2)}) = a_1f_1(\mathbf{x}^{(2)}) + a_2f_2(\mathbf{x}^{(2)}) + \cdots + a_nf_n(\mathbf{x}^{(2)}) \\ & \qquad\qquad \vdots \\ y^{(n)} &= f(\mathbf{x}^{(n)}) = a_1f_1(\mathbf{x}^{(n)}) + a_2f_2(\mathbf{x}^{(n)}) + \cdots + a_nf_n(\mathbf{x}^{(n)}) \end{split}\]

\[ \begin{bmatrix} f_1(\mathbf{x}^{(1)}) & f_2(\mathbf{x}^{(1)}) & \cdots & f_n(\mathbf{x}^{(1)}) \\ f_1(\mathbf{x}^{(2)}) & f_2(\mathbf{x}^{(2)}) & \cdots & f_n(\mathbf{x}^{(2)}) \\ \vdots & \vdots & \ddots & \vdots \\ f_1(\mathbf{x}^{(n)}) & f_2(\mathbf{x}^{(n)}) & \cdots & f_n(\mathbf{x}^{(n)}) \end{bmatrix} \begin{bmatrix} a_1 \\ a_2 \\ \vdots \\ a_n \end{bmatrix} = \begin{bmatrix} y^{(1)} \\ y^{(2)} \\ \vdots \\ y^{(n)} \end{bmatrix} \]

Even if \(f\) is nonlinear, we can still solve for the coefficients \(a_i\) using linear algebra techniques, as long as the basis functions \(f_i(\mathbf{x})\) are known and linearly independent.

Caveats when doing regressions on:

Linear function with random noise

x <- seq(-2, 2, length.out = 11)
y <- 2 * x + 1 + rnorm(length(x), sd = 0.8)

Nonlinear function

x <- seq(-5, 5, length.out = 9)
y <- abs(x)

Least Squares (LS) Regression

We wish to solve the linear system \(\mathbf{A}\mathbf{x} = \mathbf{b}\), where \(\mathbf{A} \in \mathbb{R}^{m \times n}\) and \(\mathbf{b} \in \mathbb{R}^m\).

• A tall system where \(m > n\) (i.e., having more than \(n\) observations) may yield to incompatible solutions.

• A wide system where \(m < n\) (i.e., having fewer than \(n\) observations) may yield to infinitely many solutions.

• When we cannot solve \(\mathbf{A}\mathbf{x} = \mathbf{b}\) exactly, we can instead try to find an approximate solution \(\mathbf{x}\) satisfying \(\mathbf{A}\mathbf{x} \approx \mathbf{b}\).

This can be done by solving the least squares problem: \[\min_{\mathbf{x}} \|\mathbf{A}\mathbf{x} - \mathbf{b}\|^2_2\]

Tikhonov Reguliarization

When the system is ill-conditioned, we can add a regularization term to the least squares problem: \[\min_{\mathbf{x}} \|\mathbf{A}\mathbf{x} - \mathbf{b}\|^2_2 + \lambda \|\mathbf{x}\|^2_2\] where \(0 < \lambda \ll 1\). This second term is called the Tikhonov regularizer. This asks that among the minimizers of \(\|\mathbf{A}\mathbf{x} - \mathbf{b}\|_2^2\), we would prefer those with small norm \(\|\mathbf{x}\|_2\); as \(\lambda\) increases, we prioritize the norm of \(\mathbf{x}\) more.

Tikhonov Reguliarization

\[\min_{\mathbf{x}} \|\mathbf{A}\mathbf{x} - \mathbf{b}\|^2_2 + \lambda \|\mathbf{x}\|^2_2\] To minimize this objective, we take derivative with respect to \(\mathbf{x}\) and set it to zero: \[\begin{split} \frac{\partial}{\partial \mathbf{x}} \left( \|\mathbf{A\mathbf{x} - \mathbf{b}\|^2_2 + \lambda \|\mathbf{x}\|^2_2} \right) &= 0 \\ \frac{\partial}{\partial \mathbf{x}} \left( \mathbf{x}^\top\mathbf{A}^\top \mathbf{A\mathbf{x} - 2\mathbf{b}^\top\mathbf{A}\mathbf{x} + \|\mathbf{b}\|^2_2 + \lambda \mathbf{x}^\top\mathbf{x}} \right) &= 0 \\ \quad 2\mathbf{A}^\top(\mathbf{A}\mathbf{x} - \mathbf{b}) + 2\lambda \mathbf{x} &= 0 \\ \quad (\mathbf{A}^\top\mathbf{A} + \lambda \mathbf{I})\mathbf{x} &= \mathbf{A}^\top\mathbf{b} \end{split}\]

Caveats

• The solution of the Tikhonov regularized problem is no longer exact.
• When \(\lambda\) is small, the matrix \(\mathbf{A}^\top\mathbf{A} + \lambda \mathbf{I}\) is invertible but may be poorly conditioned. Increasing \(\lambda\) solves this at the cost of less accurate solutions.

Image Alignment

Given two images, we marked \(p\) pairs of points \(\mathbf{x}^{(k)},\mathbf{y}^{(k)}\in\mathbb{R}^2\) in each image. We want to find the best rotation and translation that aligns the two images.

Suppose that there exists some transformation matrix \(\mathbf{A}\in\mathbb{R}^{2\times 2}\) and translation vector \(\mathbf{b}\in\mathbb{R}^2\) such that: \[\mathbf{y}^{(k)} = \mathbf{A}\mathbf{x}^{(k)} + \mathbf{b}, \quad k=1, ..., p\]

Given a set of corresponding points \(\{\mathbf{x}^{(k)}, \mathbf{y}^{(k)}\}_{k=1}^p\), we can set up a linear system to solve for the unknowns in \(\mathbf{A}\) and \(\mathbf{b}\).

To do so, we solve the following minimization problem: \[\min_{\mathbf{A}, \mathbf{b}} \sum_{k=1}^p \|\mathbf{y}^{(k)} - (\mathbf{A}\mathbf{x}^{(k)} + \mathbf{b})\|^2_2\]

Deconvolution

  Blurry \(\mathbf{x}_0\quad\) Sharp \(\mathbf{x}\)

Given a blurry image \(\mathbf{x}_0\) and a known blur kernel \(\mathbf{h}\), we want to recover the original sharp image \(\mathbf{x}\). One strategy is deconvolution, which can be formulated as a linear system: \[\min_{\mathbf{x}} \|\mathbf{h} * \mathbf{x} - \mathbf{x}_0\|^2_2\] where \(*\) denotes convolution.

In this case, the unknown is \(\mathbf{x}\), where in many cases, multiple solutions may exist due to the ill-posed nature of the problem. This can be solved possibly with Tikhonov regularization.

Recap on Today’s ILO

Today’s ILO

• Explain the solvability of linear systems and the conditions under which a unique solution exists.
• Explain the concept of Tikhonov regularization and how it can be used to stabilize ill-conditioned systems.
• Apply linear algebra techniques to solve problems in image alignment and deconvolution.