Nonlinear Systems

Topics in Digital Heritage:
Numerical Methods for Digital Reconstruction

Jin Woo Lee

KAIST

Fall 2026

Today’s ILO

• Formulate the one-dimensional root-finding problem
• Characterize the problem using continuity, Lipschitz continuity, and differentiability
• Solve the problem using iterative algorithms, such as bisection, fixed-point iteration, Newton’s method, and secant method
• Analyze the convergence of each method

Nonlinear Problems in Daily Life

Estimating Effective Interest Rate

• Jane took out a loan of \(10\) M KRW from the bank.
• She repays \(0.95\) M KRW each month for \(12\) months.

What is the effective monthly interest rate \(r\) of this loan?

• Let \(r\) be the monthly interest rate to be determined.
• The loan balance equation after 12 months is: \[B(r) = 10 \cdot (1 + r)^{12} - 0.95 \cdot \frac{(1 + r)^{12} - 1}{r}\]
• We want to find \(r\) such that \(B(r) = 0\).

Root-finding in a Single Variable

Root-finding

• Given a function \(f(x)\), we want to find \(x^*\) such that \(f(x^*) = 0\).
• We call \(x^*\) a root or a zero of \(f(x)\).

Example:

• Find the root of \(f(x) = x - 2\)
• Find the root of \(f(x) = x^2 - 2\)
• Find the root of \(f(x) = x^2 + e^{\cos x}\)

Characterizing Problems

Root-finding is guaranteed to fail on some problems. For example: \[ f(x) = \begin{cases} 1 & \text{if } x < 0 \\ -1 & \text{if } x \ge 0 \end{cases} \] We must add some assumptions on \(f(x)\) to guarantee the existence of a root.

Typical assumptions include:

• Continuity: \(f(x)\) is continuous if \(f(x_1) - f(x_2) \to 0\) as \(x_1 - x_2 \to 0\).
• Lipschitz continuity: \(f(x)\) is Lipschitz continuous if there exists a constant \(c\) such that \(|f(x_1) - f(x_2)| \le c |x_1 - x_2|\) for all \(x_1, x_2\).
• Differentiability: \(f(x)\) is differentiable if the derivative \(f'(x)\) exists for all \(x\).
• \(\boldsymbol{C^k}\)-continuity: \(f(x)\) is \(C^k\) continuous if the first \(k\) derivatives of \(f(x)\) exist and are continuous.

Continuity and Bisection

Intermediate Value Theorem

Suppose that \(f\) is continuous on the interval \([a, b]\) and that \(f(a) < \alpha < f(b)\) for some \(\alpha\). Then there exists \(x \in (a, b)\) such that \(f(x) = \alpha\).

def bisection(f, a, b, eps=1e-6, max_iter=100):
    for k in range(max_iter):
        c = (a + b) / 2
        if abs(f(c)) < eps or (b - a) / 2 < eps:
            return c
        elif f(c) * f(a) < 0:
            b = c
        else:
            a = c
    return c

Converges unconditionally for continuous \(f(x)\).

Continuity and Bisection

Bisection Convergence

• Error upper bound after \(k\) iterations: \[ E_k \equiv |x_k - x^*|\]
• Since we divide the interval in half each iteration, we can reduce our error bound by half in each iteration \(E_{k+1} = \frac{1}{2} E_k\)
• Since \(E_{k+1}\) is linear in \(E_k\), we say that bisection exhibits linear convergence with rate \(1/2\).

Lipschitz and Fixed-point Iteration

Suppose that we wish to find \(x^*\) such that \(g(x^*) = x^*\) and that \(g\) is Lipschitz continuous with constant \(0\leq c < 1\). Then we can use the fixed-point iteration:

1. Choose an initial guess \(x_0\).
2. Iterate \(x_{k+1} = g(x_k)\).

If this iteration converges, the resulting \(x_k\approx x^*\) is a fixed point of \(g\).

Lipschitz and Fixed-point Iteration

Fixed-point Iteration Convergence

When \(c<1\), the Lipschitz property ensures convergence to a root if one exists. \[\begin{aligned} E_k &\equiv |x_k - x^*| = |g(x_{k-1}) - g(x^*)| & \text{by design of the scheme} \\ &\leq c |x_{k-1} - x^*| & \text{since $g$ is Lipschitz} \\ &= c E_{k-1}. \\ \end{aligned}\] Applying this recursively, we have \(E_k \leq c^k E_0\). Since \(c < 1\), we have \(E_k \to 0\) as \(k \to \infty\). Thus, the fixed-point iteration converges linearly with rate \(c\).

Differentiability and Newton’s Method

Taylor expansion

Suppose \(f\) is \(C^k\) continuous at \(x_0\). Then we can write the Taylor expansion of \(f\) around \(x_0\) as \[ f(x) = f(x_0) + f'(x_0)(x - x_0) + \frac{f''(x_0)}{2!}(x - x_0)^2 + \cdots + \frac{f^{(k)}(x_0)}{k!}(x - x_0)^k + \mathcal{O}((x - x_0)^{k+1}) \] where \(\mathcal{O}((x - x_0)^{k+1})\) is the remainder term that goes to zero faster than \((x - x_0)^{k+1}\) as \(x \to x_0\).

Attempt to solve \(f(x) = 0\) by using the first-order Taylor expansion of \(f\) around \(x_k\): \[ f(x) \approx f(x_k) + f'(x_k)(x - x_k) \] Setting \(f(x) = 0\) gives the Newton’s update: \[ x_{k+1} = x_k - \frac{f(x_k)}{f'(x_k)} \]

Differentiability and Newton’s Method

Taylor expansion

Suppose \(f\) is \(C^k\) continuous at \(x_0\). Then we can write the Taylor expansion of \(f\) around \(x_0\) as \[ f(x) = f(x_0) + f'(x_0)(x - x_0) + \frac{f''(x_0)}{2!}(x - x_0)^2 + \cdots + \frac{f^{(k)}(x_0)}{k!}(x - x_0)^k + \mathcal{O}((x - x_0)^{k+1}) \] where \(\mathcal{O}((x - x_0)^{k+1})\) is the remainder term that goes to zero faster than \((x - x_0)^{k+1}\) as \(x \to x_0\).

Newton’s Method Convergence

Suppose \(f\) is differentiable and \(f'(x^*) \neq 0\) (i.e., \(x^*\) is simple root of \(f\)). Then the Newton’s method converges quadratically to \(x^*\) if the initial guess is sufficiently close to \(x^*\). \[\begin{aligned} E_k &\equiv |x_k - x^*| = \left|x_{k-1} - x^* - \frac{f(x_{k-1})}{f'(x_{k-1})}\right| & \text{by design of the scheme} \\ &= \left|\frac{f(x^*) - f(x_{k-1}) - f'(x_{k-1})(x^* - x_{k-1})}{f'(x_{k-1})}\right| & \text{since $f(x^*) = 0$} \\ &\leq \frac{1}{|f'(x_{k-1})|} \cdot \frac{1}{2} \max_{x \in [x_{k-1}, x^*]} |f''(x)| |x_{k-1} - x^*|^2 & \text{by Taylor's theorem} \\ &= C |x_{k-1} - x^*|^2 = C E_{k-1}^2. \end{aligned}\]

Remark. When \(x^∗\) is not simple, however, convergence of Newton’s method can be linear or even worse.

Secant Method

Evaluating \(f'(x_k)\) can be expensive or impossible. The secant method approximates the derivative using finite differences: \[ f'(x_k) \approx \frac{f(x_k) - f(x_{k-1})}{x_k - x_{k-1}} \] The secant method update is then given by \[ x_{k+1} = x_k - f(x_k) \frac{x_k - x_{k-1}}{f(x_k) - f(x_{k-1})} \]

• The convergence of the secant method is superlinear (between linear and quadratic) with order \((1+\sqrt{5})/2 \approx 1.618\) (the golden ratio).

Back to the Loan Example

Estimating Effective Interest Rate

• \(B(r) = 10 \cdot (1 + r)^{12} - 0.95 \cdot \frac{(1 + r)^{12} - 1}{r}\)
• We want to find \(r\) such that \(B(r) = 0\).

• Apply the secant method to find the root of \(B(r)\).
• Start with two initial guesses \(r_0 = 0.01\) and \(r_1 = 0.012\).
• Iterate using the secant method update until convergence.

Recap

Recap on Today’s ILO

• Formulate the one-dimensional root-finding problem
• Characterize the problem using continuity, Lipschitz continuity, and differentiability
  • We wish for the sequence \(x_k\) of root estimates to reach \(x^∗\) as quickly as possible. We can characterize the order of convergence:
    • Linear convergence: \(E_{k+1} \leq c E_k\) for some \(c<1\)
    • Superlinear convergence: \(E_{k+1} \leq c E_k^r\) with \(r>1\); no need for \(c<1\) when \(E_k\) is sufficiently small.
    • Quadratic convergence: \(E_{k+1} \leq c E_k^2\)
• Solve the problem using iterative algorithms, such as bisection, fixed-point iteration, Newton’s method, and secant method
• Analyze the convergence of each method
  • A method might converge quickly, needing fewer iterations to get sufficiently close to \(x^∗\), but each individual iteration may require additional computation time. In this case, it may be preferable to do more iterations of a simpler method than fewer iterations of a more complex one.