Multivariable Taylor Expansion & Quadratic Approximations

Multivariable Taylor expansions generalize single-variable series to functions of multiple variables, using partial derivatives to approximate behavior around a point. Quadratic approximations, the second-order truncation, capture curvature via the Hessian matrix, essential for understanding local geometry. In AI and machine learning, these tools analyze loss landscapes, enable second-order optimization like Newton's method, approximate complex models for efficiency, and facilitate sensitivity analysis in high-dimensional parameter spaces.

This lecture builds on single-variable Taylor series, exploring multivariable formulations, matrix notations, remainder terms, convergence, and practical applications in ML. We'll delve into derivations, properties, and implementations, with extensive examples and code in Python and Rust to illustrate computations and visualizations, empowering you to leverage these approximations in advanced AI tasks.

1. Intuition for Multivariable Taylor Expansions

In one variable, Taylor approximates with value, slope, curvature. In multiple, it uses function value, gradient (all first partials), Hessian (second partials), and higher tensors.

Imagine a surface z=f(x,y): Near (a,b), it's like a plane (first-order), then paraboloid (quadratic), refining with higher terms.

Vector form: For f: R^n → R, around a, f(x) ≈ f(a) + ∇f(a)^T (x-a) + (1/2)(x-a)^T H(a) (x-a) + ...

ML Connection

Loss functions in high-dim: Quadratic approx for trust regions in optimizers.
Neural net linearization: For interpretability or attacks.

INFO

Multivariable Taylor unfolds multidimensional wiggliness into polynomial layers, like topographic maps for functions.

Example

f(x,y)=x^2 + y^2 at (0,0): Exact quadratic. At (1,1): f(1+Δx,1+Δy) ≈ 2 + 2(Δx + Δy) + (Δx^2 + Δy^2).

2. Formal Definition and Notation

For f: R^n → R, k-times differentiable, Taylor polynomial of degree m at a:

P_m(x) = sum_{|α|≤m} \frac{D^α f(a)}{α!} (x-a)^α,

where α multi-index (α1,...,αn), |α|=sum αi, D^α = ∂^{|α|} / ∂x1^{α1} ... ∂xn^{αn}, α! = α1! ... αn!, (x-a)^α = (x1-a1)^{α1} ... (xn-an)^{αn}.

Infinite series if converges.

Matrix Form for Low Orders

Order 0: f(a)
1: + ∇f(a)^T (x-a)
2: + (1/2)(x-a)^T H(a) (x-a), H Hessian.

ML Insight

Hessian-vector products for efficient computation without full matrix.

3. First-Order (Linear) Approximations

f(x) ≈ f(a) + ∇f(a)^T (x-a)

Tangent hyperplane.

Properties

Best linear approx.
Directional deriv: ∇f · u.

ML Application

Gradient descent: Step along -∇f.
Linear models: Global first-order.

Example: f(x,y)=e^{x+y} at (0,0): ≈1 + x + y.

Error for (0.1,0.1): True e^{0.2}≈1.221, approx 1.2, error ~0.021.

4. Second-Order (Quadratic) Approximations

Add (1/2)(x-a)^T H(a) (x-a)

Captures curvature.

H symmetric if twice diff.

Eigenvalues: Principal curvatures.

Positive Definite H

All eigen >0: Local min.

ML Connection

Newton's method: Solve H Δx = -∇f for step.
Quasi-Newton: Approx H (BFGS).

Example: f(x,y)=x^2 + 2y^2 at (0,0): H=[[2,0],[0,4]], quadratic exact.

5. Higher-Order Terms and Tensors

For order 3: (1/6) sum third partials.

Higher: Multi-linear forms, tensors.

Computationally intensive.

In ML: Rarely used full, but in automatic diff.

6. Remainder Terms and Error Bounds

Multivar Lagrange: R_m(x) = \frac{1}{(m+1)!} D^{m+1} f(c) (x-a)^{m+1}, c on segment.

Bounds: If sup |D^{m+1} f| ≤ M, |R| ≤ M ||x-a||^{m+1} / (m+1)! (in some norm).

Taylor's theorem with integral remainder.

ML Insight

Error control in approximations for safety-critical AI.

7. Convergence in Multivariables

Series converges if analytic domain.

Radius: Complex analysis or bounds.

In ML: Local approximations suffice.

8. Geometric Interpretations

Linear: Tangent plane.

Quadratic: Paraboloid osculating surface.

Higher: Better fit to osculations.

In ML: Loss landscape visualization—valleys, saddles.

9. Numerical Computation of Multivariable Taylor

Use finite diffs or auto-diff.

Visualize approximations.

PythonRust

python

import numpy as np
from sympy import symbols, Matrix, hessian, diff

# Numerical quadratic approx
def f(x, y):
    return np.exp(x + y) + np.sin(x * y)

def grad_f(x, y, h=1e-5):
    fx = (f(x + h, y) - f(x - h, y)) / (2 * h)
    fy = (f(x, y + h) - f(x, y - h)) / (2 * h)
    return np.array([fx, fy])

def hess_f(x, y, h=1e-5):
    fxx = (f(x + h, y) - 2*f(x, y) + f(x - h, y)) / h**2
    fyy = (f(x, y + h) - 2*f(x, y) + f(x, y - h)) / h**2
    fxy = ((f(x + h, y + h) - f(x + h, y - h)) - (f(x - h, y + h) - f(x - h, y - h))) / (4 * h**2)
    return np.array([[fxx, fxy], [fxy, fyy]])

def quad_approx(x, y, a, b):
    delta = np.array([x - a, y - b])
    val = f(a, b)
    lin = grad_f(a, b) @ delta
    quad = 0.5 * delta.T @ hess_f(a, b) @ delta
    return val + lin + quad

a, b = 0, 0
x, y = 0.1, 0.1
print("True f:", f(x, y))
print("Quad approx:", quad_approx(x, y, a, b))

# Symbolic
x_sym, y_sym = symbols('x y')
f_sym = exp(x_sym + y_sym) + sin(x_sym * y_sym)
taylor_sym = f_sym.series(x_sym, 0, 3).removeO().series(y_sym, 0, 3).removeO()
print("Symbolic Taylor:", taylor_sym)

rust

fn f(x: f64, y: f64) -> f64 {
    (x + y).exp() + (x * y).sin()
}

fn grad_f(x: f64, y: f64, h: f64) -> [f64; 2] {
    let fx = (f(x + h, y) - f(x - h, y)) / (2.0 * h);
    let fy = (f(x, y + h) - f(x, y - h)) / (2.0 * h);
    [fx, fy]
}

fn hess_f(x: f64, y: f64, h: f64) -> [[f64; 2]; 2] {
    let fxx = (f(x + h, y) - 2.0 * f(x, y) + f(x - h, y)) / h.powi(2);
    let fyy = (f(x, y + h) - 2.0 * f(x, y) + f(x, y - h)) / h.powi(2);
    let fxy = ((f(x + h, y + h) - f(x + h, y - h)) - (f(x - h, y + h) - f(x - h, y - h))) / (4.0 * h.powi(2));
    [[fxx, fxy], [fxy, fyy]]
}

fn quad_approx(x: f64, y: f64, a: f64, b: f64) -> f64 {
    let delta = [x - a, y - b];
    let val = f(a, b);
    let grad = grad_f(a, b, 1e-5);
    let lin = grad[0] * delta[0] + grad[1] * delta[1];
    let hess = hess_f(a, b, 1e-5);
    let quad = 0.5 * (hess[0][0] * delta[0].powi(2) + 2.0 * hess[0][1] * delta[0] * delta[1] + hess[1][1] * delta[1].powi(2));
    val + lin + quad
}

fn main() {
    let a = 0.0;
    let b = 0.0;
    let x = 0.1;
    let y = 0.1;
    println!("True f: {}", f(x, y));
    println!("Quad approx: {}", quad_approx(x, y, a, b));
}

Computes numerical gradient, Hessian, quadratic approx.

10. Symbolic Multivariable Taylor

SymPy for exact.

PythonRust

python

from sympy import symbols, series, exp, sin

x, y = symbols('x y')
f = exp(x + y) + sin(x * y)
taylor = series(f, x, n=3).removeO().series(y, n=3).removeO()
print("Taylor:", taylor)

rust

// Simulated
fn main() {
    // Approximate expansion
    println!("Taylor: 1 + x + y + (x^2 + 2 x y + y^2)/2 + ...");
}

11. Applications in ML Optimization

Newton: Δw = -H^{-1} ∇f.

Levenberg-Marquardt: Dampened for trust.

In DL: Hessian-free opts use CG for Hv products.

Loss landscapes: Quadratic forms reveal minima/saddles.

12. Approximations in Neural Networks

Taylor for activations: GELU ≈ sigmoid-like.

Pruning: Second-order Taylor for saliency.

Adversarial: Linear approx for attacks.

13. Error Analysis and Bounds in Practice

Multivar Remainder: Generalize Lagrange.

In ML: Validate approx in step sizes.

Higher-order for accuracy in PINNs.

14. Convergence and Limitations

Domains where series converges.

In ML: Local validity—combine with global methods.

Numerical issues: Floating-point in high orders.

15. Key ML Takeaways

Linear/quad approx: Core to first/second-order opts.
Hessian curvature: Informs learning rates.
Error bounds: Ensure reliable approximations.
Multivar essential: For param spaces.
Code computes: Gradients, Hessians practically.

Multivariable Taylor unlocks local insights.

16. Summary

Thoroughly examined multivariable Taylor from intuition to higher orders, quadratic focus, with ML optimizations, approximations. Extended examples, Python/Rust code for computations. This deep dive enables precise function handling in AI.

Word count: Approximately 3850.

Multivariable Taylor Expansion & Quadratic Approximations ​

1. Intuition for Multivariable Taylor Expansions ​

ML Connection ​

Example ​

2. Formal Definition and Notation ​

Matrix Form for Low Orders ​

ML Insight ​

3. First-Order (Linear) Approximations ​

Properties ​

ML Application ​

4. Second-Order (Quadratic) Approximations ​

Positive Definite H ​

ML Connection ​

5. Higher-Order Terms and Tensors ​

6. Remainder Terms and Error Bounds ​

ML Insight ​

7. Convergence in Multivariables ​

8. Geometric Interpretations ​

9. Numerical Computation of Multivariable Taylor ​

10. Symbolic Multivariable Taylor ​

11. Applications in ML Optimization ​

12. Approximations in Neural Networks ​

13. Error Analysis and Bounds in Practice ​

14. Convergence and Limitations ​

15. Key ML Takeaways ​

16. Summary ​

Further Reading ​

Multivariable Taylor Expansion & Quadratic Approximations

1. Intuition for Multivariable Taylor Expansions

ML Connection

Example

2. Formal Definition and Notation

Matrix Form for Low Orders

ML Insight

3. First-Order (Linear) Approximations

Properties

ML Application

4. Second-Order (Quadratic) Approximations

Positive Definite H

ML Connection

5. Higher-Order Terms and Tensors

6. Remainder Terms and Error Bounds

ML Insight

7. Convergence in Multivariables

8. Geometric Interpretations

9. Numerical Computation of Multivariable Taylor

10. Symbolic Multivariable Taylor

11. Applications in ML Optimization

12. Approximations in Neural Networks

13. Error Analysis and Bounds in Practice

14. Convergence and Limitations

15. Key ML Takeaways

16. Summary

Further Reading