ML for Science - Lecture 5
From differential equations to predictions on a computer
We've seen how to extract models from data (learning/fitting).
Now: How do we use models to make predictions?
The most basic form of computing: apply a rule repeatedly
Peano Axioms: Generating Natural Numbers
Start with 0, apply successor function S repeatedly → all natural numbers
This is just a for loop:
i = 0
for _ in range(n):
i = i + 1 # successorSimple rule → Complex, interesting sequence
def fib(n):
a, b = 0, 1
for _ in range(n):
a, b = b, a + b
return aProperties emerge:
Planetary distances follow a pattern:
The process:
Von Neumann's idea (1940s) + Conway's rules (1970): simple local rules → complex emergent behavior
Click cells to toggle | Generation: 0
Recall Galileo's projectile: $\displaystyle\frac{d^2x}{dt^2} = -g$
We integrated analytically: $x(t) = -\frac{1}{2}gt^2 + v_0 t + x_0$
Solution: Solve on a computer by going back to discrete
Turn derivatives back into differences:
For $\frac{d^2x}{dt^2} = -g$, introduce velocity $v = \frac{dx}{dt}$:
Blue dots = Numerical (Euler) | Green dashed = Analytical solution
$\Delta t$ controls the accuracy of the approximation
The Fur Trade Business:
Hudson Bay Company records (1900-1920)
■ Hare | ■ Lynx (thousands)
Lotka (1925) and Volterra (1926) independently proposed:
The logic:
$x$ = prey (hares), $y$ = predators (lynx)
$\color{#98c379}{\alpha}$ = prey growth |
$\color{#e5c07b}{\beta}$ = predation rate
$\color{#c678dd}{\delta}$ = predator efficiency |
$\color{#56b6c2}{\gamma}$ = predator death
Apply Euler's method to each equation:
Dots = Real data | Lines = Model simulation | Hare | Lynx
The equation of motion:
For small angles: $\sin\theta \approx \theta$ → simple harmonic motion
For large angles: nonlinear! Must solve numerically.
Introduce angular velocity $\omega = \frac{d\theta}{dt}$, then apply Euler:
Left: Pendulum animation | Right: Phase space (θ, ω)
Multiple trajectories in phase space for different initial conditions
Small θ₀: circles | Large θ₀: distorted "eye" shapes
Unlike Fibonacci (exact), numerical solutions have error. Use Taylor expansion:
Rearranging for the derivative:
Key insight: Euler method has error $O(\Delta t)$ — halving step size halves the error
Central difference is more accurate for the same $\Delta t$!
Error: $O((\Delta t)^2)$
Now $u(x, t)$ depends on both space and time
Examples where diffusion appears:
Define a grid with spacing $\Delta x$ in space and $\Delta t$ in time:
Apply finite differences to both derivatives:
u value (0 to 1)
"Step Stencil" moves one cell in space | "Next Row" completes a time step
Each new value depends on neighboring values:
FTCS
(Forward Time,
Central Space)
$u_i^{n+1} = u_i^n + r(\Delta u)$
BTCS
(Backward Time,
Central Space)
Solve: $Au^{n+1} = u^n$
Crank-Nicolson
(Average
FTCS + BTCS)
$O((\Delta t)^2, (\Delta x)^2)$ accuracy
Try r > 0.5 to see instability!
How do we build numerical methods that respect physical laws?
The workhorse of scientific computing
Sample the slope at 4 points, take a weighted average → 4th-order accuracy!
ode45, SciPy's solve_ivp)When equations have conservative form:
Finite volume methods ensure conservation is preserved in the discretization.
Theme in Scientific ML: Encode physical knowledge into numerical schemes
Conservation: What enters one cell exits the neighbor. No numerical mass loss.
Red = source | Mouse = obstacle (blocks flow) | Real Navier-Stokes in browser!
Each step solves part of Navier-Stokes numerically:
Reference: Jos Stam, "Stable Fluids" (SIGGRAPH 1999)
After diffusion and advection, velocity may not satisfy $\nabla \cdot \mathbf{u} = 0$
Solve Poisson equation for pressure:
The physics: Fluid heated from below, cooled from above.
Barry Saltzman (Yale, 1961) developed a 7-equation model for convection.
He showed it to Edward Lorenz at MIT — one solution "refused to settle down."
Lorenz noticed: 4 variables quickly became tiny. Only 3 were "keeping each other going."
Saltzman's 7 equations → Lorenz's 3 equations:
Just 3 coupled ODEs, yet the dynamics are incredibly complex
The trajectory never repeats but stays on this strange "butterfly" shape
Lorenz wanted to extend a simulation. Instead of starting over, he typed in the values from a printout:
A difference of about 0.0001 — surely that can't matter?
He goes for a coffee, comes back, and...
The two simulations start nearly identical, then completely diverge.
■ Original vs ■ Perturbed by ε
No matter how small ε is, the trajectories eventually diverge completely.
"Does the flap of a butterfly's wings in Brazil set off a tornado in Texas?"
Lorenz's discovery:
If systems are chaotic and sensitive...
How do we account for uncertainty?
See you next time!