Rendering as an Integral

Computer graphics has traditionally run one way: describe a scene, produce an image. Geometry, materials, lights and a camera go in; pixels come out.

The inverse question is the one that has become interesting. Given an image, what scene produced it? Recover the shapes, the material properties, the lighting.

That is a far older ambition than graphics, and it is what vision has always wanted. The insight driving current work is that if your forward renderer is differentiable, the inverse problem becomes ordinary gradient-based optimisation: render a guess, compare to the target image, compute how each scene parameter should change, and step.

Wenzel Jakob leads the Realistic Graphics Lab at EPFL, whose work spans exactly this pair, developing rendering algorithms for creating realistic images of virtual worlds and inverse rendering algorithms for reconstructing 3D worlds from images. Their system Mitsuba 3 is a research-oriented renderer for both forward and inverse light transport.

This lesson builds the forward direction properly, because you cannot differentiate what you do not understand as a mathematical object.

Start with the physical quantity. A pixel's value is proportional to the radiance arriving at that point of the sensor from the direction the pixel looks.

Radiance is the fundamental quantity of light transport: energy per unit time, per unit area, per unit solid angle. It has a property that makes everything else work, namely that radiance is constant along a ray in empty space. So the light arriving at a pixel equals the light leaving whatever surface that ray first hits, in the direction of the camera.

That reduces rendering to a single question, asked once per ray: how much light leaves this point in this direction?

The answer has two parts. Some light is emitted, if the surface is a light source. The rest is reflected, and this is where the difficulty lives: to know how much light a surface reflects toward the camera, you must know how much light arrives at it from every other direction.

And that incoming light came from other surfaces, which were themselves lit by other surfaces. The problem is recursive by nature, not by choice of algorithm.

James Kajiya formalised this in 1986, and the resulting expression is the foundation of physically based rendering.

The outgoing radiance $L_o$ from a point x in direction $\omega_o$ is

$L_o(x, \omega_o) = L_e(x, \omega_o) + \int_{\Omega} f_r(x, \omega_i, \omega_o)\, L_i(x, \omega_i)\, (\omega_i \cdot n)\, d\omega_i$

Each piece has a physical meaning worth naming.

$L_e$ is emitted radiance, non-zero only for light sources.

The integral runs over the hemisphere of directions above the surface, collecting incoming light from everywhere.

$f_r$ is the BSDF, the material's scattering function: given light arriving from $\omega_i$, what fraction is scattered toward $\omega_o$? Mirrors, matte paint, and skin differ entirely in this one function.

$L_i$ is the incoming radiance from direction $\omega_i$.

The cosine term accounts for light striking at a glancing angle spreading over more area.

The defining feature is that $L_i$ is itself an $L_o$ from another surface. The unknown appears on both sides. This is an integral equation, and outside trivial scenes it has no closed-form solution.

Expanding the recursion shows what you are really up against. Light reaching the camera may have bounced once, twice, or many times, so the full solution is a sum over paths of every length, and a path of length k requires an integral over k directions.

Rendering is therefore a very high-dimensional integration problem, and dimension is what rules out the obvious approaches.

Deterministic quadrature fails badly. Splitting each direction into n samples costs $n^k$ evaluations for a k-bounce path, which is hopeless past a couple of bounces. This is the curse of dimensionality in its purest form.

Monte Carlo integration escapes it. Sample directions randomly, evaluate the integrand, average. The estimator is unbiased, and its error decreases as $1/\sqrt{N}$ in the number of samples regardless of dimension.

That dimension-independence is the whole reason the field is built on random sampling. The price is visible in every render: the error appears as noise, and quartering it requires sixteen times the samples.

So path tracing is not a shortcut or an approximation of some better method. It is a statistical estimator, and every render is a noisy estimate of a well-defined integral.

The algorithm follows directly. Trace a ray from the camera, and at each surface randomly choose a direction to continue, accumulating the material response as you go.

def trace(ray, depth=0):
    hit = scene.intersect(ray)
    if not hit:
        return background(ray.direction)

    L = hit.emitted(-ray.direction)          # is this a light?

    if depth >= max_depth:
        return L

    # sample a new direction from the material's BSDF
    wi, pdf = hit.bsdf.sample(-ray.direction)
    f = hit.bsdf.eval(-ray.direction, wi)
    cos = abs(dot(wi, hit.normal))

    # Monte Carlo estimate of the hemispherical integral
    L += f * cos / pdf * trace(Ray(hit.point, wi), depth + 1)
    return L

The division by pdf is the Monte Carlo correction: sampling a direction more often than uniformly means down-weighting it proportionally, which keeps the estimator unbiased whatever the sampling strategy.

That freedom is what makes importance sampling possible. Draw directions where the integrand is large, and variance collapses. Sampling proportional to the BSDF is standard; sampling directly toward light sources is the other essential strategy, and combining them well is what separates a usable renderer from a noisy one.

The pipeline is a nested estimation: each pixel averages many paths, each path is a random walk through the scene, and each bounce is one Monte Carlo sample of a hemispherical integral. Everything about differentiating a renderer follows from this structure.

Two features of the integrand cause the practical difficulty, and both matter enormously once differentiation enters.

High dynamic range and concentration. A small bright light source contributes enormously over a tiny solid angle. Sampling directions uniformly means mostly missing it, and occasionally hitting it with a huge value. That produces the characteristic bright speckles of an undersampled render, and it is why sampling light sources directly is essential.

Discontinuities. This is the one to hold onto. The integrand jumps discontinuously where visibility changes. Rotate your sampled direction slightly and it may cross the silhouette of an occluder, so the incoming radiance switches abruptly from a bright light to a dark shadowed surface.

For forward rendering, discontinuities are merely a variance problem. Monte Carlo integration is perfectly happy to integrate a discontinuous function: the estimator remains unbiased, and you handle the extra variance with better sampling.

That tolerance disappears the moment you want derivatives. The single hardest problem in differentiable rendering is these visibility discontinuities, and the next lesson is largely about why and what to do.

Summarising what has been established, because each item becomes a constraint on the inverse problem.

Two points are worth carrying forward explicitly.

A rendered image is a random variable, not a deterministic function of the scene. Two renders of the same scene with different random seeds differ. Any optimisation built on top inherits that noise in its objective.

And the physical grounding is what makes inversion meaningful at all. Because the forward model is a real simulation of light transport rather than a heuristic, the parameters it recovers, roughnesses, albedos, geometry, are physically interpretable and transferable to other scenes and other renderers.

That is the promise. The obstacle is the last row of the table.