The Discontinuity Problem

The setup is deceptively simple. You have a target image and a scene whose parameters you want to recover: vertex positions, material roughness, albedo, light intensity.

Define a loss comparing your render to the target, then minimise it by gradient descent:

for step in range(n_steps):
    image = render(scene, params)
    loss  = ((image - target) ** 2).mean()
    grad  = d_loss_d_params(loss)      # the hard part
    params -= lr * grad

Every line is routine except one. Getting $\partial L / \partial \theta$ requires differentiating the entire light transport simulation: millions of rays, recursive bounces, random sampling.

The obvious approach is automatic differentiation, which handles far more complicated programs than this. And for a large class of parameters it works immediately.

Make a surface rougher, and every pixel it influences changes smoothly. Brighten a light, and the image scales smoothly. Change an albedo, and colours shift smoothly. For material and lighting parameters, naive automatic differentiation of a path tracer gives correct gradients.

Then you try to move a triangle, and it silently fails.

Consider a single pixel that currently sees the background, just past the edge of an object. Now move the object slightly so it covers that pixel.

The pixel's value jumps, from background colour to object colour. It does not change smoothly and pass through intermediate values, because the visibility test that produced it is a hard binary: the ray either hits the object or misses it.

Now ask automatic differentiation what happens. It differentiates the operations actually executed on this run. This ray missed the object, so the code path taken never touched the object's vertex positions at all. The derivative it reports with respect to those vertices is zero.

That is not an approximation error. It is the correct derivative of the function the program computed on that particular sample, and it is completely wrong for the quantity you want.

The underlying mathematics is this. Differentiating under the integral sign requires the integrand to be continuous in the parameter. Move a silhouette and the integrand jumps, so the derivative of the integral acquires a boundary term that the interior derivative misses entirely.

Automatic differentiation computes the interior term perfectly and the boundary term not at all.

Writing the structure explicitly makes the fix obvious in principle.

When you differentiate an integral whose integrand has a moving discontinuity, the result splits in two.

The interior term integrates the derivative of the integrand over the region where it is smooth. This captures how the shading changes: as a surface moves, its orientation relative to lights changes, distances change, and the shading changes smoothly with them.

The boundary term integrates the jump in the integrand along the moving discontinuity, weighted by how fast that boundary moves. This captures the actual silhouette motion, and it is concentrated on a set of measure zero: the silhouette curves themselves.

That last property is exactly why random sampling never finds it. A path tracer samples directions randomly, and the probability that a randomly sampled direction lands precisely on a silhouette edge is zero.

So the boundary term is not merely hard to sample. It is impossible to hit by chance, and no amount of extra samples helps.

Every solution in this area is therefore some way of accounting for the boundary term deliberately rather than hoping to sample it.

The failure is silent, which is what makes it dangerous. The optimiser receives a well-formed gradient of zero and concludes that geometry does not affect the image, so shapes simply never move.

The first approach is the most direct: if the boundary term lives on silhouette edges, then find the edges and sample them explicitly.

Tzu-Mao Li and colleagues introduced this in "Differentiable Monte Carlo Ray Tracing through Edge Sampling" (2018), the first technique to differentiate Kajiya's rendering equation with a Monte Carlo treatment of the discontinuities.

The method adds a second sampling process alongside the ordinary path tracing. Identify the silhouette edges in the scene, sample points along them, evaluate the radiance difference across each edge, and weight by how the edge moves with the parameter. The result is an unbiased estimate of the boundary term, which is then added to the interior term that automatic differentiation already provides.

The conceptual clarity is its strength: it computes the missing term directly, exactly as the mathematics prescribes.

The cost is a serious one. Finding silhouette edges requires explicit geometric queries over the scene, which is expensive and scales poorly with scene complexity. Secondary silhouettes, edges visible only in reflections or through refraction, are considerably harder still.

The second approach is cleverer and avoids finding edges at all.

Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob published "Reparameterizing Discontinuous Integrands for Differentiable Rendering" (2019), and the core idea is a change of variables.

The problem is that the discontinuity moves as the parameter changes, so the integration domain effectively shifts underneath you. The fix is to change variables so that in the new coordinates, the discontinuity stays put.

Think of it as attaching the coordinate system to the moving geometry. If the silhouette does not move in your chosen coordinates, then the integrand is no longer discontinuous in the parameter, and differentiating under the integral sign becomes valid again. Ordinary automatic differentiation then produces the correct total derivative, boundary term included.

The elegance is that the discontinuity is handled implicitly, without ever enumerating an edge, which sidesteps the geometric queries that make edge sampling expensive.

The difficulty moves into constructing a suitable reparameterization, which must follow the discontinuity's motion closely enough to work while remaining cheap to evaluate.

The third family refines the reparameterization idea and addresses a weakness in it.

Early reparameterization methods introduced bias: the change of variables was approximate, so the resulting gradients were close to correct rather than correct. For optimisation that is often acceptable and sometimes not, since biased gradients can converge to the wrong answer rather than merely converging slowly.

Sai Praveen Bangaru, Tzu-Mao Li, and Fredo Durand published "Unbiased Warped-Area Sampling for Differentiable Rendering" (2020), which converts the boundary integral into an area integral using a divergence theorem argument, then samples that area integral in a way that provably recovers the correct boundary term.

The result combines the properties the earlier methods each had separately: no explicit silhouette enumeration, and unbiased gradients.

Related work formulates the whole problem as differential path integrals, tracking discontinuities at the level of entire light paths rather than individual bounces, which extends the treatment to secondary effects such as moving shadows and reflected silhouettes.

This remains an active area. Recent work has continued to develop silhouette sampling and warped-area reparameterization, which tells you the problem is important and not considered closed.

The three families and what each costs.

Two practical warnings follow directly.

Not all parameters are equal. Materials, textures, and light intensities differentiate correctly under naive automatic differentiation, because their effect on the image is smooth. Only parameters that move discontinuity boundaries, principally geometry, need this machinery. A pipeline optimising only appearance may never encounter the problem.

The failure is silent. No exception, no warning, no obviously broken image. Geometry gradients simply come back as zero or near-zero, and the optimiser dutifully leaves shapes untouched while it fits materials. If your inverse rendering setup optimises appearance beautifully and refuses to move any shape, this is the first thing to check.

Correct gradients are necessary and not sufficient. They also have to be affordable, which is the next lesson.