Cameras and Visual Perception

A pinhole camera maps a 3D point $\mathbf{P} = [X, Y, Z]^\top$ (in camera coordinates) to a 2D pixel $\mathbf{p} = [u, v]^\top$ by perspective projection:

$u = f_x \frac{X}{Z} + c_x, \quad v = f_y \frac{Y}{Z} + c_y$

In homogeneous form this is $\lambda\,\tilde{\mathbf{p}} = K\,\mathbf{P}$, where $\lambda = Z$ and the intrinsic matrix is

$K = \begin{bmatrix} f_x & s & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}$

$f_x, f_y$ are focal lengths in pixels; $(c_x, c_y)$ is the principal point (typically near image center); $s$ is the skew (zero for modern sensors). The pinhole model is linear after the perspective divide — that linearity is what makes calibration tractable. The division by $Z$ destroys depth: all 3D points on the same ray project to the same pixel, so the inverse mapping is a ray, not a unique 3D location.

The intrinsic matrix $K$ lives in camera space. To project a world point $\mathbf{P}_w$, you first transform it into camera coordinates via the extrinsic rotation $R \in SO(3)$ and translation $\mathbf{t}$:

$\mathbf{P}_c = R\,\mathbf{P}_w + \mathbf{t}$

The complete projection is $\lambda\,\tilde{\mathbf{p}} = K[R\;|\;\mathbf{t}]\,\tilde{\mathbf{P}}_w$, giving a $3 \times 4$ camera matrix $P = K[R\;|\;\mathbf{t}]$.

For a robot, $R$ and $\mathbf{t}$ encode the camera's pose — which changes every time the robot moves. $K$ is fixed at manufacturing. Getting $K$ wrong by even a few pixels in principal point or 0.5% in focal length compounds downstream in any structure-from-motion or visual odometry pipeline. Extrinsic calibration (camera-to-LiDAR, camera-to-IMU) is a separate problem — and even 1° of angular error in a camera-LiDAR extrinsic produces metres of pixel misalignment at 20 m range.

Real lenses deviate from the ideal pinhole. The dominant term is radial distortion: straight lines bow inward (barrel) or outward (pincushion). The distortion model maps an ideal pixel $\mathbf{p}$ to an observed pixel $\mathbf{p}'$:

$r^2 = (u - c_x)^2 + (v - c_y)^2$ $u' = u(1 + k_1 r^2 + k_2 r^4 + k_3 r^6) + \text{tangential terms}$

Calibration via a checkerboard (Zhang's method) jointly estimates $K$ and the distortion coefficients $[k_1, k_2, k_3, p_1, p_2]$. With OpenCV:

import cv2
ret, K, dist, rvecs, tvecs = cv2.calibrateCamera(
    obj_points, img_points, img_size, None, None
)
# Undistort a frame:
img_rect = cv2.undistort(frame, K, dist)

Never skip calibration. A well-calibrated fisheye with $k_1 = -0.4$ will project more accurately than a poorly-calibrated telephoto. Target $< 0.5$ px RMS reprojection error.

To track the camera or recognize objects, we need repeatable keypoints — image locations detectable from multiple viewpoints. The Harris corner detector finds pixels where image intensity varies strongly in two directions (large $\lambda_{\min}$ of the second-moment matrix). ORB (Oriented FAST + Rotated BRIEF) extends this with a binary descriptor computed from intensity comparisons:

orb = cv2.ORB_create(nfeatures=500)
kp1, des1 = orb.detectAndCompute(img1, None)
kp2, des2 = orb.detectAndCompute(img2, None)
bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=False)
matches = bf.knnMatch(des1, des2, k=2)
# Lowe ratio test
good = [m for m, n in matches if m.distance < 0.75 * n.distance]

After matching, RANSAC + the essential matrix filters outliers. The essential matrix $E = [\mathbf{t}]_\times R$ encodes relative camera motion and satisfies $\mathbf{p}_2^\top E\, \mathbf{p}_1 = 0$ for true correspondences. Outlier rates of 60-80% are common in textureless environments — always check inlier count before trusting a pose estimate.

A calibrated stereo pair with baseline $b$ and focal length $f$ recovers depth $Z$ from the disparity $d = u_L - u_R$ (horizontal pixel shift between left and right views):

$Z = \frac{f \cdot b}{d}$

Depth uncertainty scales with $Z^2$: double the range and depth error quadruples. The stereo matching problem (finding corresponding pixels) is solved by SGM (Semi-Global Matching) in OpenCV's StereoSGBM. Failure modes include textureless surfaces (no gradient $\Rightarrow$ ambiguous disparity), transparent objects, and specular reflections. Stereo works poorly beyond $b \cdot f / 1\text{ px}$ — roughly 10 m for a 12 cm baseline with $f = 700$ px. Baseline is a design tradeoff: wider $b$ improves long-range depth but creates more occlusion near the stereo boundary.

Deep networks have transformed robot perception. Key capabilities:

But CNNs have failure modes a perception engineer must know: (1) domain shift — a model trained on indoor images fails outdoors in rain; (2) scale ambiguity — monocular depth networks cannot recover metric scale without additional supervision; (3) adversarial fragility — small perturbations fool detectors with high confidence; (4) latency on edge hardware — a model fast on a V100 may be too slow on a Jetson Orin without quantization. Always validate on your deployment distribution, not just the benchmark.

From world coordinates to pixel — the full chain.

Visual odometry (VO) chains pairwise camera pose estimates to track robot motion. The algorithm at each frame: detect features $\rightarrow$ match to prior frame $\rightarrow$ estimate essential matrix $\rightarrow$ recover $R, \mathbf{t}$ via decomposition $\rightarrow$ triangulate new map points. The recovered translation is scale-ambiguous per frame; monocular VO fixes scale heuristically or with an IMU.

The painful truth: VO drifts. Without loop closure, a 100 m trajectory can accumulate 2-5% positional error. SLAM systems (covered in a later lesson) add loop-closure detection to correct accumulated drift globally. VO is still useful for short horizons and for bootstrapping a SLAM system's front end. A good rule of thumb: if your robot needs to return to a starting location, VO alone is insufficient — you need a map.

No single camera configuration is best for all robot tasks. Practical tradeoffs:

For production robots, fusing a stereo camera with an IMU (VIO) and optionally LiDAR (covered next) is the current best practice for robust state estimation in unstructured environments.