Computer Vision, Explained.

Features & Representation

John Canny's 1986 IEEE TPAMI paper that stated explicit performance criteria for edge detectors, derived optimal operators from them, and introduced the detector built on Gaussian derivatives, non-maximum suppression, and hysteresis thresholding.

Motion & Tracking

Rudolf Kalman's 1960 paper that recast Wiener's filtering problem in state-space form and solved it with a recursive estimator, the origin of the Kalman filter.

Image Processing & Computational Photography

The 2005 CVPR paper by Buades, Coll and Morel that introduced non-local means, averaging pixels whose surrounding patches look alike anywhere in the image, and proposed method noise as a way to evaluate denoisers.

Motion & Tracking

The 1981 IJCAI paper by Lucas and Kanade that replaced exhaustive search in image registration with a gradient-guided Newton–Raphson-type iteration, the origin of the Lucas–Kanade method.

Motion & Tracking

Why motion seen through a small window is ambiguous along edges, so that only the component of motion across an edge can be measured locally.

Motion & Tracking

Recursive estimation of the probability distribution of a hidden, changing state from a sequence of noisy measurements, by alternating a motion-model prediction with a Bayes' rule update.

Image Processing & Computational Photography

An edge-preserving smoothing filter that averages each pixel with neighbors that are both nearby and similar in value, so noise and texture are removed while strong edges stay sharp.

Image Processing & Computational Photography

The 1998 ICCV paper by Tomasi and Manduchi that combined domain and range weighting into a single non-iterative, edge-preserving smoothing filter, named it bilateral filtering, and applied it to color images in CIE-Lab.

Motion & Tracking

The assumption that a scene point keeps the same image intensity as it moves between frames, which turns motion estimation into an intensity-matching problem.

Motion & Tracking

A multi-object tracker that associates high-score detections first and then matches the remaining tracks to low-score detections, recovering occluded objects that score thresholds would discard.

Motion & Tracking

The ECCV 2022 paper by Zhang et al. that introduced BYTE, a second association round for low-confidence detections, and the ByteTrack tracker built on it with a YOLOX detector.

Features & Representation

The classic edge detection algorithm that smooths with a Gaussian, takes the gradient, thins edges by non-maximum suppression along the gradient direction, and links them with two thresholds and hysteresis.

Motion & Tracking

Estimating large motions with small-motion methods by solving on an image pyramid, from the coarsest level to full resolution, and refining the estimate at each level.

Foundations

A color space assigns numbers to colors; choosing and converting between RGB, CIE XYZ, CIELAB, YCbCr, and HSV correctly matters for both image processing and learning.

Motion & Tracking

Michael Isard and Andrew Blake's 1996 conference paper and 1998 journal paper that tracked object outlines through dense clutter by propagating a weighted random sample set over time, bringing particle filtering to computer vision.

Image Processing & Computational Photography

The operation that replaces every pixel by a weighted sum of its neighbors, with the weights given by a kernel; the basis of linear filtering and of convolutional networks.

Geometry & 3D Vision

A tensor of matching costs or similarities between each pixel of one image and its candidate correspondences in another, indexed by pixel position and candidate disparity, displacement, or depth.

Motion & Tracking

Deciding which measurements or detections belong to which tracked targets, and which are false alarms, missed detections, new targets, or targets that have disappeared.

Motion & Tracking

The 1981 paper by Horn and Schunck that computed dense optical flow by combining the brightness change constraint with a global smoothness assumption, founding the variational approach to motion estimation.

Foundations

A digital image is a grid of samples of a continuous scene, each quantized to a finite set of values and stored as an array of numbers.

Features & Representation

Locating the pixels where image intensity changes sharply, from derivative filters and the Canny detector to learned boundary detectors evaluated against human annotations.

Evaluation & Metrics

The standard accuracy measure for optical flow, the distance in pixels between an estimated flow vector and the true one, averaged over the image.

Motion & Tracking

A nonlinear extension of the Kalman filter that linearizes the motion and measurement models around the current estimate, widely used for camera pose estimation, visual-inertial odometry, and tracking with range or bearing sensors.

Features & Representation

Finding corresponding keypoints between two or more images of the same scene, from descriptor nearest-neighbor search to learned matchers.

Motion & Tracking

How distinctive image points are selected and followed across video frames, using the classic KLT tracker as the main example.

Datasets & Benchmarks

A synthetic optical flow dataset of rendered chairs moving over Flickr photographs, created to train FlowNet and still used to pretrain flow networks.

Image Processing & Computational Photography

The decomposition of an image into two-dimensional sinusoidal gratings of different frequencies, orientations, amplitudes, and phases, and the frequency-domain view of filtering, sampling, and registration that it provides.

Foundations

The bell-shaped probability distribution defined by a mean and a covariance, the default model for noise and uncertainty in estimation and tracking.

Image Processing & Computational Photography

A linear smoothing filter that replaces each pixel by an average of its neighbors weighted by a Gaussian, whose standard deviation sets the scale of the detail removed.

Evaluation & Metrics

A multi-object tracking metric that combines detection accuracy and association accuracy through their geometric mean, averaged over localization thresholds.

Image Processing & Computational Photography

A contrast enhancement method that maps each intensity level through the image's cumulative histogram so that output levels are used approximately uniformly, with adaptive variants such as CLAHE that equalize local regions.

Motion & Tracking

A global variational method that computes dense optical flow by minimizing brightness constancy errors together with a penalty on spatial variation of the flow, solved by a simple iterative averaging scheme.

Motion & Tracking

An algorithm that finds the minimum-cost one-to-one matching between two sets, used in computer vision to match detections to tracks and predictions to ground truth.

Evaluation & Metrics

An identity-based score for multi-object tracking that matches whole ground-truth and predicted trajectories one-to-one and reports the F1 score of correctly identified detections.

Image Processing & Computational Photography

Estimating a clean image from a noisy observation, removing random variation while keeping edges, texture, and fine detail.

Image Processing & Computational Photography

Computing each output pixel from a neighborhood of input pixels, the family of operations behind smoothing, sharpening, edge detection, and denoising.

Image Processing & Computational Photography

How image gradients measure the direction and strength of intensity change, and why they underpin edges, corners, and motion estimation.

Image Processing & Computational Photography

The count of pixels at each intensity or color level, an empirical distribution that summarizes exposure and contrast and underlies thresholding, equalization, and histogram-based recognition and tracking.

Foundations

The random variation in recorded pixel values that does not come from the scene, its physical sources in the camera, and how it is modeled and measured.

Image Processing & Computational Photography

A multi-scale representation of an image built by repeatedly smoothing and subsampling it, so that each level holds a coarser copy at half the resolution of the one below.

Image Processing & Computational Photography

Transforming the geometry of an image by a coordinate mapping, usually computed by sampling the input image at the inverse-mapped position of every output pixel.

Motion & Tracking

A recursive algorithm that estimates the hidden state of a linear dynamic system from a sequence of noisy measurements, widely used to smooth and predict object positions in tracking.

Datasets & Benchmarks

A benchmark suite of real driving data from Karlsruhe, with stereo, optical flow, scene flow, odometry, object detection, and tracking benchmarks built from camera, LiDAR, and GPS/IMU recordings.

Image Processing & Computational Photography

The sum of the second partial derivatives of an image, an isotropic operator whose zero crossings mark edges, whose Gaussian-smoothed form detects blobs across scales, and which appears in sharpening, pyramids, and variational methods.

Foundations

The method of fitting a model to more measurements than unknowns by minimizing the sum of squared residuals, the workhorse of estimation in computer vision.

Motion & Tracking

A local, gradient-based method that estimates the displacement of an image window by assuming constant motion within it and solving a small least-squares problem, iterated with warping.

Image Processing & Computational Photography

A theory and toolbox for processing images by shape, probing them with a small set called a structuring element through erosion, dilation, and the operators built from them, such as opening, closing, top-hats, and morphological reconstruction.

Image Processing & Computational Photography

A nonlinear filter that replaces each pixel by the median of its neighborhood, removing impulse noise while keeping step edges sharp.

Datasets & Benchmarks

Small, high-accuracy stereo and optical flow benchmarks from Middlebury College and Microsoft Research that defined how dense correspondence methods were evaluated in the 2000s.

Datasets & Benchmarks

A benchmark for multiple pedestrian tracking, released as MOT15, MOT16, MOT17, and MOT20, with hidden test annotations and a central evaluation server.

Motion & Tracking

Recovering how image content, objects, or the camera moved from a sequence of images, from per-pixel flow to global and 3D motion.

Datasets & Benchmarks

An optical flow benchmark rendered from the open-source animated film Sintel, with long sequences, large motions, motion blur, and atmospheric effects.

Motion & Tracking

Estimating the trajectories of a varying, unknown number of objects in a video while keeping each object's identity consistent over time.

Evaluation & Metrics

The CLEAR MOT accuracy score for multi-object tracking, which counts misses, false positives, and identity switches over a sequence and normalizes them by the number of ground-truth objects.

Image Processing & Computational Photography

A denoising algorithm that replaces each pixel with a weighted average of pixels whose surrounding patches look similar, wherever they are in the image.

Motion & Tracking

Estimating the position, extent, or state of one or more objects in every frame of a video, keeping each object's identity over time.

Motion & Tracking

The apparent motion of image content between two frames, represented as a two-dimensional displacement at every pixel.

Motion & Tracking

Computing a dense field of pixel displacements between two video frames, from classical variational methods to learned networks such as RAFT.

Motion & Tracking

A sequential Monte Carlo method that represents the probability distribution of a hidden state with weighted random samples, so it can track through nonlinear models and ambiguous, multimodal beliefs.

Motion & Tracking

A deep network for optical flow that matches all pairs of pixels once and refines a single flow field with a recurrent update operator.

Motion & Tracking

The ECCV 2020 best paper by Teed and Deng that replaced coarse-to-fine flow networks with an all-pairs correlation volume and a weight-tied recurrent update.

Image Processing & Computational Photography

How a continuous image becomes a grid of samples, when the samples determine it completely, and how detail too fine for the grid turns into false, coarser patterns.

Motion & Tracking

The 2016 paper by Bewley et al. that introduced SORT, showing that a Kalman filter and Hungarian matching on strong CNN detections rival far more complex online multi-object trackers.

Motion & Tracking

The 2017 paper by Wojke, Bewley, and Paulus that introduced DeepSORT, extending SORT with a CNN appearance descriptor trained for person re-identification and a matching cascade, and reporting about 45% fewer identity switches on MOT16.

Image Processing & Computational Photography

A pair of 3×3 kernels that estimate the horizontal and vertical image derivatives by combining a central difference with smoothing across it; the most common way to compute image gradients.

Motion & Tracking

Simple Online and Realtime Tracking, a multi-object tracker that links per-frame detections into tracks using a constant-velocity Kalman filter on each box and Hungarian matching on box overlap.

Motion & Tracking

Building object tracks by running a detector on every frame and linking its detections over time with a motion model and data association.