Computer Vision, Explained.

Features & Representation

Chris Harris and Mike Stephens's 1988 Alvey Vision Conference paper that replaced Moravec's shift test with the local auto-correlation matrix, introduced the det − k·trace² response, and used it to find corners and edges together for 3D interpretation of image sequences.

Features & Representation

John Canny's 1986 IEEE TPAMI paper that stated explicit performance criteria for edge detectors, derived optimal operators from them, and introduced the detector built on Gaussian derivatives, non-maximum suppression, and hysteresis thresholding.

Geometry & 3D Vision

H. C. Longuet-Higgins's 1981 Nature letter that introduced to computer vision the matrix now called the essential matrix and the linear eight-point algorithm for recovering relative pose and scene structure from two calibrated views.

Geometry & 3D Vision

Zhengyou Zhang's 1999 ICCV paper and 2000 IEEE TPAMI paper that calibrated a camera from a few views of a printed planar pattern, using a closed-form solution from plane homographies followed by maximum-likelihood refinement with radial distortion.

Motion & Tracking

Rudolf Kalman's 1960 paper that recast Wiener's filtering problem in state-space form and solved it with a recursive estimator, the origin of the Kalman filter.

Image Processing & Computational Photography

The 2005 CVPR paper by Buades, Coll and Morel that introduced non-local means, averaging pixels whose surrounding patches look alike anywhere in the image, and proposed method noise as a way to evaluate denoisers.

Geometry & 3D Vision

Scharstein and Szeliski's 2002 IJCV paper that broke dense stereo algorithms into four building blocks, compared them on images with ground truth, and started the Middlebury stereo benchmark.

Evaluation & Metrics

The fraction of test examples a classifier labels correctly, with its top-k, balanced, and per-pixel variants, and why it can mislead on imbalanced data or small test sets.

Deep Learning for Vision

The fixed nonlinear function a neural network applies to each unit's weighted sum, such as ReLU or GELU, without which any stack of layers would collapse into a single linear map.

Deep Learning for Vision

A stochastic gradient method that gives every network weight its own step size, dividing a running average of the gradient by the root of a running average of its square; the Vision Transformer, for example, was pre-trained with it.

Motion & Tracking

The 1981 IJCAI paper by Lucas and Kanade that replaced exhaustive search in image registration with a gradient-guided Newton–Raphson-type iteration, the origin of the Lucas–Kanade method.

Recognition & Detection

Reference boxes of fixed sizes and aspect ratios, tiled at every feature-map position, that a detector classifies and refines, and that decide which of its predictions are trained on which objects.

Motion & Tracking

Why motion seen through a small window is ambiguous along edges, so that only the component of motion across an edge can be measured locally.

Deep Learning for Vision

The 2017 NIPS paper by Vaswani and colleagues at Google that introduced the transformer, an encoder–decoder for machine translation built from attention alone, which later became a standard architecture for image classification and object detection.

Evaluation & Metrics

The area under a detector's precision–recall curve for one class, computed from score-ranked detections matched to ground truth by IoU, and its averages over classes and IoU thresholds (mAP and COCO AP).

Deep Learning for Vision

The algorithm that computes the gradient of a network's loss with respect to every weight by applying the chain rule backward through the computation graph, at a cost comparable to one forward pass.

Deep Learning for Vision

A layer that normalizes each channel of a feature map with the mean and variance of the current mini-batch, then rescales it with learned parameters; it allows higher learning rates in convolutional networks but depends on the batch size and behaves differently in training and inference.

Deep Learning for Vision

Sergey Ioffe and Christian Szegedy's 2015 ICML paper that introduced batch normalization and used it to train an Inception image classifier in a fraction of the steps, with higher learning rates and without dropout.

Motion & Tracking

Recursive estimation of the probability distribution of a hidden, changing state from a sequence of noisy measurements, by alternating a motion-model prediction with a Bayes' rule update.

Image Processing & Computational Photography

An edge-preserving smoothing filter that averages each pixel with neighbors that are both nearby and similar in value, so noise and texture are removed while strong edges stay sharp.

Image Processing & Computational Photography

The 1998 ICCV paper by Tomasi and Manduchi that combined domain and range weighting into a single non-iterative, edge-preserving smoothing filter, named it bilateral filtering, and applied it to color images in CIE-Lab.

Recognition & Detection

The smallest axis-aligned rectangle that encloses an object in an image, the standard way to say where an object is in detection, tracking, and annotation.

Motion & Tracking

The assumption that a scene point keeps the same image intensity as it moves between frames, which turns motion estimation into an intensity-matching problem.

Geometry & 3D Vision

The joint nonlinear least-squares refinement of all camera parameters and 3D points in a reconstruction, made practical by the sparse structure of the problem and the Schur complement.

Motion & Tracking

A multi-object tracker that associates high-score detections first and then matches the remaining tracks to low-score detections, recovering occluded objects that score thresholds would discard.

Motion & Tracking

The ECCV 2022 paper by Zhang et al. that introduced BYTE, a second association round for low-confidence detections, and the ByteTrack tracker built on it with a YOLOX detector.

Geometry & 3D Vision

Estimating a camera's intrinsic parameters, lens distortion, and pose from images of a target with known geometry, usually by a closed-form initialization followed by minimizing reprojection error.

Foundations

The parameters of the pinhole camera matrix P = K[R | t], split into intrinsics K, which map camera coordinates to pixels, and extrinsics R and t, which place the camera in the world.

Features & Representation

The classic edge detection algorithm that smooths with a Gaussian, takes the gradient, thins edges by non-maximum suppression along the gradient direction, and links them with two thresholds and hysteresis.

Datasets & Benchmarks

A dataset of stereo video from the streets of 50 cities, mostly in Germany, with 5,000 finely and 20,000 coarsely annotated frames, whose benchmarks for pixel-level, instance-level, and panoptic segmentation became the standard for urban scene understanding.

Motion & Tracking

Estimating large motions with small-motion methods by solving on an image pyramid, from the coarsest level to full resolution, and refining the estimate at each level.

Datasets & Benchmarks

Microsoft's Common Objects in Context, a dataset of everyday scenes with per-instance segmentation masks for 80 object categories, plus keypoints, captions, stuff and panoptic labels, whose evaluation protocol became the standard benchmark for object detection.

Foundations

A color space assigns numbers to colors; choosing and converting between RGB, CIE XYZ, CIELAB, YCbCr, and HSV correctly matters for both image processing and learning.

Motion & Tracking

Michael Isard and Andrew Blake's 1996 conference paper and 1998 journal paper that tracked object outlines through dense clutter by propagating a weighted random sample set over time, bringing particle filtering to computer vision.

Image Processing & Computational Photography

The operation that replaces every pixel by a weighted sum of its neighbors, with the weights given by a kernel; the basis of linear filtering and of convolutional networks.

Deep Learning for Vision

A neural network built from convolutional layers, small learned filters applied at every image position, interleaved with nonlinearities and downsampling, so that features grow from edges to object parts while the parameter count stays independent of image size.

Geometry & 3D Vision

A tensor of matching costs or similarities between each pixel of one image and its candidate correspondences in another, indexed by pixel position and candidate disparity, displacement, or depth.

Deep Learning for Vision

The loss that scores a predicted class distribution by the negative log-probability it gives the true label; it is the standard loss for image classification and per-pixel segmentation, the negative log-likelihood of a softmax or sigmoid output, and differs from the KL divergence by a constant.

Motion & Tracking

Deciding which measurements or detections belong to which tracked targets, and which are false alarms, missed detections, new targets, or targets that have disappeared.

Deep Learning for Vision

The 2016 CVPR paper by He, Zhang, Ren and Sun of Microsoft Research that introduced residual networks (ResNets), trained them to 152 layers on ImageNet, and won the ILSVRC 2015 classification task.

Segmentation

A family of semantic segmentation networks from Google and UCLA (2015–2018) built on atrous (dilated) convolution, which keeps feature maps dense without extra parameters, atrous spatial pyramid pooling for multi-scale context, a fully connected CRF for boundaries in its first two versions, and a light decoder in DeepLabv3+.

Geometry & 3D Vision

Recovering the distance to the scene at every pixel of an image, from stereo pairs, many views, active sensors, or a single image.

Geometry & 3D Vision

An image whose pixels store the distance to the scene, usually the depth Z along the optical axis, so that each pixel with a known camera can be lifted to a 3D point.

Motion & Tracking

The 1981 paper by Horn and Schunck that computed dense optical flow by combining the brightness change constraint with a global smoothness assumption, founding the variational approach to motion estimation.

Recognition & Detection

A transformer-based object detector that predicts a fixed-size set of boxes in parallel and is trained with a one-to-one Hungarian matching loss, so it needs neither anchors nor non-maximum suppression.

Evaluation & Metrics

Twice the overlap of a predicted and a ground-truth mask divided by their total size, equal to the F1 score over pixels; the standard overlap score of medical image segmentation and, in soft form, a training loss.

Foundations

A digital image is a grid of samples of a continuous scene, each quantized to a finite set of values and stored as an array of numbers.

Geometry & 3D Vision

The horizontal shift, in pixels, between the images of the same scene point in a rectified stereo pair; it is inversely proportional to depth.

Features & Representation

David Lowe's 1999 ICCV paper and 2004 IJCV journal paper that introduced SIFT, a scale- and rotation-invariant local feature, and used it to recognize objects in cluttered, partly occluded images.

Deep Learning for Vision

A regularization technique that randomly zeroes units of a neural network during training, with structured variants for convolutional feature maps and a test-time form for uncertainty estimates.

Features & Representation

Locating the pixels where image intensity changes sharply, from derivative filters and the Canny detector to learned boundary detectors evaluated against human annotations.

Deep Learning for Vision

A network design in which an encoder compresses the input into a coarse, many-channel representation and a decoder expands it back into the output space, usually at the input's resolution; the standard architecture for segmentation, depth, flow, image translation and pose heatmaps, and, without the spatial grid, for sequence-to-sequence models.

Evaluation & Metrics

The standard accuracy measure for optical flow, the distance in pixels between an estimated flow vector and the true one, averaged over the image.

Geometry & 3D Vision

The geometry of two views of the same scene, in which the two camera centers and any scene point span a plane, so that a point in one image can only match points on a single line in the other.

Geometry & 3D Vision

The 3 × 3 matrix E = [t]×R that encodes the relative rotation and translation direction between two calibrated views, how it is estimated from point matches, and how the camera pose is recovered from it.

Motion & Tracking

A nonlinear extension of the Kalman filter that linearizes the motion and measurement models around the current estimate, widely used for camera pose estimation, visual-inertial odometry, and tracking with range or bearing sensors.

Recognition & Detection

A two-stage object detector in which a region proposal network, sharing the detector's convolutional features, replaces hand-designed region proposals, making proposal generation nearly free and the whole detector one network.

Recognition & Detection

The 2015 NIPS paper by Ren, He, Girshick and Sun that replaced hand-designed region proposals with a region proposal network sharing the detector's convolutional features, making proposals nearly free and the whole detector one network.

Segmentation

The fully convolutional network of Long, Shelhamer and Darrell (2015), a semantic segmentation network trained end to end from pixels to pixels; it recasts a classification CNN's fully connected layers as convolutions, upsamples coarse class scores in-network, and fuses scores from finer layers (FCN-32s, FCN-16s, FCN-8s).

Features & Representation

Vectors that summarize the image appearance around a keypoint so that the same scene point seen in different images gets similar vectors, from raw patches and gradient histograms to binary strings and learned embeddings.

Features & Representation

Finding distinctive, repeatable points and regions in an image, such as corners, blobs, and stable regions, so that the same scene locations can be found again in other images.

Features & Representation

Finding corresponding keypoints between two or more images of the same scene, from descriptor nearest-neighbor search to learned matchers.

Recognition & Detection

A network design that turns a convolutional backbone's feature maps at strides 4 to 32 into a pyramid with strong semantic features at every scale, through a top-down pathway and lateral connections, so a detector can find small and large objects from one image scale.

Motion & Tracking

How distinctive image points are selected and followed across video frames, using the classic KLT tracker as the main example.

Geometry & 3D Vision

The minimal solver for the relative pose of two calibrated cameras, which computes every essential matrix consistent with five point matches by reducing the problem to the roots of a tenth-degree polynomial.

Datasets & Benchmarks

A synthetic optical flow dataset of rendered chairs moving over Flickr photographs, created to train FlowNet and still used to pretrain flow networks.

Image Processing & Computational Photography

The decomposition of an image into two-dimensional sinusoidal gratings of different frequencies, orientations, amplitudes, and phases, and the frequency-domain view of filtering, sampling, and registration that it provides.

Segmentation

The CVPR 2015 paper by Long, Shelhamer and Darrell that recast ImageNet classification networks as fully convolutional networks and trained them end to end, pixels to pixels, for semantic segmentation, replacing patchwise and proposal-based pipelines.

Geometry & 3D Vision

The rank-2 3 × 3 matrix F that encodes the epipolar geometry of two uncalibrated views through the constraint x'ᵀFx = 0, and how it is estimated from point correspondences.

Foundations

The bell-shaped probability distribution defined by a mean and a covariance, the default model for noise and uncertainty in estimation and tracking.

Image Processing & Computational Photography

A linear smoothing filter that replaces each pixel by an average of its neighbors weighted by a Gaussian, whose standard deviation sets the scale of the detail removed.

Foundations

An iterative method that minimizes a differentiable function by repeatedly stepping in the direction of the negative gradient, the basic optimizer behind variational image models and neural network training.

Segmentation

Energy minimization by minimum s–t cut, which labels the pixels of an image optimally for binary energies with submodular pairwise terms and approximately, through expansion and swap moves, for many labels.

Features & Representation

A corner detector that scores each pixel by how much a small window's intensities change under shifts in every direction, using the determinant and trace of the gradient structure tensor.

Evaluation & Metrics

A multi-object tracking metric that combines detection accuracy and association accuracy through their geometric mean, averaged over localization thresholds.

Image Processing & Computational Photography

A contrast enhancement method that maps each intensity level through the image's cumulative histogram so that output levels are used approximately uniformly, with adaptive variants such as CLAHE that equalize local regions.

Geometry & 3D Vision

Representing a point by a vector with one extra coordinate, defined only up to scale, so that translations, perspective projections, and points at infinity can all be handled with matrix multiplication.

Geometry & 3D Vision

An invertible projective transformation of the plane, a 3×3 matrix defined up to scale, that relates two images of a planar scene or two images taken by a camera rotating about its center.

Motion & Tracking

A global variational method that computes dense optical flow by minimizing brightness constancy errors together with a penalty on spatial variation of the flow, solved by a simple iterative averaging scheme.

Motion & Tracking

An algorithm that finds the minimum-cost one-to-one matching between two sets, used in computer vision to match detections to tracks and predictions to ground truth.

Evaluation & Metrics

An identity-based score for multi-object tracking that matches whole ground-truth and predicted trajectories one-to-one and reports the F1 score of correctly identified detections.

Recognition & Detection

Assigning one or more category labels to a whole image, usually by predicting a probability for each of a fixed set of classes.

Image Processing & Computational Photography

Estimating a clean image from a noisy observation, removing random variation while keeping edges, texture, and fine detail.

Image Processing & Computational Photography

Computing each output pixel from a neighborhood of input pixels, the family of operations behind smoothing, sharpening, edge detection, and denoising.

Image Processing & Computational Photography

How image gradients measure the direction and strength of intensity change, and why they underpin edges, corners, and motion estimation.

Image Processing & Computational Photography

The count of pixels at each intensity or color level, an empirical distribution that summarizes exposure and contrast and underlies thresholding, equalization, and histogram-based recognition and tracking.

Foundations

The random variation in recorded pixel values that does not come from the scene, its physical sources in the camera, and how it is modeled and measured.

Image Processing & Computational Photography

A multi-scale representation of an image built by repeatedly smoothing and subsampling it, so that each level holds a coarser copy at half the resolution of the one below.

Segmentation

Dividing an image into regions by assigning every pixel to a segment, a class, or an object instance, the task behind semantic, instance, panoptic and interactive segmentation.

Image Processing & Computational Photography

Transforming the geometry of an image by a coordinate mapping, usually computed by sampling the input image at the inverse-mapped position of every output pixel.

Datasets & Benchmarks

A large database of web images organized by the noun hierarchy of WordNet, whose 1,000-class ILSVRC subset became the standard benchmark for image classification and the default source of pretrained networks.

Deep Learning for Vision

Krizhevsky, Sutskever and Hinton's 2012 NIPS paper, often called the AlexNet paper, which trained a large convolutional network on two GPUs, won ILSVRC-2012 with a top-5 test error of 15.3% against 26.2% for the next entry, and turned computer vision toward deep learning.

Datasets & Benchmarks

Deng, Dong, Socher, Li, Li and Fei-Fei's CVPR 2009 paper that introduced ImageNet, a database of web images organized by the WordNet noun hierarchy and verified by crowd workers, which later became the basis of the ILSVRC benchmark.

Geometry & 3D Vision

Richard Hartley's 1995 ICCV paper and 1997 journal paper, which showed that translating and scaling image coordinates before running the linear eight-point algorithm makes it a practical estimator of the fundamental matrix, nearly as accurate as iterative methods.

Segmentation

Detecting every object of a fixed set of "thing" classes and giving each one its own pixel mask, class label and confidence score, so that two touching cars come out as two cars rather than one car-colored region.

Evaluation & Metrics

The area where a predicted region and the ground truth overlap, divided by the area they cover together, the standard way to score localization in detection, segmentation, and tracking.

Geometry & 3D Vision

An algorithm that rigidly aligns two point clouds without known correspondences by alternating closest-point matching with a least-squares estimate of the rotation and translation.

Motion & Tracking

A recursive algorithm that estimates the hidden state of a linear dynamic system from a sequence of noisy measurements, widely used to smooth and predict object positions in tracking.

Datasets & Benchmarks

A benchmark suite of real driving data from Karlsruhe, with stereo, optical flow, scene flow, odometry, object detection, and tracking benchmarks built from camera, LiDAR, and GPS/IMU recordings.

Image Processing & Computational Photography

The sum of the second partial derivatives of an image, an isotropic operator whose zero crossings mark edges, whose Gaussian-smoothed form detects blobs across scales, and which appears in sharpening, pyramids, and variational methods.

Foundations

The method of fitting a model to more measurements than unknowns by minimizing the sum of squared residuals, the workhorse of estimation in computer vision.

Foundations

The systematic deviation of a real lens from the pinhole model's straight-line projection, which bends straight lines in the image and is modeled, estimated, and removed as part of camera calibration.

Deep Learning for Vision

The function that scores how far a network's output is from the desired output on one example; training minimizes its average over the training images, and the choice of loss encodes what counts as a good prediction for each vision task.

Motion & Tracking

A local, gradient-based method that estimates the displacement of an image window by assuming constant motion within it and solving a small least-squares problem, iterated with warping.

Segmentation

An instance segmentation model that adds a small fully convolutional mask branch and RoIAlign to the Faster R-CNN detector, predicting a box, a class and a binary mask for every detected object.

Segmentation

The 2017 ICCV paper by He, Gkioxari, Dollár and Girshick that turned Faster R-CNN into an instance segmentation system by adding a parallel mask branch and RoIAlign, won the ICCV 2017 Best Paper award (Marr Prize), and became a widely used baseline for instance-level recognition.

Image Processing & Computational Photography

A theory and toolbox for processing images by shape, probing them with a small set called a structuring element through erosion, dilation, and the operators built from them, such as opening, closing, top-hats, and morphological reconstruction.

Image Processing & Computational Photography

A nonlinear filter that replaces each pixel by the median of its neighborhood, removing impulse noise while keeping step edges sharp.

Datasets & Benchmarks

Small, high-accuracy stereo and optical flow benchmarks from Middlebury College and Microsoft Research that defined how dense correspondence methods were evaluated in the 2000s.

Datasets & Benchmarks

A benchmark for multiple pedestrian tracking, released as MOT15, MOT16, MOT17, and MOT20, with hidden test annotations and a central evaluation server.

Motion & Tracking

Recovering how image content, objects, or the camera moved from a sequence of images, from per-pixel flow to global and 3D motion.

Datasets & Benchmarks

An optical flow benchmark rendered from the open-source animated film Sintel, with long sequences, large motions, motion blur, and atmospheric effects.

Motion & Tracking

Estimating the trajectories of a varying, unknown number of objects in a video while keeping each object's identity consistent over time.

Geometry & 3D Vision

The geometry that relates cameras, scene points, and their images across one, two, three, or many views, and that determines what can be reconstructed from images and up to which ambiguity.

Geometry & 3D Vision

Recovering dense 3D geometry, as depth maps, point clouds, meshes, or volumes, from many images whose camera poses and intrinsics are known.

Evaluation & Metrics

The CLEAR MOT accuracy score for multi-object tracking, which counts misses, false positives, and identity switches over a sequence and normalizes them by the number of ground-truth objects.

Deep Learning for Vision

A function built from layers of simple units, each computing a weighted sum of its inputs plus a bias followed by a nonlinearity, whose weights are learned from data, for example to map an image's pixels to class scores.

Image Processing & Computational Photography

A denoising algorithm that replaces each pixel with a weighted average of pixels whose surrounding patches look similar, wherever they are in the image.

Recognition & Detection

The post-processing step that removes duplicate detections by repeatedly keeping the highest-scoring box and discarding the boxes that overlap it beyond an IoU threshold, with its soft, distance-aware and learned variants.

Foundations

Fitting a model that is nonlinear in its parameters by iteratively minimizing a sum of squared residuals, the optimization behind calibration, triangulation, homography refinement, and bundle adjustment.

Segmentation

A graph-partitioning method for image segmentation that measures a cut relative to the total connection of each side, and approximates the best partition with the second eigenvector of a generalized eigenvalue problem; the basis of spectral clustering in vision.

Segmentation

The 2000 IEEE TPAMI paper by Shi and Malik, first presented at CVPR 1997, that cast image segmentation as partitioning a weighted pixel graph, defined the normalized cut criterion, and approximated it with a generalized eigenvector, bringing spectral graph partitioning into computer vision.

Recognition & Detection

Finding every object of a set of categories in an image and returning, for each one, a bounding box, a class label and a confidence score.

Motion & Tracking

Estimating the position, extent, or state of one or more objects in every frame of a video, keeping each object's identity over time.

Recognition & Detection

Object detectors that predict class scores and box offsets densely at every position of a feature map, or of a feature pyramid, in a single network pass, with no region-proposal stage.

Motion & Tracking

The apparent motion of image content between two frames, represented as a two-dimensional displacement at every pixel.

Motion & Tracking

Computing a dense field of pixel displacements between two video frames, from classical variational methods to learned networks such as RAFT.

Segmentation

Giving every pixel of an image a class label and, for countable objects, an instance id, so that stuff such as sky and road and things such as cars and people are segmented together without overlaps, and scoring the result with panoptic quality (PQ).

Motion & Tracking

A sequential Monte Carlo method that represents the probability distribution of a hidden state with weighted random samples, so it can track through nonlinear models and ambiguous, multimodal beliefs.

Datasets & Benchmarks

The PASCAL Visual Object Classes challenge (2005–2012), whose 20-class datasets of Flickr photographs with bounding boxes and segmentations set the standard protocol for evaluating object detection.

Geometry & 3D Vision

Estimating the position and orientation of a calibrated camera from known 3D points and their 2D image projections, the absolute pose problem behind visual localization, augmented reality, and camera tracking.

Geometry & 3D Vision

Snavely, Seitz and Szeliski's 2006 SIGGRAPH paper and 2008 IJCV journal paper that recovered camera poses and sparse 3D points from internet photos of landmarks and used them to browse the photos in 3D.

Foundations

The ideal camera in which every scene point projects through a single center onto an image plane, giving the perspective projection x = fX/Z, y = fY/Z.

Geometry & 3D Vision

A set of 3D points sampled from the surfaces of a scene, possibly with color, intensity, or normals per point, and the most direct output of 3D sensors and reconstruction.

Deep Learning for Vision

A fixed, parameter-free layer that summarizes each small window of a feature map by its maximum or average, shrinking the map, widening what later layers see, and tolerating small shifts at the cost of resolution and aliasing.

Evaluation & Metrics

Two complementary measures of a classifier or detector, the fraction of its positive predictions that are correct and the fraction of true positives it finds, with the F-score, the confusion matrix, and the precision–recall and ROC curves built from them.

Geometry & 3D Vision

The geometry of properties that survive perspective projection, such as straightness, incidence, and the cross-ratio, which underlies how computer vision models cameras and relates images.

Motion & Tracking

A deep network for optical flow that matches all pairs of pixels once and refines a single flow field with a recurrent update operator.

Motion & Tracking

The ECCV 2020 best paper by Teed and Deng that replaced coarse-to-fine flow networks with an all-pairs correlation volume and a weight-tied recurrent update.

Geometry & 3D Vision

Martin Fischler and Robert Bolles's 1981 Communications of the ACM paper that introduced RANSAC and used it to solve the location determination problem, including a closed-form analysis of the perspective-three-point problem.

Geometry & 3D Vision

Random sample consensus, a randomized method that fits a model to data containing many outliers by repeatedly fitting minimal random samples and keeping the model that the most data points agree with.

Deep Learning for Vision

The region of the input image that can affect one unit of a convolutional network, how kernel size, depth, stride, pooling and dilation make it grow, and why the effective, gradient-weighted region is much smaller than the theoretical one.

Deep Learning for Vision

Any change to how a network is trained that aims to lower its error on new images rather than on the training images, from weight penalties and data augmentation to early stopping and the implicit bias of SGD.

Geometry & 3D Vision

The image distance, in pixels, between where a point was measured and where the estimated cameras and 3D structure say it should appear; the standard cost minimized in calibration, triangulation, pose estimation, and bundle adjustment.

Deep Learning for Vision

A shortcut that adds a block's input to its output, y = x + F(x), so that the layers learn only a correction to the identity; it lets very deep convolutional networks and transformers train, because signals and gradients always have a path that bypasses the weights.

Recognition & Detection

The CVPR 2014 paper by Girshick, Donahue, Darrell and Malik that introduced R-CNN, classifying bottom-up region proposals with fine-tuned ImageNet CNN features, and raised PASCAL VOC detection mAP by more than 30% relative to the previous best.

Segmentation

The Segment Anything Model, a promptable segmentation model that returns object masks for points, boxes or rough masks, trained with a model-in-the-loop data engine on the SA-1B dataset of 1.1 billion masks, and the starting point of a family of faster, sharper, video and medical variants.

Image Processing & Computational Photography

How a continuous image becomes a grid of samples, when the samples determine it completely, and how detail too fine for the grid turns into false, coarser patterns.

Deep Learning for Vision

An operation in which every token of a set, such as the patches of an image, computes its output as a weighted average of all tokens, with weights given by a softmax over scaled dot products between learned queries and keys.

Segmentation

Labeling every pixel of an image with one of a fixed set of classes, such as road, sky, car or person, without separating individual objects; the dense-prediction counterpart of image classification, also called scene parsing.

Geometry & 3D Vision

A dense stereo algorithm that approximates a global smoothness energy by summing one-dimensional dynamic-programming path costs from several directions through every pixel.

Features & Representation

The Scale-Invariant Feature Transform, which detects blob-like keypoints at extrema of a difference-of-Gaussians scale space and describes each with a 128-dimensional histogram of gradient orientations that is invariant to scale and rotation.

Motion & Tracking

The 2016 paper by Bewley et al. that introduced SORT, showing that a Kalman filter and Hungarian matching on strong CNN detections rival far more complex online multi-object trackers.

Motion & Tracking

The 2017 paper by Wojke, Bewley, and Paulus that introduced DeepSORT, extending SORT with a CNN appearance descriptor trained for person re-identification and a matching cascade, and reporting about 45% fewer identity switches on MOT16.

Geometry & 3D Vision

Building a map of an unknown environment while estimating the trajectory of the sensor moving through it, and using recognized revisits to keep both globally consistent.

Image Processing & Computational Photography

A pair of 3×3 kernels that estimate the horizontal and vertical image derivatives by combining a central difference with smoothing across it; the most common way to compute image gradients.

Motion & Tracking

Simple Online and Realtime Tracking, a multi-object tracker that links per-frame detections into tracks using a constant-velocity Kalman filter on each box and Hungarian matching on box overlap.

Geometry & 3D Vision

Finding, for every pixel of one rectified stereo image, its corresponding pixel on the same row of the other image, producing a disparity map from which depth follows.

Geometry & 3D Vision

Heiko Hirschmüller's 2005 CVPR paper and 2008 TPAMI journal paper that introduced semi-global matching, a dense stereo method combining a pixel-wise mutual information cost with a smoothness energy approximated along many one-dimensional paths.

Geometry & 3D Vision

Warping a pair of stereo images by one homography each so that corresponding epipolar lines become the same image row, turning the correspondence search into a scan along rows.

Geometry & 3D Vision

Recovering depth from two images taken by cameras with a known relative pose, by measuring how far each point shifts between the views.

Deep Learning for Vision

Gradient descent in which each step uses the average gradient over a small random mini-batch of training images instead of the whole dataset, usually with momentum; it is the basic optimizer for training neural networks.

Geometry & 3D Vision

Recovering the poses and intrinsics of the cameras that took a set of images, together with a sparse 3D point cloud of the scene, from the images alone, by incremental or global reconstruction.

Segmentation

An oversegmentation of an image into a few hundred compact, roughly uniform regions that follow image boundaries, used in place of pixels to make later processing cheaper.

Segmentation

Segmenting an image by comparing each pixel's intensity with a threshold, chosen by hand, from the histogram as in Otsu's method, or from each pixel's neighborhood as in Sauvola's method.

Motion & Tracking

Building object tracks by running a detector on every frame and linking its detections over time with a motion model and data association.

Deep Learning for Vision

The practical workflow for fitting a neural network to labeled images, from splitting the data and normalizing pixels through the training loop, learning curves and early stopping to hyperparameter search, debugging checks, reproducibility and memory.

Deep Learning for Vision

A neural network architecture that stacks identical blocks of multi-head self-attention and a position-wise MLP, each wrapped in a residual connection with layer normalization; introduced for machine translation and now used for image classification, detection and segmentation on sequences of image patches.

Deep Learning for Vision

The layer that maps a small feature map to a larger one by applying the transpose of a strided convolution's matrix, equivalently the convolution's gradient with respect to its input; the learned upsampling step of segmentation decoders and image generators.

Geometry & 3D Vision

Recovering the 3D position of a point from its projections in two or more images taken by cameras whose calibration and poses are known.

Recognition & Detection

Object detectors that first propose a few thousand class-agnostic candidate regions and then classify and refine each one with features pooled from a shared CNN feature map, the design line of R-CNN, Fast R-CNN, Faster R-CNN, and Mask R-CNN.

Segmentation

A convolutional encoder–decoder for segmentation whose contracting path gathers context and whose symmetric expanding path, fed by concatenated skip connections, restores pixel-accurate boundaries; designed in 2015 for biomedical images with few labels, and since the base of 3D U-Net, Attention U-Net, UNet++, nnU-Net and the denoisers of diffusion models.

Segmentation

The 2015 MICCAI paper by Ronneberger, Fischer and Brox that introduced U-Net, a fully convolutional network with a symmetric expanding path, and showed that with elastic-deformation augmentation it could learn microscopy segmentation from 20 to 35 annotated images.

Geometry & 3D Vision

Estimating the motion of a camera, and of the vehicle or device carrying it, incrementally from a video stream, frame after frame, without building a globally consistent map.

Segmentation

A morphological segmentation algorithm that treats an image, usually its gradient magnitude, as a relief and floods it from its minima or from markers, so that regions are catchment basins and boundaries are the lines where floods meet.

Recognition & Detection

A family of real-time one-stage object detectors, begun by Redmon et al. in 2016, that predict every box and class probability of an image in a single network pass; later versions from several groups added anchors, multi-scale heads, anchor-free heads and NMS-free training.

Recognition & Detection

The 2016 CVPR paper by Redmon, Divvala, Girshick and Farhadi that introduced YOLO, recasting object detection as one regression from image pixels to boxes and class probabilities, solved by a single network fast enough for live video.