Keypoints, Landmarks, and Human Pose

A pair of coordinates means nothing on its own. The point “left wrist” depends on image orientation, person identity, anatomy, and whether the joint is visible at all. Without those conventions it cannot be read correctly.

COCO does not leave that to convention. It writes the contract into the annotation format. The keypoint detection section of its data format page specifies: “Each keypoint has a 0-indexed location x,y and a visibility flag v defined as v=0: not labeled (in which case x=y=0), v=1: labeled but not visible, and v=2: labeled and visible.” The same page fixes what visibility means operationally rather than perceptually: “A keypoint is considered visible if it falls inside the object segment”. A landmark in COCO is three numbers. The third one tells you how to read the first two.

That flag is not bookkeeping. Ronchi and Perona diagnosed the COCO keypoint benchmark in 2017 and found that 86.6% of person annotations have five or more visible keypoints. The thin end of that distribution is where detection stops working: “Over 30% of the annotations are missed when the number of visible keypoints is less than 5”. The state has to be recorded, because it cannot be inferred.

Pose datasets must define pixel centers, missing points, self-occlusion, truncation, mirrored images, and which person owns each landmark.

Representation choice changes how uncertainty and structure are expressed. Direct coordinates predict x and y for each named point. Heatmaps predict a spatial confidence surface per landmark. Part affinity or relation fields represent connections between body parts. Kinematic or graph structure adds constraints among joints, limbs, or object landmarks. A shipped system rarely picks only one of them.

Google's BlazePose, published in 2020 and shipped in MediaPipe, states the trade in one sentence: “In contrast to heatmap-based techniques, regression-based approaches, while less computationally demanding and more scalable, attempt to predict the mean coordinate values, often failing to address the underlying ambiguity.” Its answer was to use both, at different times. An encoder-decoder heatmap-and-offset branch exists purely as supervision: “We use the heatmap and offset loss only in the training stage and remove the corresponding output layers from the model before running the inference.” At inference only the coordinate-regression head runs. It was trained by a distribution it no longer computes.

The landmark set is a design decision too, not an inheritance. Google's own disclosure gives the reason for leaving COCO's 17 points behind: “the COCO keypoints only localize to the ankle and wrist points, lacking scale and orientation information for hands and feet”. Instead, “we present a new topology of 33 human body keypoints, which is a superset of COCO, BlazeFace and BlazePalm topologies”, and the model “predicts the location of all 33 person keypoints with three degrees of freedom each (x, y location and visibility)”. Visibility travels as a first-class channel, not as an afterthought bolted onto a coordinate pair.

The price of that choice is documented rather than argued. On Google's AR dataset BlazePose scores 84.1 PCK@0.2 against OpenPose's 87.8, with a human annotator baseline of 97.2. But it runs at 10 FPS on a single Pixel 2 core, where OpenPose manages 0.4 FPS on a 20-core desktop CPU.

Both strategies can work. Their bottlenecks differ — until the scene fills up, at which point both collapse.

Both routes have produced state-of-the-art results on the same benchmark. Cao and colleagues took the bottom-up route with part affinity fields. Their architecture allows “a greedy bottom-up parsing step that maintains high accuracy while achieving realtime performance, irrespective of the number of people in the image”. The method “placed first in the inaugural COCO 2016 keypoints challenge”. Papandreou and colleagues went the other way, describing “a simple, yet powerful, top-down approach consisting of two stages”. Person boxes come from “the Faster RCNN detector”, and then “For each keypoint type we predict dense heatmaps and offsets using a fully convolutional ResNet”. “Trained on COCO data alone, our final system achieves average precision of 0.649 on the COCO test-dev set”. Neither ordering is inherently right.

CrowdPose measures what actually separates them, and the answer is: less than the architecture diagrams suggest. Its 2019 paper assembled 20,000 images containing about 80,000 persons, chosen for overlap. “It turns out that CrowdPose has an average bounding box IoU of 0.27, while MSCOCO, MPII, and AI Challenger have 0.06, 0.11 and 0.12 respectively.” Split by Crowd Index into easy, medium and hard, both families fall by roughly the same amount. The bottom-up OpenPose goes 62.7 → 48.7 → 32.3 AP. The top-down AlphaPose goes 71.2 → 61.4 → 51.1. Twenty points of AP is not a grouping problem or a detection problem. It is the same crowding problem arriving through two different doors.

OpenMMLab's MMPose reproduces the collapse independently on CrowdPose test with a YOLOv3 human detector: HRNet-W32 at 256x192 scores 0.770 AP easy, 0.687 medium, 0.553 hard. Top-down keeps person ownership simple and inherits the detector's failures. Bottom-up shares image computation and inherits the grouping's. Single-object landmarking assumes one aligned crop and has the lowest association complexity of the three, which is another way of saying it has no answer when a second person walks into frame.

Visibility and identity flags carry information that a bare point omits. Both benchmarks this lesson keeps returning to measured what happens when that information is missing or ignored.

Pose errors often originate outside the landmark head. Verify crop and transforms first; separate visibility cases; inspect association; test temporal stability; then evaluate the downstream use.

Step one is not a formality, and the evidence is published rather than folklore. The standard horizontal-flip test used at inference is quietly broken. Huang and colleagues reported it in 2020: “we find that the results obtained by common flipping strategy are unaligned with the original ones in inference”. The pixel-based coordinate-system transformation carries a statistical error on top of that. Fixing both, under the name Unbiased Data Processing: “UDP promotes SimpleBaseline-ResNet50-256x192 by 1.5 AP (70.2 to 71.7) and HRNet-W32-256x192 by 1.7 AP (73.5 to 75.2) on COCO test-dev set.” HRNet-W48-384x288 reaches 76.5 AP. MMPose reproduces it on COCO val2017 under a detector with human AP 56.4: HRNet-W32 at 256x192 moves from 0.749 to 0.762 AP with UDP, and HRNet-W48-384x288 with UDP reaches 0.773. Mirror handling and inverse mapping are worth more than most architecture changes.

Step two has the visibility states from the annotation format to separate on. Step three has a vocabulary and a price list rather than an impression. Ronchi and Perona decompose keypoint localization failures against COCO ground truth into four named types — jitter, inversion, swap and miss — and rank them: “misses are the most costly error in terms of AP (∼15%), followed by inversions (∼4%)”. Counting inversions and swaps separately in crowded scenes tells you whether to fix the detector, the grouping, or the left-right supervision. A single AP number does not.

Steps four and five stay where the numbers cannot reach. Measure jitter, missing points and identity changes across frames. Then check whether the gesture, ergonomics or control decision built on those coordinates remains correct. And note the second half of that flipping paper, which is the one people skip: “there is a statistical error in some keypoint format transformation methods”. The bug is in the format, not in the network.

Multiple peaks or a broad heatmap can indicate several plausible landmark locations; converting that distribution to a single coordinate discards uncertainty. That conversion is not a formatting step. It is a modelling decision with a measurable price.

The 2020 paper behind the DARK method opens on exactly this point: “For the first time, we find that the process of decoding the predicted heatmaps into the final joint coordinates in the original image space is surprisingly significant for the performance.” The standard quarter-pixel shifting step of coordinate decoding, taken alone, is worth as much as 5.7% AP with HRNet-W32 on COCO validation — 61.2 with no shifting against 66.9 with standard shifting. An unremarked line of post-processing moves the score more than most reported architectural gains.

The distribution matters most exactly where it is hardest to read. The same figure holds on the encoding side: “the aforementioned shifting operation of coordinate encoding brings as high as 5.7% AP on the challenging COCO validation set”. At low resolution the loss compounds: “the model performance of HRNet-W32 drops significantly from 74.4% to 66.9% on the COCO validation set” when the input falls from 256x192 to 128x96. Distribution-aware decoding with unbiased encoding lifts that same 128x96 model from 66.9 to 70.7 AP. It recovers a large share of the 7.5 AP the resolution drop cost, by reading the heatmap better rather than by making the model bigger. A different group, the Unbiased Data Processing authors, reported the same class of statistical error in the same year.

Downstream policies may use peak sharpness, visibility scores, temporal consistency, or ensemble variation; none of them says on a calibrated scale that the pose is right until someone has validated it.

Pose estimates become valuable only after their limits are connected to the application. Each of these questions has a published number behind it somewhere.