Daily Learning Experience · AI & Machine Learning · 30 August 2026
You are one pixel on the shin of a stranger dancing in their living room. You have twenty yes-or-no questions and about five thousandths of a second, and at the end of it you must say which part of a human being you are. Three hundred thousand other pixels are doing exactly the same thing, and none of you are allowed to talk to each other.
On 4 November 2010 a black bar went on sale that you put under your television. It had no buttons. Inside it, a small laser threw many thousands of invisible infrared dots across your carpet, your sofa, your cat and you — a fixed, known, speckled constellation. A camera beside the laser watched where each dot landed. A dot that fell on something near you sat slightly to one side of where it would have fallen on the far wall, and from that sideways nudge the bar computed a distance.
Thirty times a second it handed the console a picture where every pixel was not a colour but a number of metres. It could not tell your skin from your shirt. It could not see your face. What it produced was a landscape: near hills and far valleys, no labels, no names. The console now knew the shape of the room. It had no idea there was a person in it.
The real sensor was made by an Israeli company called PrimeSense and Microsoft called the trick light coding. The bench above is a simulation of the idea, not of the hardware: real dot patterns are pseudo-random and matched against a stored reference image. But the physics is the physics. Everything downstream of this point — everything the rest of this page is about — starts from a grid of distances and nothing else.
Here is the problem the engineers at Microsoft Research in Cambridge were staring at in 2009. You have a landscape of distances. Somewhere in it there is a person, in a pose nobody has ever struck before, wearing a coat, holding a cat, standing behind a coffee table, and you have to find their elbows. Thirty times a second. On a games console that also has a game to run.
The obvious approach is to fit a skeleton and then track it from frame to frame. The trouble with tracking is that when it loses you — and it loses you the moment you flop onto the sofa — it stays lost. So Jamie Shotton and his colleagues did something that sounds like giving up. They threw away tracking. They threw away the whole body. They decided to look at one pixel at a time and ask it a much stupider question: not "where is this person's elbow", but "which body part is this single dot on?"
Answer that for every pixel and the elbows fall out for free. And a stupid enough question can be answered stupidly fast.
So: you are a pixel. You know exactly one number — your own distance from the sensor. You are not allowed to see the picture. You are allowed to probe: name an offset, and someone will tell you the distance at that other spot. That's it. That's your whole world.
This is the entire feature the Kinect used. Written out, it is:
Read it aloud: take the depth at one offset, subtract the depth at another offset. The clever part is the little division. Both offsets are divided by your own depth, so an offset that means "thirty centimetres to the left" stays thirty centimetres whether you are standing at the telly or at the back of the room. In pixels it shrinks as you walk away. In the world it does not move.
One probe tells you almost nothing. A probe thirty centimetres above your head returns "far away — that's the wall" if you are on top of a shoulder, and "quite near" if you are on a knee with a torso above it. That is one bit of evidence. Now ask another. And another.
A decision tree is just a game of twenty questions where the questions are fixed in advance and the next one depends on the last answer. Each node holds one probe pair and one threshold. You fall left or right. Twenty levels down you arrive at a leaf, and in the leaf somebody has already written the answer — not one answer, but a tally: of the million training pixels that ever landed here, 71% were forearms, 18% were hands, 6% were shins…
The machine does not reason. It falls. Below is a real tree, trained in your browser a few seconds from now. Take a pixel and fall down it yourself.
Train the forest in Act Four first, then come back — this bench uses the tree you grow.
Where do a million labelled example pixels come from? Not from people. Nobody was ever going to hand-colour a million photographs of strangers into thirty-one body parts. Microsoft did something better: they built the people.
They took around 500,000 frames of motion capture — actors kicking, driving, dancing, running — and threw most of it away, keeping about 100,000 poses chosen so that no two were closer than five centimetres to each other. Then they hung those poses on fifteen different body shapes: tall, short, heavy, light, with different clothes and hair. Then they rendered the lot as computer graphics, and because they had rendered it, they knew the true body-part label of every single pixel for free.
That is what you are about to do, at one two-hundredth of the scale. Below, your browser will invent human beings — random heights, random builds, random distances, random poses — render each one as a depth picture with a perfect colour-coded answer key, and then grow real decision trees on them by measuring which probe pair splits the tally most cleanly. The measure it uses is Shannon's entropy: a node's job is to reduce your surprise.
Slide the bodies down to eight and grow the forest. The picture goes to confetti — it has memorised eight people and met a ninth. Slide it up and the confetti resolves into limbs. That gap between how well it does on the people it studied and how well it does on a stranger is overfitting, and it is the oldest problem in the subject. The cure here is the cure Microsoft used: not a cleverer machine, more people.
A single tree is a confident idiot. It commits hard to one chain of questions, and a chain of questions has a personality — it is unlucky in particular ways, blind in particular places. Grow a second tree from a different random handful of candidate questions and a different random handful of pixels, and it will be unlucky somewhere else.
So you grow several and let them vote. Not "majority wins" — each tree hands over its whole tally, its whole shade of opinion, and you add the tallies up. The mistakes, being uncorrelated, partly cancel. The agreements reinforce. This is Leo Breiman's random forest, published in 2001, and it is one of the few ideas in machine learning that is simultaneously easy to explain, easy to implement, and hard to beat.
One pixel, three opinions. Click a pixel in Act Three to change which one.
Remember the little "divide by your own depth" in the feature. Here is what it buys. Train a forest on people standing between 2.2 and 2.5 metres away — a narrow band, a single spot on the carpet — and then walk the stranger toward the sensor and away from it, out to places the forest has never seen anybody stand.
You now have a picture in which every pixel has an opinion about which body part it is. You still do not have an elbow. The last step is almost embarrassingly simple: every pixel votes for where it thinks the centre of its own body part is, the votes for each part pile up, and the densest pile is the joint. The real system used a technique called mean shift, weighted so that big blobs near the camera did not shout down small ones far away, and pushed each answer a few centimetres into the body, because a joint is inside you and pixels are on your surface.
The number that matters is the last line. On this page a whole frame is classified in well under a millisecond, because a pixel's entire cost is twenty array look-ups and twenty comparisons. That is the whole reason this method shipped instead of something cleverer.
The real forest was three trees, each twenty levels deep, grown from a million synthetic images on a thousand-core cluster in about a day. Thirty-one body parts. On the Xbox 360's graphics chip it classified an entire frame in under five milliseconds — around two hundred frames a second, when the sensor only produced thirty. The rest of the budget went to the game.
Kinect went on sale on 4 November 2010 and sold eight million units in sixty days, which Guinness certified as the fastest-selling consumer electronics device ever recorded — roughly 133,000 a day, ahead of the iPhone and the iPad over the same window. In June 2011 the Cambridge team won the Royal Academy of Engineering's MacRobert Award, and days later the paper won best paper at CVPR, the largest computer vision conference in the world.
And then it ended. Kinect was bundled with the Xbox One in 2013 at a price gamers resented, unbundled in 2014, and discontinued in October 2017. It is one of the strangest artefacts in the history of the field: a piece of research so successful that it dissolved. The body-tracking idea did not die with the hardware — it went into surgical theatres, physiotherapy clinics, retail analytics, sign-language research and a generation of robotics labs who bought a $150 depth camera because nothing else that cheap existed. The team scattered into self-driving cars, mixed reality and biology.
Eight questions. Wrong answers are more fun than right ones.
Sources: J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman & A. Blake, "Real-Time Human Pose Recognition in Parts from Single Depth Images", CVPR 2011 (best paper) and Communications of the ACM 56(1), 2013 · L. Breiman, "Random Forests", Machine Learning 45, 2001 · L. Breiman, "Statistical Modeling: The Two Cultures", Statistical Science 16(3), 2001 · T. K. Ho, "Random Decision Forests", ICDAR 1995 · Guinness World Records / Microsoft, March 2011 · Royal Academy of Engineering MacRobert Award 2011.
Every image, chart and body on this page is drawn from scratch in canvas. Nothing is stored, sent or remembered after you close the tab.