Sous · CMU Human-AI Interaction
A hands-free AI cooking companion that watches your pan through a camera, tracks where you are in the recipe, and speaks up only when it's useful. In progress — final presentation early December 2026.
Timers tell you when time is up, not whether the onion is actually translucent. Recipe apps show the next step, but they don't know you're still on the last one. When something goes differently than the recipe expects, you're on your own.
Sous watches the pan. A camera sends snapshots to a vision-language model every few seconds, the model compares what it sees to the expected state of the current step, and Sous either advances, waits, or asks. Voice controls keep your hands free. Manual override keeps you in charge.
When should Sous move on? Too early and it's wrong; too late and it's useless. The answer we landed on: two consecutive confident observations at 85% or above before auto-advancing. One confident look could be a fluke. Two in a row is a pattern.
Sous also shows you its current belief: what it thinks it sees, what it's waiting for. If it's wrong, you correct it and move on. The design principle it comes from is the same as WorkBench HR: AI makes a suggestion, the person decides. "Wrong call? You're still the chef."
The MVP checkpoint covered recipe selection, step tracking with manual override, timer controls, and the camera panel. Action detection works for simple actions; more complex state recognition is the next problem. User testing in a real kitchen is mid-November. This page will update after the final presentation in December.