A New Way to Teach Robots: In-Context Learning In Robotics for High-Mix Manufacturing
TL;DR
At Shiraz AI, we’re building robots that adapt directly from human demonstrations without specialized gloves or a new training cycle for every task variation.
We recently showcased this approach live at Y Combinator’s Demo Day, where we showed a task to the robot and the robot picked it up instantly. For example, in the videos below we demonstrate picking a package and placing it in different orientations on the conveyor belt. The robot sees the demonstration, and performs the task over and over again.
For a manufacturer, a robot’s value depends on more than how well it performs a task. It also depends on how easily it can adapt when that task changes.
That is especially important in high-mix manufacturing, where workstations handle different products, configurations, and workflows. Automation must accommodate that variety, and the effort required to configure and reconfigure a robot is central to its economic value.
We are developing a robot foundation model that uses human demonstrations as context to adapt to new task variations.
The capability behind this is called robot in-context learning. We believe it can fundamentally change the economics of deployment and open new markets for robotic automation.
What does it mean for a robot to learn “in context”?
In-context learning changes the role of a demonstration.
In a conventional demonstration-based training workflow, examples become training data: collect demonstrations, update the model, evaluate the resulting policy, and deploy it. In an in-context workflow, the demonstration becomes an input to an already-trained model. The robot uses it to determine how to act, without updating the model’s parameters for that task.
The distinction is between using an example to retrain the model and using an example to instruct the model.
This does not mean the robot starts with no prior knowledge. Its underlying capabilities still come from training. In-context learning allows it to apply those capabilities using information supplied at execution time, rather than requiring a separate fine-tuning cycle for every new instruction.
A demonstration can also communicate information that is awkward to express in words. “Pack these parts” leaves room for interpretation. Showing the operation can clarify which object to handle, where to place it, and the sequence of intermediate steps.
The goal is not simply to replay a recorded motion. A useful policy must interpret the demonstration alongside the robot’s current observations and generate actions appropriate to the situation in front of it.
Why this matters for high-mix manufacturing
The business case for automation becomes harder when the work changes frequently.
Every changeover has a cost. If adapting the robot requires substantial engineering effort, that expense must be justified by the work that follows. The shorter the production run, the fewer units are available to absorb the changeover cost.
This is why we see in-context learning as more than an improvement in model performance. It is a fundumental paradigm shift that significantly reduces the effort required to change a robot’s job.
Imagine an operator demonstrating a new packing arrangement rather than waiting for an engineer to revise the robot’s behavior. Or a familiar handling operation being adapted to a new product variant through an example rather than a fresh data-collection campaign. These are the kinds of workflows we want to enable.
The potential benefit extends beyond faster setup. Lower adaptation costs could make shorter runs worth automating and allow a robot to remain useful as the production mix evolves.
This addresses an important step of industrial automation: communicating the desired behavior to the robot. Programming complexity and integration effort are already recognized barriers to automation in high-mix manufacturing. In-context learning offers a new way to reduce this burden.
How the field is approaching it
Robot in-context learning is not a single method. Different systems use different kinds of demonstrations and different mechanisms for turning those demonstrations into actions.
One approach uses demonstrations recorded in a robot-compatible gloves/UMIs. These contain observations, motion information, and actions in a form closely aligned with the robot’s own inputs and outputs. Behavior Prompting Policy, for example, conditions its policy on a demonstration and the current observation. Its iPhUMI interface lets a person collect demonstrations using a handheld manipulation device, producing a prompt with richer sensor and action information than an ordinary video. (Behavior Prompting Policy)
Another approach retrieves relevant demonstration segments during execution. ReCAP, introduced in Retrieve, Don’t Retrain, keeps its policy frozen and supports new tasks by adding demonstrations to a retrieval pool. At each control step, the model uses retrieved examples to guide its actions. In its real-robot experiments, human demonstrations include VR-based wrist-pose tracking, rather than relying solely on uninstrumented video. (ReCAP paper)
Test-time training. WAM-TTT takes raw human videos and updates a lightweight memory module inside a pretrained world-action model. The memory is adapted through self-supervised video prediction and then guides robot execution. An earlier training stage uses paired human and robot demonstrations to make this memory useful for control; at deployment, only unlabeled human videos are needed. Crucially, the base model stays frozen, but the memory’s parameters are updated at test time. This differs from simply conditioning a fixed-weight policy on a demonstration. (WAM-TTT paper)
Demonstration-conditioned behavior as emergent property. Generalist’s GEN-1.5 uses sensorimotor demonstrations for in-context prompting and even reports cases where a person demonstrates directly with their own hands in view of the robot’s cameras. (Generalist GEN-1.5)
These approaches differ technically, but they point toward a compelling interaction: demonstrate the desired behavior and let the robot use that example to guide execution.
The practical details matter. What must the operator wear or hold? What information must be recorded? How much preparation is required? Does adaptation involve model updates? And what kinds of changes can the robot handle reliably?
Those questions are central to how we are building Shiraz AI.
Shiraz AI: learning directly from a human demonstration
At Shiraz AI, we have developed our own approach to in-context learning directly from human demonstrations. For supported task variations, there is no need for specialized gloves, a lengthy on-site data-collection campaign, or task-specific retraining.
Our teaching workflow starts with an operator showing the task, rather than translating it into a robot-specific demonstration interface. A person records the operation on video, and our model uses that demonstration as context for the robot’s behavior.
The distinction matters because the teaching interface is part of the product.
For a factory operator, the relevant question is not how sophisticated the model architecture is. It is whether their existing knowledge of the job can be communicated to the robot with minimal additional effort.
We want the person who knows the process to be able to show it.
There is also an important boundary to what we mean by adaptation. Our focus is on new variations within families of tasks the model has already learned, rather than claiming that a robot can acquire any entirely new physical skill from one example.
For high-mix manufacturing, that is a meaningful capability. The economic opportunity does not require a robot to learn every conceivable task. A robot that can handle a useful family of operations—and adapt efficiently as requirements change—can address a substantial production need.
The important connection in each example is between what the person demonstrates and what the robot does. The demonstration is not merely footage illustrating the task. It is the input that tells the robot how the task should be performed.
That is the interaction we want to bring to the factory floor.
From a teaching interaction to a working production system
A live demonstration shows a capability. A production deployment has to deliver sustained value.
For manufacturers, the bar includes throughput, consistency, robustness to variation, and the amount of human intervention required when something goes wrong. Faster teaching is valuable only when it is paired with dependable execution.
That is why Shiraz AI delivers the complete system: robot hardware, the model, and on-site deployment—not just a model that customers must integrate themselves.
We are not removing engineering from robotics. It is to stop making every supported task variation a new engineering project.
We believe in-context learning can create a more useful relationship between robots and the people who operate them: the robot arrives with capabilities, the operator provides the context, and the system adapts to the work.
For high-mix manufacturing, that could change the question from “Is this job repetitive enough to automate?” to “Can we show the robot how this version of the job should be done?”
That is what we are building at Shiraz AI. To learn more or discuss a deployment, reach us at founders@shiraz.ai.