RoboAPI
Zero-Shot Generalization, Now Within Reach
October 30th, 2026
Today, we are introducing RoboAPI: a publicly accessible, ready-to-use API for robots in the physical world. Taking robots’ observations and state as input, RoboAPI follows user instructions and predicts the next action chunk in near real time.
Physical AI has advanced at an unprecedented pace over the past few years. As embodied models become more capable and robots learn increasingly complex tasks, generalization has become one of the most important challenges. The most ambitious goal is zero-shot generalization: enabling robots to follow user instructions without additional data collection or model training.
The development of language models offers a useful parallel. GPT-3 first demonstrated that language models could evolve from task-specific NLP tools into general-purpose foundation models capable of handling a wide range of tasks. The GPT-3 API that followed made frontier foundation models available out of the box: for the first time, developers could access them through API calls without training or deploying a language model themselves.
Bringing this approach to the physical world is far more complex. Unlike standardized digital formats such as text, images, and video, robots differ substantially in their mechanical structures, sensors, and control methods. As a result, even when an embodied foundation model is publicly available, a long and difficult cycle of data collection, model training, evaluation, and further data collection — often repeated several times — is essential before it can actually empower the robot to perform tasks.
The core objective of RoboAPI is to make zero-shot generalization accessible to everyone. We believe that genuine generalization must be tested broadly across robot configurations, tasks, and environments. This is closely tied to our long-term goal: bringing general-purpose robots into homes. Every home has its own layout, objects, and routines. Adapting to such complex, diverse, and constantly changing environments requires both stronger models and continuous testing of their capabilities and limits in more real-world settings. Launching RoboAPI is our first step toward that goal. We hope developers can go beyond watching demo clips and use the model on their own robots, putting it to the test in real environments.
During the technical preview, RoboAPI will be available free of charge to everyone*. If you are interested, follow the links below to get started.
Build with RoboAPI
Internal Testing
We deployed robots in dozens of different settings across our own facilities and invited colleagues to give them any instruction they chose based on the environment around them. In general, RoboAPI performed the tasks well as long as clear instructions were given.
External Testing
We also invited our friends around the world to join our private beta. Using robots and everyday objects they already had, they connected to RoboAPI in their own labs, tested its capabilities on tasks of their choosing, and sent us videos of their execution. If you have a robot arm, we hope you will be able to enjoy the same straightforward, out-of-the-box experience.
Powered by Poke-Aurora
We trained Poke-Aurora, our first world-action model, to make zero-shot generalization readily accessible.
Pretraining: Joint Prediction of World Dynamics and Robot Actions for Generalizable Action Reasoning
By predicting future visual observations, world-action models explicitly learn patterns shared across object motion and interactions, providing a basis for transferring manipulation experience across objects and scenes. In practice, however, we found that a world-action model’s ability to model the world often does not naturally translate into reliable action generation. Making video learning and action learning reinforce each other remains a key challenge.
To address this bottleneck, we introduce Modulated Residual Coupling (MRC) in Poke-Aurora. MRC establishes learnable interactions between modules while preserving their original computation path. We observed that MRC improves gradient alignment between video prediction and action prediction, bringing their learning directions into closer agreement within their shared representations. This allows the model’s understanding of world modeling to contribute more effectively to learning robot actions. We will share more details about MRC in the Poke-Aurora technical report. Stay tuned!
Post-training: Alignment with a UMI-Style Configuration for Seamless Generalization Across Robot Setups
In robotic manipulation, a robot’s “configuration” encompasses the combined constraints of its arm structure, control method, and sensor layout. Joint arrangements vary across robot arms, while the number and placement of visual sensors can differ even more widely. Together, these differences leave embodied models trained on different configurations highly incompatible with each other.
To address this challenge, we took inspiration from UMI and designed a post-training configuration that combines a wrist-mounted fisheye camera with end-effector pose control. See the hardware setup for details. With an end-effector-centered design, this UMI-style configuration is particularly well suited to generalization across embodiments. Using consumer action cameras also makes the setup easier and less expensive to build.
After aligning Poke-Aurora with this configuration through post-training, we observed two results:
The model performed well on configurations used in training. Switching between fisheye camera brands (e.g., from an Insta360 Ace Pro2 to a DJI Action 6) had almost no effect on performance.
Encouragingly, the model also transferred successfully to entirely unfamiliar robot configurations, including substantially different ones (e.g., moving from a UR7e to a YAM Standard).
These results indicate that mounting an action camera on the wrist of a robot arm is enough for RoboAPI to control it for task execution, provided the arm has sufficiently capable inverse kinematics (IK) solving.
Credible Evaluation at Scale
Unlike language models, physical AI still lacks a widely accepted evaluation benchmark. One well-known reason is that models often use highly customized robot configurations and task definitions, making their results difficult to compare directly. A deeper issue is that current embodied models generally have limited generalization capabilities. Evaluating them often requires additional data collection and fine-tuning for specific tasks. This substantially increases evaluation costs and makes results harder to reproduce, limiting the ability of existing evaluations to reflect a model’s general-purpose capabilities.
As training progressed, we began to see Poke-Aurora generalize along several dimensions: environments and distractors, object categories and instances, instruction following, and coherent execution over long horizons. These improvements were noticeable but difficult to quantify. We realized that evaluations based on task-specific fine-tuning were becoming less able to capture Poke-Aurora’s true generalization potential. After a period of exploration, we developed a large-scale evaluation framework covering nearly 1,000 distinct instances of common, manipulable objects, organized into nearly 10,000 unique instructions. The instructions span three broad categories of fundamental manipulation tasks: moving and organizing objects, aligning poses and shapes, and interacting with the environment and using tools.
While Poke-Aurora leads across the benchmark we built, we are well aware of the complexity of the real world and cannot guarantee that your experience with RoboAPI will exactly match the statistical results. We believe this is precisely why RoboAPI matters: the generalization capability of a frontier embodied model should not be judged solely by its creators. It should be open to real-world generalization tests from any external user, at any time.
Performance by manipulation goal
- Poke-Aurora
- Astra
- π0.5
- MolmoAct2
Transfer & Organization
| Poke-Aurora | Astra | π0.5 | MolmoAct2 |
|---|---|---|---|
| 86.0% | 71.3% | 30.9% | 26.9% |
Pose & Shape Alignment
| Poke-Aurora | Astra | π0.5 | MolmoAct2 |
|---|---|---|---|
| 54.1% | 36.3% | 14.8% | 7.9% |
Environment Interaction
| Poke-Aurora | Astra | π0.5 | MolmoAct2 |
|---|---|---|---|
| 87.7% | 67.9% | 32.9% | 23.6% |
Useful Tips
Watch for occlusion!
A single-camera setup does not provide a global view of the scene. In some situations, the object being held may block much of the camera’s view, reducing model performance.
Give it time!
RoboAPI retains historical context and will keep retrying after a failed attempt. Giving it a little more time may allow it to succeed.
Try different instructions!
Instructions in English generally yield better performance.
Some instructions may confuse RoboAPI. If an attempt does not succeed, try rephrasing the request.
Breaking long-horizon tasks into shorter, atomic tasks and using more precise adjectives to describe the target objects can improve task success rates.
Always safety first!
RoboAPI executes actions quickly, but its speed remains within the safe operating range of the vast majority of robot arms. The operating speed itself need not be a cause for concern.
We recommend setting up safety boundaries around the robot arm’s workspace in advance, or using an arm with compliant control.
RoboAPI’s safety gate rejects instructions that could potentially harm people or the surrounding environment.
RoboAPI can make mistakes. If it causes a robot arm to behave in any way you consider dangerous, please report what happened to safety@pokebot.tech. Thank you for your help!
*RoboAPI is currently a non-commercial offering and does not provide products or services to for-profit third parties.