Uranus: Building the Next-Generation Simulation Infrastructure for Embodied AI

Background: Why a learned simulator

“What I cannot create, I do not understand.” — Richard Feynman

To act intelligently, a robot must first imagine the consequences of its actions — yet real interaction is costly, slow, and impossible to reproduce, while traditional simulators (Isaac Sim, MuJoCo) demand hand-authored assets and physics and still suffer a sim-to-real gap. Uranus closes this gap with a data-driven world model: feed it observations, camera calibrations, joint states, and a robot description file, and it autoregressively generates multi-view video conditioned on actions — simulation learned from data, not hand-built.

Technologies

Uranus is the product of four lines of engineering — algorithm, data, training, and infrastructure. This section explains what is special about each. More details are available in the tech report.

Algorithm

Built as an action-conditioned autoregressive diffusion world model, Uranus supports interactive and open-ended robot simulation through a two-level generation process: an outer causal rollout loop predicts the next latent frame, while an inner flow-matching process produces high-fidelity visual dynamics. Each latent prediction is decoded into four consecutive RGB frames per camera and fed back as historical context. Alternating spatial and temporal attention enables efficient cross-view fusion and motion modeling without full attention over all camera-frame tokens. Embodiment-agnostic skeleton maps encode robot actions through online forward kinematics, while calibration-derived Plücker ray embeddings represent fixed or moving camera viewpoints. Together, these components provide a unified interface across robot embodiments, control sources, camera configurations, and language-conditioned tasks.

Data

More than 3,500 hours of curated real-world robot interactions form the training corpus, spanning single-arm, dual-arm, and humanoid platforms across diverse tasks, scenes, and camera setups. A unified pipeline converts heterogeneous raw, RLDS, LeRobot, and HDF5 datasets into standardized MP4 streams and Lance trajectory records. It validates video integrity and robot-state completeness, synchronizes frames with state trajectories, and recovers missing camera parameters through MoGe-3-based intrinsic estimation, multi-view feature matching, and bundle adjustment. Calibration and synchronization are further verified by comparing kinematic robot projections with SAM 3 segmentation masks, supplemented by VLM-based audits for visual corruption, occlusion, and temporal discontinuities.

Training

Direct causal pretraining aligns the model’s training behavior with autoregressive inference from the outset. Teacher-forcing causal masks expose each noisy target only to reference observations and valid historical frames, preventing information leakage from clean targets or future observations. Frame-relative 3D RoPE supports temporal extrapolation and sliding-window cache eviction, while persistent reference attention sinks preserve scene identity, camera layout, and robot context over long rollouts. A self-forcing post-training stage combines Distribution Matching Distillation with Phased Consistency Model regularization, reducing inference to four denoising steps while mitigating accumulated autoregressive errors. LoRA-based parameter-efficient fine-tuning additionally enables cost-effective adaptation to new robots, scenes, tasks, and camera configurations.

Infrastructure

An end-to-end systems stack supports large-scale data processing, distributed training, real-time inference, and stateful serving. Ray and Daft orchestrate fault-tolerant multimodal data curation, while Lance provides columnar storage, schema evolution, versioning, and reproducible dataset snapshots. Object storage is accelerated through NVMe caching, asynchronous byte-range reads, read-ahead prefetching, multi-process decoding, and a Rust-based I/O path, reducing average access latency from 61 ms to 11 ms. At training time, HSDP, Ulysses sequence parallelism, gradient checkpointing, and distributed tiled VAE processing scale the 1.3B-parameter model across 64 GPUs. At inference time, sliding-window KV caching, streaming VAE decoding, FP8 execution, CUDA Graphs, optimized RoPE, Flash-VAED, cache-affinity routing, and disaggregated DiT–VAE serving enable bounded-memory, low-latency rollout at up to 24 FPS.

Features

Action Controllable

Uranus accepts action commands as input and predicts what happens next: given the current state and an action, it forecasts the resulting multi-view observations — motion, object interaction, and environmental response — so that any candidate policy can be tried and evaluated in the learned world before it ever touches hardware.

Streaming, Autoregressive Generation

Unlike mainstream bidirectional models, Uranus adopts an LLM-style causal architecture designed for autoregressive generation, where each step leverages the full preceding context to predict subsequent observations — enabling closed-loop interaction with both humans and policies.

Multi-view Extension

Uranus’s architecture natively adapts to samples and scenes with varying numbers of camera views, and can even synthesize observations under novel view configurations specified only by camera calibration.

Capabilities

Evaluation

WorldOlympiad Long-Horizon Benchmark

On the robotics setting of WorldOlympiad — a real-world benchmark measuring physical rule adherence, geometric and camera consistency, semantic alignment, and interaction fidelity — Uranus achieves the strong overall (0.722) and interaction (0.730) scores:

We treat this as a reference evaluation rather than a definitive ranking: some baselines target general video-generation settings, whereas Uranus is explicitly designed for action-conditioned embodied interaction.

Closed-Loop Policy Evaluation

Uranus works as a reliable proxy for policy evaluation. Unlike open-loop scoring from a fixed action sequence, closed-loop evaluation lets simulation errors propagate into later decisions and final outcomes — a much stricter test. Nine policies run in closed loop on five RoboTwin tasks, 15 episodes each: the policy observes frames generated by Uranus, predicts an action, and the rollout advances. For domain adaptation, Uranus is fine-tuned on the clean subsets of the official RoboTwin tasks, augmented with 7,891 failed trajectories collected from policy rollouts. The resulting policy rankings match the RoboTwin clean environment almost perfectly (Pearson 0.98, MMRV 0.01), making Uranus an always-available testbed for evaluating robot policies.

What Remains Hard

An honest simulator report states its limits — and so do we. Our demo videos are not cherry-picked to showcase only the best-case scenarios, but rather to reflect the model’s current capabilities as objectively as possible. The demos and evaluations above show both what works and what still breaks. Precise contact dynamics and object-state transitions are the largest open gap — plausible motion can coexist with an incorrect grasp. Errors still accumulate over very long horizons despite causal training and persistent anchors. Generalization weakens under large shifts in tasks, embodiments, and viewpoints, keeping reliability tied to training-data coverage.