6 Nov 2025
Traditional impressive robot demonstrations occur in controlled environments and struggle with complex, messy, and novel real-world tasks, highlighting a significant sim-to-real gap. A new AI, NeRD (neural robot dynamics), enables robots to learn physics in simulation and generalize effectively to reality, even outperforming traditional physics simulators.
Many impressive robot acrobatics, parkour, flips, and dance routines are demonstrated in controlled environments where every step is known in advance, making these 'easy' problems. The truly hard challenges involve handling and grasping small, deformable, or new objects on new surfaces under unseen lighting, requiring robots to adapt to new or changing situations.
Scientists typically train a robot in a simulation before deploying it into the real world, much like learning inside a video game. However, things that work really well in simulation often break like crazy when moved to reality, indicating a significant 'sim-to-real' gap.
An incredible new work introduces NeRD, a neural robot dynamics system, setting out to solve two tough problems: performing predictions over thousands of simulation steps and generalizing across different tasks, environments, and robot morphologies.
Reference physics simulators, based on hand-written equations, are super precise and trusted but are painfully slow and brittle. These require retuning large parts of the setup by hand if the robot's shape or environment changes.
Scientists wrote an AI, NeRD, that looks at lots of footage from physics simulators, studying how the world works frame by frame. This AI learns enough examples of physics behavior to skip equations and predict what happens next, functioning as a neural physics solver.
NeRD absolutely nails classic tasks like cartpole balancing and accurately predicts pendulum behavior, closely resembling the real simulation. A robot learned to walk inside NeRD's imagination and then behaves very similarly when dropped into the game, with no retraining or fine-tuning necessary.
A controller trained inside NeRD's imagination can move a robot effectively, demonstrating its capability with complex robot arms. When put into reality to touch specific red points, the NeRD robot not only works but makes it look easy, a task usually extremely challenging in real-world labs.
NeRD learns to predict the next change in the robot's state by applying motion in the robot’s own coordinate frame and then transforming it back to world coordinates, akin to learning to move through a dark room. When fine-tuned on real-world cube tossing data, NeRD matched it better and was faster than the physics simulator (Warp) that created the data, meaning the 'student beat the teacher' by being 'street smart' rather than just 'book smart'.
NeRD is not yet perfect and still hasn't been tested on very complex robots like humanoids.
NeRD matched it better than the physics simulator called Warp that created it in the first place, with the student beating the teacher and being faster too.
| aspect | traditional_robot_demos | traditional_physics_simulators | nerd_approach |
|---|---|---|---|
| Problem Type Addressed | Easy (acrobatics, known sequences, controlled environments) | Attempts hard problems, but struggles with real-world messiness | Solves hard, mundane, messy problems (generalization across tasks/environments) |
| Adaptability & Generalization | Low (often hand-coded, no adaptation to new situations) | Brittle (requires manual retuning for changes in robot/environment) | High (generalizes across tasks, environments, and robot morphologies) |
| Speed of Prediction | N/A (focus on execution, not prediction) | Painfully slow | Faster than traditional physics simulators |
| Fidelity/Accuracy | Visually impressive, but not representative of real challenges | Super precise in idealized conditions | High, closely matches and can outperform traditional simulators on real-world data |
| Sim-to-Real Gap | Minimizes by controlled, known setups | Major issue; things 'break like crazy' in reality | Effectively bridges the gap, allowing smooth real-world deployment |
| Underlying Mechanism | Often hand-coded or specific control algorithms | Hand-written physics equations | Neural physics solver learns dynamics from data, predicts state changes (robot's coordinate frame) |
