Track 3 · CoRL 2026 Challenge Suite

RoboChess Challenge

Build a robot that plays chess. Isaac Lab and Newton physics provide two ways to develop in simulation; train with sim and real data, then test on a physical robot.

See the requirements ↓ Explore the codebase ↗
Simulation rendering of a robot arm positioned over chess pieces
RoboChess simulation scene from the challenge project

Start here

Challenge requirements

A complete RoboChess entry covers all four parts of the workflow.

1

Robot plays chess

Perceive the board, choose a legal move, and move chess pieces with a robot arm.

2

Train using sim and real data

Use both simulated and real robot data to train your policy.

3

Test with sim and a real robot

Evaluate the policy in simulation and on physical robot hardware.

Two simulation implementations

Build with Isaac Lab and Newton physics

The project shares chess layouts and assets across both implementations. Each supports a different part of the development workflow.

Isaac Lab · scene and data pipeline

Set up tasks and record demonstrations

Manager-based tasks cover chess scenes from a 1D board to a standard 8 × 8 opening. A scripted pick policy uses GraspGen grasps and Isaac Lab's RecorderManager to save demonstration data.

  • Visual scenes for SO-101, Piper, UR10, Flexiv Rizon, Rebot, and YAM
  • Chess-piece picking examples for Franka, Piper, Rebot, and YAM
  • HDF5 recording and replay for simulated demonstrations
Explore Isaac Lab implementation ↗

Newton physics · parallel evaluation

Inspect scenes and test pick-and-place

The Newton port reuses the Isaac Lab chess layouts and grasp assets. It provides scene inspection and a scripted pick task with explicit success and failure outcomes.

  • Franka, Piper, Rebot, and YAM robot options
  • 1D, 3 × 3, 4 × 4, 8 × 8, and mixed-piece layouts
  • Replicated worlds for batched simulation runs
Explore Newton implementation ↗

The challenge

From a chessboard to a real robot

RoboChess uses chess to study robot policy learning across simulation and physical deployment. A successful system must connect board perception and move selection with accurate piece manipulation.

The task tests whether a policy trained with simulated and real data can handle the precision of chess moves and transfer from virtual scenes to a real robot.

How to participate

Build, train, test, submit

Use the RoboChess project as a starting point, then document your approach and results for both evaluation settings.

01 · Build

Set up the chess task

The challenge package ↗ provides chess assets and simulation task examples. Its README and setup guide describe the current environment.

02 · Train and test

Use both data sources

Train with simulation and real robot data. Test your resulting policy in simulation and on a real robot, recording successes and failures.

In the codebase

Chess tasks at several scales

The project includes compact boards for manipulation development and a standard 8 × 8 setup. The scenes below come from its simulation assets.

Simulation view of a robot and a three by three chess board
3 × 3 board Small board for early pick-and-place experiments
Simulation view of a robot and a four by four chess board
4 × 4 board More pieces and more complex moves
Simulation view of a robot and a standard eight by eight chess board
8 × 8 board Standard full chess setup

Project screenshots

See the simulated chess tasks

Close views from the project's Isaac Lab scenes and demonstration replays. Select a view to open its full screenshot or six-frame sequence.

Franka robot arm gripping a chess piece beside a four by four board
Franka chess pick sequence Close view from a recorded Isaac Lab demonstration on a 4 × 4 board · Open all six frames
Franka robot arm over the pieces of a full chess board
Full-board manipulation Close view from an Isaac Lab demonstration on an 8 × 8 board · Open all six frames
Yellow SO-101 arm beside a row of chess pieces in Isaac Lab
SO-101 1D chess scene
White Piper arm beside a row of chess pieces in Isaac Lab
Piper 1D chess scene
Rebot arm beside a row of chess pieces in Isaac Lab
Rebot 1D chess scene

Newton uses the same board layouts; the screenshots above are from Isaac Lab, not Newton renders.

Resources

Start with the project

The repository documents an Isaac Sim manipulation extension, Isaac Lab task examples, and a Newton simulation port.

This challenge folder is a snapshot of an actively developed project. Check the upstream README for later setup and task updates.

Your next move

Ready to submit RoboChess?

Bring your chess-playing robot, training with sim and real data, and test results from simulation and a physical robot to the workshop submission page.

Go to OpenReview ↗