nuReasoning¶

nuReasoning is a reasoning-centric autonomous driving dataset and benchmark focused on long-tail scenarios. It contains 20,000 clips of 20 seconds each (~105 hours), collected from multiple cities, combining synchronized multi-view camera images, LiDAR point clouds, ego state, HD maps, and traffic-signal context with human-verified reasoning annotations. The reasoning annotations span three categories — spatial reasoning, driving decisions, and counterfactual reasoning — and support both a Reasoning VQA benchmark and a planning benchmark.

Note

py123d currently exposes only the mini subset of nuReasoning (parts 1–3 of the HuggingFace train split).

Available Modalities¶

Name	Available	Description
Ego Vehicle	✓	State of the ego vehicle, including poses, dynamic state, and vehicle parameters, see `EgoStateSE3`.
Map	(✓)	The HD-Maps are in 2D vector format and stored per-log (one map per clip). For more information, see `MapAPI`.
Bounding Boxes	✓	The bounding boxes are available with the `NureasoningBoxDetectionLabel`. For more information, see `BoxDetectionsSE3`.
Traffic Lights	✓	Traffic-signal states are provided per frame.
Cameras	✓	nuReasoning includes 8x `Camera`: `PCAM_F0`: front `PCAM_B0`: back `PCAM_L0`: front_left `PCAM_L1`: left `PCAM_L2`: back_left `PCAM_R0`: front_right `PCAM_R1`: right `PCAM_R2`: back_right
Lidars	✓	A single merged `Lidar` point cloud fusing five sensors (top, front, side-left, side-right, back). The source sensor of each point is encoded in a `lidar_info` channel, and the cloud additionally carries `ring`, `intensity`, `azimuth`, `range` and return information. Point clouds are stored as LZF-compressed PCD files.
Reasoning	✓	Human-verified spatial, decision, and counterfactual reasoning annotations are available per frame. They are currently passed through as the raw nuReasoning reasoning JSON (no dedicated py123d datatype yet).

Download¶

nuReasoning is hosted as a public, Apache-2.0 licensed dataset on HuggingFace at qixuewei/nuReasoning. py123d ships an automated downloader that fetches and extracts the per-clip archives for you.

# Download the configured selection into $NUREASONING_DATA_ROOT
py123d-download dataset=nureasoning

The downloader exposes several selection knobs (see py123d/script/config/download/dataset/nureasoning.yaml):

splits — e.g. [train], [train, validation] (null discovers all live from the repo)
parts — e.g. [part_1, part_2]
log_names — explicit clip names <log>_<token> (mutually exclusive with num_logs)
num_logs — the first N clips of the selection (or N random with sample_random=true and seed)
max_workers — parallel clip download/extract workers (default 8)
keep_zip — keep each .zip next to its extracted directory (default: extract then discard)

Each selected clip is downloaded as a single .zip and extracted to <output_dir>/<split>/<part>/<clip>/. The 123D conversion expects the following directory structure:

$NUREASONING_DATA_ROOT
  └── train/
      ├── part_1/
      │   ├── <log_name>_<keyframe_token>/
      │   │   ├── metadata.json
      │   │   ├── map.pkl
      │   │   ├── ego_state/
      │   │   │   └── <timestamp_us>.pkl
      │   │   ├── annotations/
      │   │   │   └── <timestamp_us>.pkl
      │   │   ├── reasoning/
      │   │   │   └── <timestamp_us>.json
      │   │   ├── cameras/
      │   │   │   ├── front.jpg
      │   │   │   ├── back.jpg
      │   │   │   └── ...
      │   │   └── lidar/
      │   │       └── <timestamp_us>.pcd
      │   └── ...
      ├── part_2/
      └── part_3/

Lastly, you need to add the following environment variable to your ~/.bashrc according to your installation path:

export NUREASONING_DATA_ROOT=/path/to/nureasoning/data/root

Or configure the config py123d/script/config/common/default_dataset_paths.yaml accordingly.

Installation¶

Downloading and streaming nuReasoning requires the HuggingFace Hub client, included as an optional dependency in py123d. You can install it via:

PyPI

pip install py123d[hf]

Source

pip install -e .[hf]

Conversion¶

Local mode — data already extracted to $NUREASONING_DATA_ROOT (see the Download section above):

py123d-conversion dataset=nureasoning-mini

Note

The local conversion of nuReasoning by default does not store sensor data in the logs, but only relative file paths (camera_store_option: "path" and lidar_store_option: "path"), which are resolved against the nuReasoning sensor root at read time. To change this behavior, adapt the nureasoning-mini.yaml converter configuration.

Streaming mode — materialize the selected clips from the HuggingFace repo into a session-scoped temp directory at parser construction time, convert from it, and delete the temp directory afterwards. The mini subset corresponds to parts 1–3 of the HuggingFace train split.

py123d-conversion dataset=nureasoning-mini-stream

The repo is public, so no token is required; if needed, hf_token falls back to $HF_TOKEN / $HUGGINGFACE_HUB_TOKEN.

Note

Streaming mode forces camera_store_option: "jpeg_binary" and lidar_store_option: "binary" (with the laz codec) — the temp directory is deleted immediately after conversion, so any "path" references would point at vanished sources.

Citation¶

If you use nuReasoning in your research, please cite:

@misc{huang2026nureasoning,
  title={nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving},
  author={Huang, Zhiyu and Liu, Johnson and Song, Rui and others},
  year={2026},
  eprint={2605.31572},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}


Papers	nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving Project page
Download	qixuewei/nuReasoning on HuggingFace
Code	A public devkit is announced as Coming Soon on the project page.
License	Apache License 2.0
Available splits	`nureasoning-mini_train`