Adding Datasets¶

This tutorial walks through adding a new source dataset to the 123D conversion pipeline. A dataset is integrated by implementing a set of parser classes that read the raw data and yield 123D’s unified datatypes. The framework then handles configurable synchronization, log writing, and exposing the result through the unified API.

1. General¶

Architecture overview¶

Conversion is organized around three base classes defined in BaseDatasetParser. A top-level dataset parser produces lightweight, per-log and per-map handles that . The orchestrator constructs these once on the main process, then distributes them to parallel workers, which perform the heavy I/O internally and hand the parsed data to the log/map writers:

BaseDatasetParser
  ├── get_log_parsers()  ──▶  [ LogParser_1, ..., LogParser_N ]  ──▶  ArrowLogWriter
  └── get_map_parsers()  ──▶  [ MapParser_1, ..., MapParser_M ]  ──▶  ArrowMapWriter
                                                                          │
                                                                          ▼
                                                                Arrow storage ──▶ unified API

Each new dataset adds a package under src/py123d/parser/<dataset>/. Existing packages (i.e., nuscenes/, nuplan/, wod/, av2/, kitti360/, pandaset/) are implemented as references.

The dataset parser¶

Subclass BaseDatasetParser and implement two methods:

from py123d.parser.base_dataset_parser import BaseDatasetParser

class MyDatasetParser(BaseDatasetParser):

    def get_log_parsers(self) -> list[BaseLogParser]:
        """One lightweight, picklable handle per log/scene."""
        ...

    def get_map_parsers(self) -> list[BaseMapParser]:
        """One handle per map region (return [] if the dataset has no maps)."""
        ...

Important

The returned handles are shipped to worker processes. Keep them lightweight, i.e., store only paths and parameters, not open file handles, database connections, or decoded sensor data. All expensive I/O belongs inside the iterators (covered below), which run on the worker.

Optional and gated dependencies¶

Dataset-specific third-party packages (devkits, format readers, cloud SDKs) should not be hard dependencies of 123D. Declare them as an extra in pyproject.toml under [project.optional-dependencies]:

[project.optional-dependencies]
mydataset = ["mydataset-devkit==1.2.3"]

Then gate the import at the top of your parser (or lazily inside the method that needs it) with check_dependencies(), which raises a helpful, install-instruction error if the package is missing:

from py123d.common.utils.dependencies import check_dependencies

check_dependencies(["mydataset"], optional_name="mydataset")
# ImportError: Missing 'mydataset'. Install with: pip install py123d[mydataset]

Entry point¶

Conversion is driven by the py123d-conversion CLI (defined in [project.scripts], mapping to py123d.script.run_conversion.main()). It uses Hydra to instantiate your parser from a YAML config via its _target_ field (see section 5):

py123d-conversion dataset=mydataset

2. Log Parser¶

A log parser is a handle to a single continuous driving sequence (i.e., called ‘log’ in 123D). Subclass BaseLogParser and implement the log metadata getter and the synchronized-frame iterator.

Log metadata¶

get_log_metadata() returns a LogMetadata describing the log:

Field	Description
`dataset`	Dataset name, lowercase and dashed (e.g. `"my-dataset"`).
`split`	Data split, prefixed by dataset (e.g. `"my-dataset_train"`).
`log_name`	Unique name of the log.
`location`	Geographic location (optional); links a log to a per-location map.
`map_metadata`	Associated `MapMetadata`, if any.
`version`	123D version that produced the metadata (defaults to the current version).

Iterating modalities¶

iter_modalities_sync() lazily yields one ModalitiesSync per synchronized frame. Each frame bundles a timestamp with a list of 123D modalities. The unified BaseModality types parsed from the dataset’s native formats (ego state, bounding boxes, cameras, lidar, radar, …):

from py123d.parser.base_dataset_parser import ModalitiesSync

def iter_modalities_sync(self):
    for frame in self._read_frames():
        modalities = [
            self._parse_ego_state(frame),
            self._parse_boxes(frame),
            *self._parse_cameras(frame),
            self._parse_lidar(frame),
        ]
        yield ModalitiesSync(timestamp=frame.timestamp, modalities=modalities)

Parsed sensor datatypes and lazy I/O¶

Sensor data is the bulk of any dataset, so 123D may optionally skip decoding it during parsing. Instead, you can pass a parsed modality helpers, which carry only a file path (or raw bytes) plus metadata. Depending on the storage option for sensor data, the writer can just store the sensor reference, or decode and re-compress the data internally for self-contained storage.

ParsedCamera: camera metadata, a camera_to_global_se3 pose, and either a dataset_root/relative_path pair or an in-memory byte_string.
ParsedLidar: lidar metadata, start_timestamp/end_timestamp (the sweep window), a dataset_root/relative_path, and optional load_kwargs.
ParsedRadar: radar metadata, a single timestamp (a radar scan is an instantaneous snapshot), a file path, and optional load_kwargs.

If sensor storage is configured as “path”, the whole conversion process is substantially faster, as the conversion is not memory- or I/O-bound. On the other hand, the converted logs are not self-contained and require access to the original dataset files for loading.

Tip

load_kwargs is a generic, dataset-specific dictionary forwarded verbatim to the point-cloud loader and persisted in the Arrow frame under the "path" store option. nuScenes uses it to carry the keyframe’s panoptic-label path alongside the lidar file.

Where to add sensor I/O¶

If your dataset uses a point-cloud or image format not already supported, add a loader under src/py123d/common/io/{camera,lidar,radar}/ and add it in the dispatcher py123d.common.io.lidar.path_lidar_io.load_point_cloud_data_from_path() (i.e. for lidar). The dispatcher routes by file extension / store option and is where load_kwargs is consumed.

Sync vs. async parsing¶

For simplicity, the default configuration writes all modalities as synchronized frames. Dataset with asynchronous modalities can be implemented by overriding the async iterator iter_modalities_async(), which yields each modality independently as it is produced. The default implementation simply unwraps the synchronized frames, so overriding is optional. Async writing is selected by the async_conversion flag in the config (section 5). Importantly, an individual modality must be parsed in increasing timestamp order, but modalities can be interleaved in any way.

3. Map Parser¶

Maps are usually the trickiest part to unify since HD-map schemas vary widely across datasets. Maps are optional: if your dataset ships none, return [] from get_map_parsers() and skip this section. We generally recommend starting with log parsing and adding maps later.

A map parser is a handle to one map region. Subclass BaseMapParser and implement:

from py123d.parser.base_dataset_parser import BaseMapParser

class MyMapParser(BaseMapParser):

    def get_map_metadata(self) -> MapMetadata:
        ...

    def iter_map_objects(self):
        """Yield map objects lazily, one at a time."""
        yield from self._parse_lanes()
        yield from self._parse_crosswalks()
        # ...

Since map elements are static datatypes, they have no timestamp information and are expected to be expressed in the logs global coordinate frame.

Map metadata¶

get_map_metadata() returns a MapMetadata. Two fields deserve particular attention:

map_has_z: whether the map geometry carries elevation (3D) or is 2D only.
map_is_per_log: whether each log has its own map (i.e., as in Waymo datasets) or maps are shared per location across logs (e.g., for each city in nuScenes/nuPlan).

Supported map elements¶

iter_map_objects() yields BaseMapObject instances. Each object is one of the supported element types (the MapLayer enum), defined in src/py123d/datatypes/map_objects/map_objects.py:

Element	Geometry	Notes
`Lane`	surface	Individual driving lane with boundaries and centerline. Lane connectors are also yielded as `Lane` objects.
`LaneGroup`	surface	Group of lanes travelling in the same direction.
`Intersection`	surface	Intersection area.
`Crosswalk`	surface	Pedestrian crossing.
`Walkway`	surface	Pedestrian walkway.
`Carpark`	surface	Parking area.
`GenericDrivable`	surface	Other drivable surface.
`StopZone`	surface	Zone governed by a stop sign / light.
`RoadEdge`	line	Road boundary.
`RoadLine`	line	Painted lane marking.
`SpeedBump`	surface	Speed-reduction area.

Surface elements derive from BaseMapSurfaceObject (a polygon outline), and line elements derive from BaseMapLineObject (a polyline).

Note

If a map element type isn’t listed above, then new layers can be added. Please open an issue or PR (see Contributing) so the schema stays consistent across datasets.

4. Downloader (optional)¶

A downloader is optional, but it can significantly lower the onboarding barrier: A dataset or subset can be downloaded directly from the 123D CLI. The existing implementations in 123D also show examples how to download and convert in one script (i.e. using the streaming during conversion approach described below).

Subclass BaseDownloader, placing it at src/py123d/parser/<dataset>/<dataset>_download.py:

from py123d.parser.base_downloader import BaseDownloader

class MyDatasetDownloader(BaseDownloader):

    def download(self) -> None:
        """Fetch the selected data into ``self.output_dir``."""
        ...

The base class provides output_dir (destination; None means a temp directory is assigned at runtime) and dry_run (log the plan without writing).

Integration follows two paths:

Standalone CLI: py123d-download dataset=mydataset instantiates the downloader and calls download().
Streaming during conversion: the parser accepts an optional downloader= argument, redirects its output_dir to a session-scoped temp directory, downloads before reading, and cleans up on destruction.

5. Hydra config¶

Dataset Paths¶

123D centralizes the dataset roots that parsers read from in a single Hydra config group at src/py123d/script/config/common/default_dataset_paths.yaml, typed by the DatasetPaths dataclass in src/py123d/common/runtime/dataset_paths.py. For a new dataset, add a mydataset_data_root: ${oc.env:MYDATASET_DATA_ROOT,null} entry to the YAML.

By default, we use a environment variable (e.g. MYDATASET_DATA_ROOT) to avoid hardcoding local paths in the config. In your dataset config (see below), you can reference the path to pass it to the log/map parsers.

Dataset Parsers¶

Conversion configs live in src/py123d/script/config/conversion/dataset/, one <name>.yaml per dataset variant. The base defaults are in default_conversion.yaml. Each dataset config has four sections:

parser: instantiates your BaseDatasetParser (via _target_) and its data-root paths / options.
log_writer_config: storage options specific to the arrow log writer, i.e. configures sensor storage (camera_store_option, lidar_store_option, lidar_codec), the async_conversion flag, and inference toggles.
log_writer — the ArrowLogWriter, including a sync_config (the reference_column and direction used to snap modalities onto a common timeline).
map_writer — the ArrowMapWriter.

Provide a local config and, if you wrote a downloader, a separate stream config. Keep them as distinct files as they differ in data roots and storage:

Local (mydataset.yaml)

Data is pre-downloaded; roots point at it and sensors are stored as paths.

parser:
  _target_: py123d.parser.mydataset.mydataset_parser.MyDatasetParser
  _convert_: 'all'
  splits: ["mydataset_train", "mydataset_val"]
  mydataset_data_root: ${dataset_paths.mydataset_data_root}

log_writer_config:
  _target_: py123d.api.scene.arrow.utils.log_writer_config.LogWriterConfig
  _convert_: 'all'
  camera_store_option: "path"   # "path", "jpeg_binary", "png_binary"
  lidar_store_option: "path"    # "path", "binary"
  lidar_codec: null
  async_conversion: ${async_conversion}
  # ...

Stream (mydataset-stream.yaml)

Roots are null (auto-detected in a temp dir); a downloader block is added and sensors are stored as binary (temp files vanish after conversion).

parser:
  _target_: py123d.parser.mydataset.mydataset_parser.MyDatasetParser
  _convert_: 'all'
  splits: ["mydataset_train", "mydataset_val"]
  mydataset_data_root: null     # auto-detected
  downloader:
    _target_: py123d.parser.mydataset.mydataset_download.MyDatasetDownloader
    _convert_: 'all'
    output_dir: null            # temp dir assigned at runtime
    splits: ${dataset.parser.splits}
    dry_run: false

log_writer_config:
  # ...
  camera_store_option: "jpeg_binary"
  lidar_store_option: "binary"
  lidar_codec: "laz"

6. Documentation¶

Finally, document the dataset so users know how to obtain, install, and convert it:

Add a page docs/datasets/<name>.rst: You can mirror an existing one. The established structure is: an Overview dropdown (paper, download, license, splits), an Available Modalities table, Download, Installation (the pip extra), Conversion (local + streaming), any dataset-specific sections, Dataset Issues, and Citation.
Register the page in the docs/datasets/index.rst toctree.
Build the docs locally and confirm the page renders without warnings before opening your PR.

Checklist¶

Step	Location
`BaseDatasetParser` subclass	`src/py123d/parser/<dataset>/<dataset>_parser.py`
`BaseLogParser` subclass	same package
`BaseMapParser` subclass (if maps)	same package
Sensor I/O loader (if new format)	`src/py123d/common/io/{camera,lidar,radar}/`
Optional dependency extra + gating	`pyproject.toml` + `check_dependencies(...)`
Downloader (optional)	`src/py123d/parser/<dataset>/<dataset>_download.py`
Hydra config pair	`src/py123d/script/config/conversion/dataset/<name>{,-stream}.yaml`
Dataset doc page + index entry	`docs/datasets/<name>.rst` + `docs/datasets/index.rst`

Thank you for contributing a dataset to 123D! If anything in this guide is unclear or you hit a case it doesn’t cover, please open an issue or PR.