Depth Camera¶

A depth camera is a per-pixel metric depth stream, pixel-aligned to a sibling RGB Camera whose camera_id, projection model, intrinsics, and extrinsics it shares. It is stored as CAMERA_DEPTH in its own stream, camera_depth.<camera_id>.arrow, separate from the RGB camera.<camera_id>.arrow of the same camera_id.

The storage contract — how metric depth is quantized into the stored raster, and the max_depth, depth_bits, depth_transform, min_depth, has_invalid and depth_type knobs that control it — is documented on DepthCameraMetadata below. The examples here show it end to end.

Reading and writing¶

from py123d.datatypes import Camera, DepthCameraMetadata

# Writing: quantize metric depth into the stored raster.
depth_metadata = DepthCameraMetadata(camera_metadata=rgb_metadata, max_depth=96.0, depth_bits=16)
log_writer.write_async(
    Camera(
        metadata=depth_metadata,
        image=depth_metadata.encode_depth(depth_in_metres),
        camera_to_global_se3=camera_to_global_se3,
        timestamp=timestamp,
    )
)

# Reading: `image` is the raw quantized raster, not metres. `scale` optionally downsamples.
camera = scene.get_camera_depth_at_iteration(iteration, camera_id=CameraID.PCAM_F0, scale=2)
depth_in_metres = camera.metadata.decode_depth(camera.image)  # NaN where has_invalid marked no data

# `rgb_image` colorizes the raster with a TURBO ramp for display.
preview = camera.rgb_image

The default above (linear, no sentinel, z-depth) matches a simulator that renders a finite depth for every pixel. A sparse real sensor — e.g. lidar projected into the image — wants the inverse transform for near-field precision and the invalid sentinel for dropouts:

depth_metadata = DepthCameraMetadata(
    camera_metadata=rgb_metadata,
    max_depth=120.0,
    depth_bits=16,
    depth_transform="inverse",   # fine near, coarse far
    min_depth=0.5,               # required by the inverse transform
    has_invalid=True,            # code 0 -> NaN for pixels with no measurement
    depth_type="z_depth",        # or "ray_distance" for a native range sensor
)

The scale argument on the read accessors (get_camera_depth_at_iteration / get_camera_depth_at_timestamp) is an integer downscale denominator — 2 for half size, 4 for quarter — applied at decode time. It resamples with nearest-neighbour, never bilinear: averaging depth across an occlusion boundary would invent a surface floating between foreground and background.

Depth Camera Metadata¶

class py123d.datatypes.DepthCameraMetadata[source]¶

Metadata for a per-pixel depth camera stream.

A depth camera is pixel-aligned to a sibling RGB camera: it shares that camera’s id, projection model, intrinsics, and extrinsics, and composes its BaseCameraMetadata for all geometry. In addition, it records how the continuous metric depth was quantized into the stored integer raster.

Its channel_type is always DEPTH (modality CAMERA_DEPTH, file camera_depth.<camera_id>.arrow), a separate stream from the RGB camera.<camera_id>.arrow of the same camera_id.

Storage contract. Depth is continuous, so it is quantized to an unsigned integer raster (uint8 or uint16) and stored as a lossless PNG. Quantization clips to [min_depth, max_depth], maps onto the unit interval (linearly in depth, or in inverse depth — see depth_transform), and rounds onto the integer codes [offset, max_raw]:

t   = to_unit(clip(depth_m, min_depth, max_depth))   # depth -> [0, 1]  (transform-dependent)
raw = offset + round(t * (max_raw - offset))         # encode
# decode inverts to_unit on (raw - offset) / (max_raw - offset)

where max_raw = 2 ** depth_bits - 1 and offset is 1 when has_invalid reserves code 0 (else 0). Points to keep in mind:

max_depth clips the far range. Anything at or beyond max_depth saturates to max_raw and decodes back as exactly max_depth, so a sky pixel at 1000 m and a wall at max_depth are indistinguishable once encoded.
Invalid pixels. With has_invalid=False (the default) there is no sentinel: the full integer range encodes depth, 0 means zero metres, and non-finite inputs clamp to the range. This suits simulators, which render a finite depth for every pixel. With has_invalid=True, code 0 is reserved for “no measurement”: any non-finite or non-positive input encodes to 0 and decodes back to NaN, and valid depth uses [1, max_raw]. Set this for real sensors (lidar-projected, ToF, stereo) that have dropouts.
depth_transform chooses linear vs. inverse-depth spacing; depth_type records whether the scalar is planar z-depth or euclidean range (see those knobs).

depth_bits trades resolution against file size; max_depth trades range against resolution. For linear the worst-case error is half a quantization step (depth_resolution / 2):

depth_bits	max_depth	depth_resolution	max error
8	50 m	196 mm	98 mm
16	96 m	1.46 mm	0.73 mm
16	1024 m	15.6 mm	7.8 mm

Public Data Attributes:

`camera_metadata`	The sibling camera metadata that provides this stream's geometry.
`max_depth`	The far clipping plane in metres; depth at or beyond this saturates to `max_raw`.
`min_depth`	The near clipping plane in metres.
`depth_bits`	the bit depth of the stored integer raster (and of the PNG).
`depth_transform`	how metric depth is spaced across the integer codes.
`has_invalid`	Whether code `0` is a reserved "no measurement" sentinel (valid depth then uses `[1, max_raw]`).
`depth_type`	what the stored scalar measures.
`max_raw`	The largest storable integer, `2 ** depth_bits - 1` (i.e. `255` or `65535`).
`depth_dtype`	`uint8` or `uint16`.
`depth_resolution`	Metres per integer unit for the `linear` transform, `max_depth / code_span`.
`channel_type`	Always `CameraChannelType.DEPTH`.
`camera_model`	Inherited, see superclass.
`camera_id`	Inherited, see superclass.
`camera_name`	Inherited, see superclass.
`camera_to_imu_se3`	Inherited, see superclass.
`width`	Inherited, see superclass.
`height`	Inherited, see superclass.
`modality_id`	The camera id, so the stream sits at `camera_depth.<camera_id>`.

Inherited from BaseCameraMetadata

`camera_model`	The projection model of the camera.
`camera_id`	The camera ID, unique within a sensor rig.
`camera_name`	The camera name, according to the dataset naming convention.
`camera_to_imu_se3`	The static extrinsic pose of the camera relative to the IMU frame.
`width`	The width of the camera image in pixels.
`height`	The height of the camera image in pixels.
`channel_type`	The channel type of the camera image.
`modality_type`	Returns the type of the modality that this metadata describes.
`modality_id`	Returns the camera ID as the modality ID.
`aspect_ratio`	The aspect ratio (width / height) of the camera.

Inherited from BaseModalityMetadata

`modality_type`	Returns the type of the modality that this metadata describes.
`modality_id`	Optional identifier for the modality, e.g. sensor ID for sensor modalities.
`modality_key`	Returns a unique key for this modality, combining type and id if applicable.

Public Methods:

`encode_depth`(depth)	Quantize a metric depth map `(H, W)` in metres into the stored integer raster.
`decode_depth`(raw)	Dequantize a stored integer raster back to metric depth in metres.
`project_to_image`(points_cam)	Inherited, see superclass.
`to_dict`()	Serialize the metadata, embedding the sibling camera and the quantization contract.
`from_dict`(data_dict)	Construct a `DepthCameraMetadata` from a dictionary.

Inherited from BaseCameraMetadata

project_to_image(points_cam)

Project 3D points in camera frame to image pixel coordinates.

Inherited from BaseMetadata

`to_dict`()	Serialize the metadata instance to a plain Python dictionary.
`from_dict`(data_dict)	Construct a metadata instance from a plain Python dictionary.

Private Methods:

`_depth_to_unit`(depth_m)	Map clipped metric depth in `[min_depth, max_depth]` onto `[0, 1]`, monotonically increasing.
`_unit_to_depth`(t)	Inverse of `_depth_to_unit()`: map `[0, 1]` back onto metric depth.

Inherited from BaseCameraMetadata

_compute_in_fov_mask(pixel_coords, depth[, eps])

Compute a boolean mask for points in front of the camera and within image bounds.

property camera_metadata: BaseCameraMetadata¶: The sibling camera metadata that provides this stream’s geometry.

property max_depth: float¶: The far clipping plane in metres; depth at or beyond this saturates to max_raw.

property min_depth: float¶: The near clipping plane in metres. 0 (no near clip) unless set, and required for inverse.

property depth_bits: int¶

the bit depth of the stored integer raster (and of the PNG).

Type:: 8 or 16

property depth_transform: Literal['linear', 'inverse']¶

how metric depth is spaced across the integer codes.

Type:: "linear" or "inverse"

property has_invalid: bool¶: Whether code 0 is a reserved “no measurement” sentinel (valid depth then uses [1, max_raw]).

property depth_type: Literal['z_depth', 'ray_distance']¶

what the stored scalar measures.

Type:: "z_depth" (planar) or "ray_distance" (euclidean)

property max_raw: int¶: The largest storable integer, 2 ** depth_bits - 1 (i.e. 255 or 65535).

property depth_dtype: dtype¶

uint8 or uint16.

Type:: The numpy dtype of the stored raster

property depth_resolution: float¶

Metres per integer unit for the linear transform, max_depth / code_span.

For inverse the step is not constant (fine near, coarse far); this returns the same nominal value only as a rough far-plane-ish scale, and the half-step error bound does not apply.

property channel_type: CameraChannelType¶: Always CameraChannelType.DEPTH.

encode_depth(depth)[source]¶

Quantize a metric depth map (H, W) in metres into the stored integer raster.

Clips to [min_depth, max_depth], maps onto [0, 1] via depth_transform, rescales onto the valid codes [offset, max_raw], and rounds to nearest (rather than truncating, which would bias every pixel downward by half a step on average).

With has_invalid=False (default), non-finite values (NaN/inf, e.g. an unrendered pixel) clamp to the near/far plane. With has_invalid=True, any non-finite or non-positive pixel is instead written as the 0 sentinel.

Parameters:: depth (ndarray[tuple[Any, ...], dtype[floating]]) – A 2D (H, W) float array of depths in metres.
Return type:: ndarray[tuple[Any, ...], dtype[unsignedinteger]]
Returns:: A 2D (H, W) array of depth_dtype.

decode_depth(raw)[source]¶

Dequantize a stored integer raster back to metric depth in metres.

The inverse of encode_depth, up to the quantization error. Pixels that saturated on encode decode to exactly max_depth, not to their true distance. When has_invalid is set, the 0 sentinel decodes to NaN.

Parameters:: raw (ndarray[tuple[Any, ...], dtype[integer]]) – A 2D (H, W) integer array as returned by encode_depth.
Return type:: ndarray[tuple[Any, ...], dtype[float32]]
Returns:: A 2D (H, W) float32 array of depths in metres (NaN for invalid pixels).

property camera_model: CameraModel¶: Inherited, see superclass.

property camera_id: CameraID¶: Inherited, see superclass.

property camera_name: str¶: Inherited, see superclass.

property camera_to_imu_se3: PoseSE3¶: Inherited, see superclass.

property width: int¶: Inherited, see superclass.

property height: int¶: Inherited, see superclass.

project_to_image(points_cam)[source]¶

Inherited, see superclass.

Return type:: Tuple[ndarray[tuple[Any, ...], dtype[float64]], ndarray[tuple[Any, ...], dtype[bool]], ndarray[tuple[Any, ...], dtype[float64]]]
Parameters:: points_cam (ndarray[tuple[Any, ...], dtype[float64]])

to_dict()[source]¶

Serialize the metadata, embedding the sibling camera and the quantization contract.

Return type:: Dict[str, Any]

classmethod from_dict(data_dict)[source]¶

Construct a DepthCameraMetadata from a dictionary.

The transform/sentinel/type keys default to the historic behaviour (linear, no near clip, no invalid sentinel, z-depth) so .arrow files written before these knobs existed still read.

Return type:: DepthCameraMetadata
Parameters:: data_dict (Dict[str, Any])

property aspect_ratio: float¶: The aspect ratio (width / height) of the camera.

property modality_key: str¶: Returns a unique key for this modality, combining type and id if applicable.

property modality_type: ModalityType¶

Returns the type of the modality that this metadata describes.

The channel type drives the modality: a CameraChannelType.SEMANTIC camera is a per-pixel semantic segmentation stream (ModalityType.CAMERA_SEMANTIC), a CameraChannelType.INSTANCE camera a per-pixel panoptic/instance stream (ModalityType.CAMERA_INSTANCE), and a CameraChannelType.DEPTH camera a per-pixel metric depth stream (ModalityType.CAMERA_DEPTH), each written to its own Arrow file so it sits alongside — and never collides with — the RGB camera that shares its camera_id. All other channel types are regular ModalityType.CAMERA.

property modality_id: str | SerialIntEnum | None¶: The camera id, so the stream sits at camera_depth.<camera_id>.

Depth Colorization¶

py123d.datatypes.colorize_depth_map(depth_raw, max_raw=None)[source]¶

Colorize a stored integer depth raster into a (H, W, 3) uint8 RGB image for display.

Normalizes against the dtype’s full range rather than the frame’s min/max, so the colour of a given distance is stable across frames (a per-frame min/max would make the palette flicker as objects enter and leave the view). Near is warm, far is cool.

Parameters:

depth_raw (ndarray[tuple[Any, ...], dtype[integer]]) – A 2D (H, W) integer depth raster (uint8 or uint16).
max_raw (Optional[int]) – The largest storable integer. Defaults to the maximum of depth_raw.dtype.

Return type:

ndarray[tuple[Any, ...], dtype[uint8]]

Returns:

A (H, W, 3) uint8 RGB image.