Documentation
  • Introduction
  • Tutorials
    • Getting started
    • Python SDK quickstart
    • Model-assisted labeling
  • How to annotate
    • Label images
      • View and navigate in the image interfaces
      • Image interface settings
      • Image segmentation interface
      • Image vector interface
    • Label 3D point clouds
      • View and navigate in the 3D interface
      • Upload, view, and overlay images
      • 3D interface settings
      • 3D point cloud cuboid interface
      • 3D point cloud vector interface
      • 3D point cloud segmentation interface
      • Merged point cloud view (for static objects)
      • Batch mode (for dynamic objects)
      • Smart cuboid propagation
      • 3D to 2D Projection
      • Tips for labeling cuboid sequences
    • Label sequences of data
      • Use track IDs in sequences
      • Use keyframe interpolation
    • Annotate object links (beta)
    • Customize hotkeys
  • How to manage
    • Add collaborators to a dataset
    • Create an organization
    • Configure the label editor
    • Customize label queue
    • Search within a dataset
    • Clone a dataset
    • Work with issues
    • Bulk change label status
    • Manage QA processes
  • How to integrate
    • Import data
      • Cloud integrations
    • Export data
      • Structure of the release file
      • Exporting image annotations to different formats
    • Integrations
      • Hugging Face
      • W&B
      • Databricks
      • SceneBox
    • Create an API key
    • Upload model predictions
    • Set up webhooks
  • Background
    • Main concepts
    • Sequences
    • Label queue mechanics
    • Labeling metrics
    • 3D Tiles
    • Security
  • Reference
    • Python SDK
    • Task types
    • Sample formats
      • Supported file formats
    • Label formats
    • Categories and attributes
    • API
Powered by GitBook
On this page
  • Image
  • Segmentation labels
  • Vector labels (bounding box, polygon, polyline, keypoint)
  • Image sequence
  • Segmentation labels
  • Vector labels (bounding box, polygon, polyline, keypoint)
  • 3D point cloud
  • Segmentation label
  • Cuboid label
  • Cuboid annotation
  • Vector label (polygon, polyline, keypoint)
  • 3D point cloud sequence
  • Segmentation label
  • Cuboid label
  • Vector label (polygon, polyline, keypoint)
  • Multi-sensor sequence
  • Object attributes
  • Image attributes

Was this helpful?

  1. Reference

Label formats

PreviousSupported file formatsNextCategories and attributes

Last updated 21 days ago

Was this helpful?

When you label a sample and press the save button, you've created a for that sample. Labels come in different types, with the available options determined by the type of the corresponding sample.

When downloading or uploading labels using the , the format of the attributes field depends on the type of label. The different formats are described here.

A label can additionally also contain Object attributes and Image attributes.

Image

Segmentation labels

Format of the attributes field in :

{
  "format_version": "0.1",
  "annotations": [
    {
      "track_id": 1, // the track id. A single object has the same track_id across frames. 0.
      "id": 1, // this is a legacy field and is always equal to track_id.
      "category_id": 1 // this is a category id
    },
    {
      "track_id": 2,
      "id": 2,
      "category_id": 1
    },
    {
      "track_id": 3,
      "id": 3,
      "category_id": 4
    }
  ],
  "segmentation_bitmap": {
    "url": "https://segmentsai-staging.s3.eu-west-2.amazonaws.com/assets/davy/ddf55e99-1a6f-42d2-83e9-8657de3259a1.png"
  }
}

Thesegmentation_bitmap_urlrefers to a 32-bit RGBA png image which contains the segmentation masks. The alpha channel is set to 255, and the remaining 24-bit values in the RGB channels correspond to the object ids in the annotations list. Unlabeled regions should have a value of 0. Because of the large dynamic range, these png images may appear black in an image viewer.

When downloading a label, you can use the utility function utils.load_label_bitmap_from_url(url) in the Python SDK to load the label bitmap as a numpy array containing object ids.

When uploading a label, the easiest way to transform a segmentation bitmap into this format and upload it is by using the util functionbitmap2file:

from segments.utils import bitmap2file

# segmentation_bitmap is a numpy array of type np.uint32, with values corresponding to instance_ids
file = bitmap2file(segmentation_bitmap)
asset = client.upload_asset(file, "label.png")
segmentation_bitmap_url = asset.url

Vector labels (bounding box, polygon, polyline, keypoint)

{
  "format_version": "0.1",
  "annotations": [
    {
      "track_id": 1, // the track id. A single object has the same track_id across frames.
      "id": 1, // this is a legacy field and is always equal to track_id.
      "category_id": 1, // the category id
      "type": "bbox", // refers to the annotation type (bounding box)
      "points": [
        [12.34, 56.78], // x0, y0 (upper left corner of bbox)
        [90.12, 34.56]  // x1, y1 (lower right corner of bbox)
      ]
    },
    {
      "track_id": 2,
      "id": 2,
      "category_id": 2,
      "type": "polygon", // refers to the annotation type (polygon)
      "points": [
        [12.34, 56.78], // x0, y0 (starting point of the polygon)
        [90.12, 34.56], // x1, y1
        [78.91, 23.45], // x2, y2
        [67.89, 98.76], // x3, y3
        [54.32, 10.01]  // x4, y4
      ]
    },
    {
      "track_id": 3,
      "id": 3,
      "category_id": 3,
      "type": "polyline", // refers to the annotation type (polyline)
      "points": [
        [12.34, 56.78], // x0, y0 (starting point of the polyline)
        [90.12, 34.56], // x1, y1
        [78.91, 23.45], // x2, y2
        [67.89, 98.76], // x3, y3
        [54.32, 10.01]  // x4, y4
      ]
    },
    {
      "id": 4,
      "track_id": 4,
      "category_id": 4,
      "type": "point", // refers to the annotation type (keypoint)
      "points": [
        [12.34, 56.78] // x, y (coordinates of keypoint)
      ]
    },
  ],
}

Image sequence

Segmentation labels

{
  "format_version": "0.1",
  "frames": [
    { ... },
    { ... },
    { ... }
  ]
}
Name
Type

format_version

string

Format version.

frames

List of segmentation labels (one per frame in the sequence).

Vector labels (bounding box, polygon, polyline, keypoint)

{
  "format_version": "0.2",
  "frames": [
    { ... },
    { ... },
    { ... }
  ]
}

Where each frames object has the following format:

{
  "format_version": "0.1",
  "timestamp": "00001", // In nanoseconds. This field is only included if the sample has a timestamp
  "annotations": [
    {
      "track_id": 1, // the track id. A single object has the same track_id across frames.
      "id": 1, // this is a legacy field and is always equal to track_id.
      "category_id": 1, // the category id
      "is_keyframe": true, // whether this frame is a keyframe
      "type": "bbox", // refers to the annotation type (bounding box)
      "points": [
        [12.34, 56.78], // x0, y0 (upper left corner of bbox)
        [90.12, 34.56]  // x1, y1 (lower right corner of bbox)
      ]
    },
    {
      "track_id": 2,
      "id": 2,
      "category_id": 2,
      "is_keyframe": true,
      "type": "polygon", // refers to the annotation type (polygon)
      "points": [
        [12.34, 56.78], // x0, y0 (starting point of the polygon)
        [90.12, 34.56], // x1, y1
        [78.91, 23.45], // x2, y2
        [67.89, 98.76], // x3, y3
        [54.32, 10.01]  // x4, y4
      ]
    },
    {
      "track_id": 3,
      "id": 3, 
      "category_id": 3,
      "is_keyframe": true,
      "type": "polyline", // refers to the annotation type (polyline)
      "points": [
        [12.34, 56.78], // x0, y0 (starting point of the polyline)
        [90.12, 34.56], // x1, y1
        [78.91, 23.45], // x2, y2
        [67.89, 98.76], // x3, y3
        [54.32, 10.01]  // x4, y4
      ]
    },
    {
      "track_id": 4,
      "id": 4,
      "category_id": 4,
      "is_keyframe": true,
      "type": "point", // refers to the annotation type (keypoint)
      "points": [
        [12.34, 56.78] // x, y (coordinates of keypoint)
      ]
    },
  ],
}

3D point cloud

Segmentation label

The point_annotations array contains the object/annotation id for each point in the point cloud. The order of the ids in this array is the same as the order of the points in the point cloud.

{
  "format_version": "0.1",
  "annotations": [
    {
      "track_id": 1, // the track id. A single object has the same track_id across frames.
      "id": 1, // this is a legacy field and is always equal to track_id.
      "category_id": 1 // the category id
    },
    {
      "track_id": 2,
      "id": 2,
      "category_id": 1
    },
    {
      "track_id": 3,
      "id": 3,
      "category_id": 4
    }
  ],
  "point_annotations": [0, 0, 0, 3, 2, 2, 2, 1, 3...], // refers to track ids
}

Cuboid label

{
  "format_version": "0.2",
  "annotations": [ // list of cuboid annotations, see below
    {
      "track_id": 1, // the track id. A single object has the same track_id across frames.
      "id": 1, // this is a legacy field and is always equal to track_id
      "type": "cuboid",
      ...
    },
    { 
      ... 
    }
  ]
}
Name
Type
Description

format_version

string

Format version.

annotations

List of the cuboid annotations.

Cuboid annotation

A cuboid annotation represents a single cuboid in a point cloud (frame).

{
  "track_id": 1, // the track id. A single object has the same track_id across frames.
  "id": 1, // this is a legacy field and is always equal to track_id
  "category_id": 1,
  "type": "cuboid",
  "position": {
    "x": 0.0,
    "y": 0.2,
    "z": 0.5
  },
  "dimensions": {
    "x": 1.2,
    "y": 1,
    "z": 1
  },
  "yaw": 0.63,
  "rotation": {
    "qx": 0,
    "qy": 0.0491566,
    "qz": 0.3096865,
    "qw": 0.9495672
  },  // only when 3D cuboid rotation is enabled in dataset settings
  "is_keyframe": true,  // only in sequences
  "index": 0,  // only in sequences 
}
Name
Type
Description

id

integer

Object id.

category_id

integer

Category id.

type

string

Object type, which is always "cuboid" for cuboid annotations.

position

object: { "x": float, "y": float, "z": float }

XYZ position of the center of the cuboid in world coordinates.

dimensions

object: { "x": float, "y": float, "z": float }

Dimensions of the cuboid. "x" corresponds to the length, "y" to the width, and "z" to the height. See diagram 1.

yaw

float

Cuboid rotation along the z-axis in radians between [-π, π]. 0 yaw corresponds to a cuboid aligned with the x-axis pointing to increasing x-values. The yaw value increases with a counter-clockwise rotation up to π, and decreases with a clockwise rotation up to -π. See diagram 2.

rotation

object: { "qx": float, "qy": float, "qz": float,

"qw": float }

track_id

integer

Track ID of the object. This ID is used to track an object over multiple frames. Only relevant for sequences.

is_keyframe

boolean

Whether this cuboid annotation is a keyframe or an interpolated frame. Only relevant for sequences.

index

integer

The frame index. Only relevant for sequences.

Vector label (polygon, polyline, keypoint)

  "format_version": "0.1",
  "annotations": [
    {
      "track_id": 1, // the track id. A single object has the same track_id across frames.
      "id": 1, // this is a legacy field and is always equal to track_id
      "category_id": 2,
      "type": "polygon", // refers to the annotation type (polygon)
      "points": [
        [12.34, 56.78, 0], // x0, y0, z0 (starting point of the polygon)
        [90.12, 34.56, 0], // x1, y1, z1
        [78.91, 23.45, 0], // x2, y2, z2
        [67.89, 98.76, 0], // x3, y3, z3
        [54.32, 10.01, 0]  // x4, y4, z4
      ],
      "is_keyframe": true, // only in sequences
      "index": 0 // only in sequences 
    },
    {
      "track_id": 2,
      "id": 2,
      "category_id": 3,
      "type": "polyline", // refers to the annotation type (polyline)
      "points": [
        [12.34, 56.78, 0], // x0, y0, z0 (starting point of the polyline)
        [90.12, 34.56, 0], // x1, y1, z1
        [78.91, 23.45, 0], // x2, y2, z2
        [67.89, 98.76, 0], // x3, y3, z3
        [54.32, 10.01, 0]  // x4, y4, z4
      ],
      "is_keyframe": false, // only in sequences
      "index": 1 // only in sequences 
    },
    {
      "track_id": 3,
      "id": 3,
      "category_id": 4,
      "type": "point", // refers to the annotation type (keypoint)
      "points": [
        [12.34, 56.78, 0] // x, y, z (coordinates of keypoint)
      ],
      "is_keyframe": false, // only in sequences
      "index": 2 // only in sequences 
    }
  ]
}

3D point cloud sequence

Segmentation label

{
  "format_version": "0.2",
  "frames": [
    { ... },
    { ... },
    { ... }
  ]
}

Where each frames object has the following format:

{
  "format_version": "0.2",
  "annotations": [
    {
      "track_id": 1, // the track id. A single object has the same track_id across frames.
      "id": 1, // this is a legacy field and is always equal to track_id
      "category_id": 1, // the category id
    },
    {
      "track_id": 2,
      "id": 2,
      "category_id": 1,
    },
    {
      "track_id": 3,
      "id": 3,
      "category_id": 4,
    },
  ],
  "point_annotations": [0, 0, 0, 3, 2, 2, 2, 1, 3...], // refers to object ids
}

Cuboid label

{
  "format_version": "0.2",
  "frames": [
    { ... },
    { ... },
    { ... }
  ]
}
Name
Type

format_version

string

Format version.

frames

List of cuboid labels (one per frame in the sequence).

Vector label (polygon, polyline, keypoint)

{
  "format_version": "0.2",
  "frames": [
    { ... },
    { ... },
    { ... }
  ]
}
Name
Type

format_version

string

Format version.

frames

List of vector labels (one per frame in the sequence).

Multi-sensor sequence

{
  "sensors": [
    {
      "name": "Lidar", 
      "task_type": "pointcloud-cuboid-sequence",
      "attributes": { ... }
    },
    {
      "name": "Camera 1", 
      "task_type": "image-vector-sequence",
      "attributes": { ... } 
    },
    ...
  ]
}
Name
Type
Description

sensors

List of the sensors with label attributes.

Sensor

Name
Type
Description

name

string

Name of the sensor.

task_type

string

attributes

object

Object attributes

{
  "format_version": "0.1",
  "annotations": [
    {
      "id": 1, 
      "category_id": 1,
      "attributes": { // object-level attributes
        "is_crowd": "1",
        "color": "red"
      }
    },
    {
      "id": 2, 
      "category_id": 1,
      "attributes": {
        "is_crowd": "0",
        "color": "blue"
      }
    },
    {
      "id": 3, 
      "category_id": 4,
      "attributes": {
        "is_crowd": "1",
        "color": "yellow"
      }
    }
  ],
  ...
}

Image attributes

{
  "format_version": "0.1",
  "annotations": [...],
  "image_attributes": { // sample-level attributes
    "scene_type": "crossroads",
    "weather": "sunny"
  }
}

For a full example of uploading model-generated labels to Segments.ai, please refer to .

Format of the attributes field in :

Format of the attributes field in :

array of

Format of the attributes field in :

The annotations array contains the different objects ("annotations") in the label with their category (the category_id should correspond to an id defined in the ).

array of

Cuboid rotation defined as a . Only available when 3D rotation is enabled in the dataset settings (under "Labeling").

array of

Format of the attributes field in :

array of

array of

The of the sensor. Currently, pointcloud-cuboid-sequence and image-vector-sequence are supported.

The label attributes for the sensor. Currently, and are supported.

Objects in the annotations list can optionally also contain an attributes field to store object-level attributes. Make sure to properly if you're using object-level attributes.

You can also define image-level attributes. These can be useful in image classification tasks. Make sure to properly if you're using image-level attributes.

this blogpost
client.get_label()
client.get_label()
client.get_label()
client.get_label()
configure the label editor
configure the label editor
rotation quaternion
task type
segmentation labels
cuboid annotations
cuboid labels
vector labels
sensors
3D point cloud sequence
image sequence
Python SDK
client.get_label()
label
categories
Diagram 1: x and y attributes of the cuboid dimensions. The red arrow shows the cuboid heading.
Diagram 2: yaw rotation of a cuboid. The red arrow shows the cuboid heading. yaw = π/2 corresponds to a heading in the direction of increasing y values, while yaw = -π/2 corresponds to a heading in the direction of decreasing y values.