Label formats
When you label a sample and press the save button, you've created a label for that sample. Labels come in different types, with the available options determined by the type of the corresponding sample.
When downloading or uploading labels using the Python SDK, the format of the attributes
field depends on the type of label. The different formats are described here.
A label can additionally also contain Object attributes and Image attributes.
Image
Segmentation labels
Format of the attributes
field in client.get_label()
:
{
"format_version": "0.1",
"annotations": [
{
"track_id": 1, // the track id. A single object has the same track_id across frames. 0.
"id": 1, // this is a legacy field and is always equal to track_id.
"category_id": 1 // this is a category id
},
{
"track_id": 2,
"id": 2,
"category_id": 1
},
{
"track_id": 3,
"id": 3,
"category_id": 4
}
],
"segmentation_bitmap": {
"url": "https://segmentsai-staging.s3.eu-west-2.amazonaws.com/assets/davy/ddf55e99-1a6f-42d2-83e9-8657de3259a1.png"
}
}
Thesegmentation_bitmap_url
refers to a 32-bit RGBA png image which contains the segmentation masks. The alpha channel is set to 255, and the remaining 24-bit values in the RGB channels correspond to the object ids in the annotations
list. Unlabeled regions should have a value of 0. Because of the large dynamic range, these png images may appear black in an image viewer.
Vector labels (bounding box, polygon, polyline, keypoint)
Format of the attributes
field in client.get_label()
:
{
"format_version": "0.1",
"annotations": [
{
"track_id": 1, // the track id. A single object has the same track_id across frames.
"id": 1, // this is a legacy field and is always equal to track_id.
"category_id": 1, // the category id
"type": "bbox", // refers to the annotation type (bounding box)
"points": [
[12.34, 56.78], // x0, y0 (upper left corner of bbox)
[90.12, 34.56] // x1, y1 (lower right corner of bbox)
]
},
{
"track_id": 2,
"id": 2,
"category_id": 2,
"type": "polygon", // refers to the annotation type (polygon)
"points": [
[12.34, 56.78], // x0, y0 (starting point of the polygon)
[90.12, 34.56], // x1, y1
[78.91, 23.45], // x2, y2
[67.89, 98.76], // x3, y3
[54.32, 10.01] // x4, y4
]
},
{
"track_id": 3,
"id": 3,
"category_id": 3,
"type": "polyline", // refers to the annotation type (polyline)
"points": [
[12.34, 56.78], // x0, y0 (starting point of the polyline)
[90.12, 34.56], // x1, y1
[78.91, 23.45], // x2, y2
[67.89, 98.76], // x3, y3
[54.32, 10.01] // x4, y4
]
},
{
"id": 4,
"track_id": 4,
"category_id": 4,
"type": "point", // refers to the annotation type (keypoint)
"points": [
[12.34, 56.78] // x, y (coordinates of keypoint)
]
},
],
"links": [
{
"from_id": 1,
"to_id": 2,
"attributes": {} // the link attributes
}
]
}
Image sequence
Segmentation labels
Format of the attributes
field in client.get_label()
:
{
"format_version": "0.1",
"frames": [
{ ... },
{ ... },
{ ... }
]
}
format_version
string
Format version.
Vector labels (bounding box, polygon, polyline, keypoint)
Format of the attributes
field in client.get_label()
:
{
"format_version": "0.2",
"frames": [
{ ... },
{ ... },
{ ... }
],
"links": [
{
"from_id": 1,
"to_id": 2,
"attributes": {} // the link attributes
}
]
}
Where each frames object has the following format:
{
"format_version": "0.1",
"timestamp": "00001", // In nanoseconds. This field is only included if the sample has a timestamp
"annotations": [
{
"track_id": 1, // the track id. A single object has the same track_id across frames.
"id": 1, // this is a legacy field and is always equal to track_id.
"category_id": 1, // the category id
"is_keyframe": true, // whether this frame is a keyframe
"type": "bbox", // refers to the annotation type (bounding box)
"points": [
[12.34, 56.78], // x0, y0 (upper left corner of bbox)
[90.12, 34.56] // x1, y1 (lower right corner of bbox)
]
},
{
"track_id": 2,
"id": 2,
"category_id": 2,
"is_keyframe": true,
"type": "polygon", // refers to the annotation type (polygon)
"points": [
[12.34, 56.78], // x0, y0 (starting point of the polygon)
[90.12, 34.56], // x1, y1
[78.91, 23.45], // x2, y2
[67.89, 98.76], // x3, y3
[54.32, 10.01] // x4, y4
]
},
{
"track_id": 3,
"id": 3,
"category_id": 3,
"is_keyframe": true,
"type": "polyline", // refers to the annotation type (polyline)
"points": [
[12.34, 56.78], // x0, y0 (starting point of the polyline)
[90.12, 34.56], // x1, y1
[78.91, 23.45], // x2, y2
[67.89, 98.76], // x3, y3
[54.32, 10.01] // x4, y4
]
},
{
"track_id": 4,
"id": 4,
"category_id": 4,
"is_keyframe": true,
"type": "point", // refers to the annotation type (keypoint)
"points": [
[12.34, 56.78] // x, y (coordinates of keypoint)
]
},
],
}
3D point cloud
Segmentation label
The annotations array contains the different objects ("annotations") in the label with their category (the category_id
should correspond to an id defined in the categories).
The point_annotations
array contains the object/annotation id for each point in the point cloud. The order of the ids in this array is the same as the order of the points in the point cloud.
{
"format_version": "0.1",
"annotations": [
{
"track_id": 1, // the track id. A single object has the same track_id across frames.
"id": 1, // this is a legacy field and is always equal to track_id.
"category_id": 1 // the category id
},
{
"track_id": 2,
"id": 2,
"category_id": 1
},
{
"track_id": 3,
"id": 3,
"category_id": 4
}
],
"point_annotations": [0, 0, 0, 3, 2, 2, 2, 1, 3...], // refers to track ids
}
Cuboid label
{
"format_version": "0.2",
"annotations": [ // list of cuboid annotations, see below
{
"track_id": 1, // the track id. A single object has the same track_id across frames.
"id": 1, // this is a legacy field and is always equal to track_id
"type": "cuboid",
...
},
{
...
}
],
"links": [
{
"from_id": 1,
"to_id": 2,
"attributes": {} // the link attributes
}
]
}
format_version
string
Format version.
Cuboid annotation
A cuboid annotation represents a single cuboid in a point cloud (frame).
{
"track_id": 1, // the track id. A single object has the same track_id across frames.
"id": 1, // this is a legacy field and is always equal to track_id
"category_id": 1,
"type": "cuboid",
"position": {
"x": 0.0,
"y": 0.2,
"z": 0.5
},
"dimensions": {
"x": 1.2,
"y": 1,
"z": 1
},
"yaw": 0.63,
"rotation": {
"qx": 0,
"qy": 0.0491566,
"qz": 0.3096865,
"qw": 0.9495672
}, // only when 3D cuboid rotation is enabled in dataset settings
"is_keyframe": true, // only in sequences
"index": 0, // only in sequences
}
id
integer
Object id.
category_id
integer
Category id.
type
string
Object type, which is always "cuboid" for cuboid annotations.
position
object
: {
"x": float
,
"y": float
,
"z": float
}
XYZ position of the center of the cuboid in world coordinates.
dimensions
object
: {
"x": float
,
"y": float
,
"z": float
}
Dimensions of the cuboid. "x" corresponds to the length, "y" to the width, and "z" to the height. See diagram 1.
yaw
float
Cuboid rotation along the z-axis in radians between [-π, π]. 0 yaw corresponds to a cuboid aligned with the x-axis pointing to increasing x-values. The yaw value increases with a counter-clockwise rotation up to π, and decreases with a clockwise rotation up to -π. See diagram 2.
rotation
object
: {
"qx": float
,
"qy": float
,
"qz": float
,
"qw": float
}
Cuboid rotation defined as a rotation quaternion. Only available when 3D rotation is enabled in the dataset settings (under "Labeling").
track_id
integer
Track ID of the object. This ID is used to track an object over multiple frames. Only relevant for sequences.
is_keyframe
boolean
Whether this cuboid annotation is a keyframe or an interpolated frame. Only relevant for sequences.
index
integer
The frame index. Only relevant for sequences.


Vector label (polygon, polyline, keypoint)
"format_version": "0.1",
"annotations": [
{
"track_id": 1, // the track id. A single object has the same track_id across frames.
"id": 1, // this is a legacy field and is always equal to track_id
"category_id": 2,
"type": "polygon", // refers to the annotation type (polygon)
"points": [
[12.34, 56.78, 0], // x0, y0, z0 (starting point of the polygon)
[90.12, 34.56, 0], // x1, y1, z1
[78.91, 23.45, 0], // x2, y2, z2
[67.89, 98.76, 0], // x3, y3, z3
[54.32, 10.01, 0] // x4, y4, z4
],
"is_keyframe": true, // only in sequences
"index": 0 // only in sequences
},
{
"track_id": 2,
"id": 2,
"category_id": 3,
"type": "polyline", // refers to the annotation type (polyline)
"points": [
[12.34, 56.78, 0], // x0, y0, z0 (starting point of the polyline)
[90.12, 34.56, 0], // x1, y1, z1
[78.91, 23.45, 0], // x2, y2, z2
[67.89, 98.76, 0], // x3, y3, z3
[54.32, 10.01, 0] // x4, y4, z4
],
"is_keyframe": false, // only in sequences
"index": 1 // only in sequences
},
{
"track_id": 3,
"id": 3,
"category_id": 4,
"type": "point", // refers to the annotation type (keypoint)
"points": [
[12.34, 56.78, 0] // x, y, z (coordinates of keypoint)
],
"is_keyframe": false, // only in sequences
"index": 2 // only in sequences
}
]
}
3D point cloud sequence
Segmentation label
{
"format_version": "0.2",
"frames": [
{ ... },
{ ... },
{ ... }
]
}
Where each frames object has the following format:
{
"format_version": "0.2",
"annotations": [
{
"track_id": 1, // the track id. A single object has the same track_id across frames.
"id": 1, // this is a legacy field and is always equal to track_id
"category_id": 1, // the category id
},
{
"track_id": 2,
"id": 2,
"category_id": 1,
},
{
"track_id": 3,
"id": 3,
"category_id": 4,
},
],
"point_annotations": [0, 0, 0, 3, 2, 2, 2, 1, 3...], // refers to object ids
}
Cuboid label
{
"format_version": "0.2",
"frames": [
{ ... },
{ ... },
{ ... }
]
}
format_version
string
Format version.
Vector label (polygon, polyline, keypoint)
Format of the attributes
field in client.get_label()
:
{
"format_version": "0.2",
"frames": [
{ ... },
{ ... },
{ ... }
],
"links": [
{
"from_id": 1,
"to_id": 2,
"attributes": {} // the link attributes
}
]
}
format_version
string
Format version.
Multi-sensor sequence
{
"sensors": [
{
"name": "Lidar",
"task_type": "pointcloud-cuboid-sequence",
"attributes": { ... }
},
{
"name": "Camera 1",
"task_type": "image-vector-sequence",
"attributes": { ... }
},
...
],
"links": [
{
"from_id": 1,
"to_id": 2,
"attributes": {} // the link attributes
}
]
}
Sensor
name
string
Name of the sensor.
task_type
string
The task type of the sensor. Currently, pointcloud-cuboid-sequence
and image-vector-sequence
are supported.
attributes
object
The label attributes for the sensor. Currently, 3D point cloud sequence and image sequence are supported.
Object attributes
Objects in the annotations list can optionally also contain an attributes field to store object-level attributes. Make sure to properly configure the label editor if you're using object-level attributes.
{
"format_version": "0.1",
"annotations": [
{
"id": 1,
"category_id": 1,
"attributes": { // object-level attributes
"is_crowd": "1",
"color": "red"
}
},
{
"id": 2,
"category_id": 1,
"attributes": {
"is_crowd": "0",
"color": "blue"
}
},
{
"id": 3,
"category_id": 4,
"attributes": {
"is_crowd": "1",
"color": "yellow"
}
}
],
...
}
Image attributes
You can also define image-level attributes. These can be useful in image classification tasks. Make sure to properly configure the label editor if you're using image-level attributes.
{
"format_version": "0.1",
"annotations": [...],
"image_attributes": { // sample-level attributes
"scene_type": "crossroads",
"weather": "sunny"
}
}
Links
The links are stored at the top level of the annotations.
[
{
"from_id": 1,
"to_id": 2,
"attributes": {
"side": "left",
"strength": 10
}
]
from_id
number
Link's object/track origin
to_id
number
Link's object/track destination
attributes
object
The link attributes.
Last updated
Was this helpful?