Documentation
Search…
Label types
When downloading or uploading labels using the Python SDK, the format of theattributes field depends on the type of label. The different formats are described here.

Image

Segmentation labels

Format of the attributes field in client.get_label():
1
{
2
"format_version": "0.1",
3
"annotations": [
4
{
5
"id": 1, // this is an object id. Should be > 0.
6
"category_id": 1 // this is a category id
7
},
8
{
9
"id": 2,
10
"category_id": 1
11
},
12
{
13
"id": 3,
14
"category_id": 4
15
}
16
],
17
"segmentation_bitmap": {
18
"url": "https://segmentsai-staging.s3.eu-west-2.amazonaws.com/assets/davy/ddf55e99-1a6f-42d2-83e9-8657de3259a1.png"
19
}
20
}
Copied!
Thesegmentation_bitmap_urlrefers to a 32-bit RGBA png image which contains the segmentation masks. The alpha channel is set to 255, and the remaining 24-bit values in the RGB channels correspond to the object ids in the annotations list. Unlabeled regions should have a value of 0. Because of the large dynamic range, these png images may appear black in an image viewer.
When downloading a label, you can use the utility function utils.load_label_bitmap_from_url(url) in the Python SDK to load the label bitmap as a numpy array containing object ids.
When uploading a label, the easiest way to transform a segmentation bitmap into this format and upload it is by using the util functionbitmap2file:
1
from segments.utils import bitmap2file
2
3
# segmentation_bitmap is a numpy array of type np.uint32, with values corresponding to instance_ids
4
file = bitmap2file(segmentation_bitmap)
5
asset = client.upload_asset(file, "label.png")
6
segmentation_bitmap_url = asset["url"]
Copied!
For a full example of uploading model-generated labels to Segments.ai, please refer to this blogpost.

Vector labels (bounding box, polygon, polyline, keypoint)

Format of the attributes field in client.get_label():
1
{
2
"format_version": "0.1",
3
"annotations": [
4
{
5
"id": 1, // the object id
6
"category_id": 1, // the category id
7
"type": "bbox", // refers to the annotation type (bounding box)
8
"points": [
9
[12.34, 56.78], // x0, y0 (upper left corner of bbox)
10
[90.12, 34.56] // x1, y1 (lower right corner of bbox)
11
]
12
},
13
{
14
"id": 2,
15
"category_id": 2,
16
"type": "polygon", // refers to the annotation type (polygon)
17
"points": [
18
[12.34, 56.78], // x0, y0 (starting point of the polygon)
19
[90.12, 34.56], // x1, y1
20
[78.91, 23.45], // x2, y2
21
[67.89, 98.76], // x3, y3
22
[54.32, 10.01] // x4, y4
23
]
24
},
25
{
26
"id": 3,
27
"category_id": 3,
28
"type": "polyline", // refers to the annotation type (polyline)
29
"points": [
30
[12.34, 56.78], // x0, y0 (starting point of the polyline)
31
[90.12, 34.56], // x1, y1
32
[78.91, 23.45], // x2, y2
33
[67.89, 98.76], // x3, y3
34
[54.32, 10.01] // x4, y4
35
]
36
},
37
{
38
"id": 4,
39
"category_id": 4,
40
"type": "point", // refers to the annotation type (keypoint)
41
"points": [
42
[12.34, 56.78] // x, y (coordinates of keypoint)
43
]
44
},
45
],
46
}
Copied!

Image sequence

Segmentation labels

Coming soon.

Vector labels (bounding box, polygon, polyline, keypoint)

Format of the attributes field in client.get_label():
1
{
2
"format_version": "0.2",
3
"frames": [
4
{ ... },
5
{ ... },
6
{ ... }
7
]
8
}
Copied!
Where each frames object has the following format:
1
{
2
"format_version": "0.1",
3
"timestamp": "00001", // this field is only included if the sample has a timestamp
4
"annotations": [
5
{
6
"id": 1, // the object id
7
"category_id": 1, // the category id
8
"track_id": 6, // this id is used to links objects across frame
9
"is_keyframe": true, // whether this frame is a keyframe
10
"type": "bbox", // refers to the annotation type (bounding box)
11
"points": [
12
[12.34, 56.78], // x0, y0 (upper left corner of bbox)
13
[90.12, 34.56] // x1, y1 (lower right corner of bbox)
14
]
15
},
16
{
17
"id": 2,
18
"category_id": 2,
19
"track_id": 5, // this id is used to links objects across frame
20
"is_keyframe": true, // whether this frame is a keyframe
21
"type": "polygon", // refers to the annotation type (polygon)
22
"points": [
23
[12.34, 56.78], // x0, y0 (starting point of the polygon)
24
[90.12, 34.56], // x1, y1
25
[78.91, 23.45], // x2, y2
26
[67.89, 98.76], // x3, y3
27
[54.32, 10.01] // x4, y4
28
]
29
},
30
{
31
"id": 3,
32
"category_id": 3,
33
"track_id": 4, // this id is used to links objects across frame
34
"is_keyframe": true, // whether this frame is a keyframe
35
"type": "polyline", // refers to the annotation type (polyline)
36
"points": [
37
[12.34, 56.78], // x0, y0 (starting point of the polyline)
38
[90.12, 34.56], // x1, y1
39
[78.91, 23.45], // x2, y2
40
[67.89, 98.76], // x3, y3
41
[54.32, 10.01] // x4, y4
42
]
43
},
44
{
45
"id": 4,
46
"category_id": 4,
47
"track_id": 3, // this id is used to links objects across frame
48
"is_keyframe": true, // whether this frame is a keyframe
49
"type": "point", // refers to the annotation type (keypoint)
50
"points": [
51
[12.34, 56.78] // x, y (coordinates of keypoint)
52
]
53
},
54
],
55
}
Copied!

3D point cloud

Segmentation labels

Format of the attributes field in client.get_label():
1
{
2
"format_version": "0.1",
3
"annotations": [
4
{
5
"id": 1, // the object id
6
"category_id": 1 // the category id
7
},
8
{
9
"id": 2,
10
"category_id": 1
11
},
12
{
13
"id": 3,
14
"category_id": 4
15
}
16
],
17
"point_annotations": [0, 0, 0, 3, 2, 2, 2, 1, 3...], // refers to object ids
18
}
Copied!

Cuboid labels

Format of the attributes field in client.get_label():
1
{
2
"format_version": "0.1",
3
"annotations": [
4
{
5
"id": 1, // the object id
6
"category_id": 1, // the category id
7
"type": "cuboid", // refers to the annotation type (cuboid)
8
"position": {
9
"x": 0.0,
10
"y": 0.2,
11
"z": 0.5
12
},
13
"dimensions": {
14
"x": 1.2,
15
"y": 1,
16
"z": 1
17
},
18
"yaw": 1.63
19
}
20
]
21
}
Copied!

3D point cloud sequence

Segmentation labels

Format of the attributes field in client.get_label():
1
{
2
"format_version": "0.2",
3
"frames": [
4
{ ... },
5
{ ... },
6
{ ... }
7
]
8
}
Copied!
Where each frames object has the following format:
1
{
2
"format_version": "0.2",
3
"annotations": [
4
{
5
"id": 1, // the object id
6
"category_id": 1, // the category id
7
"track_id": 3 // this id is used to link objects across frames
8
},
9
{
10
"id": 2,
11
"category_id": 1,
12
"track_id": 4
13
},
14
{
15
"id": 3,
16
"category_id": 4,
17
"track_id": 5
18
},
19
],
20
"point_annotations": [0, 0, 0, 3, 2, 2, 2, 1, 3...], // refers to object ids
21
}
Copied!

Cuboid labels

Format of the attributes field in client.get_label():
1
{
2
"format_version": "0.2",
3
"frames": [
4
{ ... },
5
{ ... },
6
{ ... }
7
]
8
}
Copied!
Where each frames object has the following format:
1
{
2
"format_version": "0.2",
3
"timestamp": "00001", // this field is only included if the sample has a timestamp
4
"annotations": [
5
{
6
"id": 1, // the object id
7
"category_id": 1, // the category id
8
"type": "cuboid", // refers to the annotation type (cuboid)
9
"position": {
10
"x": 0.0,
11
"y": 0.2,
12
"z": 0.5
13
},
14
"dimensions": {
15
"x": 1.2,
16
"y": 1,
17
"z": 1
18
},
19
"yaw": 1.63,
20
"is_keyframe": true, // whether this frame is a keyframe
21
"track_id": 6, // this id is used to links objects across frames
22
},
23
{
24
"id": 2,
25
"category_id": 2,
26
"type": "cuboid",
27
"position": {
28
"x": 0.0,
29
"y": 0.2,
30
"z": 0.5
31
},
32
"dimensions": {
33
"x": 1.2,
34
"y": 1,
35
"z": 1
36
},
37
"yaw": 1.63,
38
"is_keyframe": false,
39
"track_id": 7
40
}
41
]
42
},
Copied!

Object attributes

Objects in the annotations list can optionally also contain an attributes field to store object-level attributes. Make sure to properly configure the label editor if you're using object-level attributes.
1
{
2
"format_version": "0.1",
3
"annotations": [
4
{
5
"id": 1,
6
"category_id": 1,
7
"attributes": { // object-level attributes
8
"is_crowd": "1",
9
"color": "red"
10
}
11
},
12
{
13
"id": 2,
14
"category_id": 1,
15
"attributes": {
16
"is_crowd": "0",
17
"color": "blue"
18
}
19
},
20
{
21
"id": 3,
22
"category_id": 4,
23
"attributes": {
24
"is_crowd": "1",
25
"color": "yellow"
26
}
27
}
28
],
29
...
30
}
Copied!

Image attributes

You can also define image-level attributes. These can be useful in image classification tasks. Make sure to properly configure the label editor if you're using image-level attributes.
1
{
2
"format_version": "0.1",
3
"annotations": [...],
4
"image_attributes": { // sample-level attributes
5
"scene_type": "crossroads",
6
"weather": "sunny"
7
}
8
}
Copied!
Last modified 14d ago