# Introduction
Segments.ai is the training data platform for computer vision engineers and labeling teams. Our powerful labeling interfaces, easy-to-use management features, and extensive API integrations help you iterate quickly between data labeling, model training and failure case discovery.
The documentation is divided into 4 sections:
1. [Tutorials](/tutorials/getting-started): end-to-end tutorials where you are guided through a complete project.
2. How-to guides: recipes to solve specific problems relating to
* [Annotating data](/how-to-annotate/label-images)
* [Managing data and collaborators](/how-to-integrate/import-data)
* [Integrating Segments.ai in your workflows and ML pipelines](/how-to-integrate/import-data)
3. [Background](/background/main-concepts): provides clarifying background knowledge on certain topics.
4. [Reference](/reference/sample-and-label-types): description of the full Python SDK functionality and input and output formats.
# Getting started
This tutorial shows how you can create a dataset, upload some images in it, label and review them, and finally create a dataset release to export your data.
{% hint style="info" %}
If you want to know how to interact programmatically with Segments.ai, follow the [Python SDK quickstart](/tutorials/python-sdk-quickstart) instead.
{% endhint %}
First, make sure you've [created an account](https://segments.ai/join) and are logged in. Clicking on the Segments.ai logo in the upper left corner, takes you to your home screen:
## Create a new dataset
From the home screen, click the "New dataset" button. Choose a dataset name and description, set the visibility, and choose a category. Then click "Next".
Choose "Segmentation (bitmap)" for the task type in this tutorial. We will be labeling a dataset of pets in this tutorial, so we add three object categories: cat, dog and other.
Finally, click "Create dataset".
## Upload images
Now that you've created a new dataset, let's add some images to it. In your dataset, go to the Samples tab and click the "Add Samples" button.
{% hint style="info" %}
A sample can be an image, video or 3D pointcloud. In this tutorial we're working with images.
{% endhint %}
Select a few images and upload them. The images appear as thumbnails in the Samples tab.
## Label and review
Press the "Start labeling" button. This brings you into the labeling workflow, where you will be presented with unlabeled images until the labeling queue is empty.
Label all objects in the image and press "Submit", until all images are labeled. More details on effectively using the segmentation interface can be found [here](/how-to-annotate/label-images/image-segmentation-interface).
When done labeling, press the little cross icon in the top right corner to exit the labeling interface.
Optionally, press the "Start reviewing" button. This brings you into the reviewing workflow, where you can accept or reject labeled images. Rejected images go back to the labeling queue for correction.
{% hint style="info" %}
Instead of using the "Start labeling" and "Start reviewing" buttons, you can also open and edit any image directly by clicking its thumbnail. Note that in this case, the image is not prevented from being edited by someone else at the same time.
{% endhint %}
## Export data
Go to the Releases tab. Here you can create a snapshot of your dataset with a download link to the finished labels.
Press the "Create a new release button" and choose a name and description for the release. Creating the release may take a few seconds.
For more details on exporting data to different formats, see the [Export data](/how-to-integrate/export) section.
# Python SDK quickstart
On the Segments.ai web platform you can create datasets, upload samples, create releases and download labels. All of these - and more - can also be done programmatically with the Python SDK.
{% hint style="info" %}
This tutorial walks you through the most common Python SDK functions. The complete list of functions is documented in detail in the [Python SDK reference](https://sdkdocs.segments.ai/).
{% endhint %}
First install the Segments.ai Python SDK using pip:
```
pip install --upgrade segments-ai
```
Import the necessary packages, and initialize the Segments client using your API key:
```python
from segments import SegmentsClient
import json
# You can find your api key at https://segments.ai/account
api_key = "YOUR_API_KEY_HERE"
client = SegmentsClient(api_key)
```
## Create a new dataset
Let's create a new image segmentation dataset programmatically using [`client.add_dataset()`](https://sdkdocs.segments.ai/en/latest/client.html#create-a-dataset). Note that this dataset will be created under the user account corresponding to the API key.
{% hint style="info" %}
The format of the `task_attributes` field is documented [here](/reference/categories-and-attributes).
{% endhint %}
```python
name = "pets"
description = "A dataset with images of cats and dogs."
task_type = "segmentation-bitmap"
task_attributes = {
"format_version": "0.1",
"categories": [
{
"name": "cat",
"id": 1
},
{
"name": "dog",
"id": 2
},
{
"name": "other",
"id": 3
}
]
}
dataset = client.add_dataset(name, description, task_type, task_attributes)
print(dataset)
```
## Add samples to a dataset
Now let's upload some images to this dataset using [`dataset.add_sample()`](https://sdkdocs.segments.ai/en/latest/client.html#create-a-sample).
```python
dataset_identifier = 'jane/pets' # a dataset is always referred to as user/dataset.
images = [
{
'name': 'Bombay_220.jpg',
'url': 'https://segmentsai-prod.s3.eu-west-2.amazonaws.com/assets/jane/a13358ef-a1ae-443c-8ea1-5f61dd9cdc26.jpg'
},
{
'name': 'shiba_inu_178.jpg',
'url': 'https://segmentsai-prod.s3.eu-west-2.amazonaws.com/assets/jane/4f47973f-5568-47f1-8a7d-44bfeb6f0f76.jpg'
},
{
'name': 'havanese_196.jpg',
'url': 'https://segmentsai-prod.s3.eu-west-2.amazonaws.com/assets/jane/2a6b3cd9-0688-422b-b555-8730d0813e1b.jpg'
}
]
dataset = client.get_dataset(dataset_identifier)
for image in images:
name = image['name']
attributes = {
'image': {
'url': image['url']
}
}
sample = dataset.add_sample(name, attributes)
print(sample)
```
{% hint style="warning" %}
If the image file is on your local computer, you should first upload it to our asset storage service (using [`upload_asset()`](https://sdkdocs.segments.ai/en/latest/client.html#upload-an-asset-to-segments-s3-bucket)) or to another cloud storage service.
{% endhint %}
We can verify that the dataset now contains 3 images using [`dataset.get_samples()`](https://sdkdocs.segments.ai/en/latest/client.html#list-samples).
```python
dataset_identifier = 'jane/pets'
dataset = client.get_dataset(dataset_identifier)
samples = dataset.get_samples()
print(samples)
```
Now switch to the Segments.ai web platform and label the three images you just uploaded by pressing the "Start labeling" button.
## Get the label of a sample
Once you've labeled some samples, you can programmatically retrieve their labels using [`sample.get_label()`](https://sdkdocs.segments.ai/en/latest/client.html#get-a-label).
```python
dataset_identifier = 'jane/pets'
sample = client.get_dataset(dataset_identifier).get_samples()[0]
label = sample.get_label(labelset='ground-truth')
print(label)
```
## Optional: visualize the instance and semantic labels
When working with image segmentation datasets, you'll probably want to visualize the image and label at this point. The `segments.utils` module offers some helper functions for that:
```python
import numpy as np
import matplotlib.pyplot as plt
from segments.utils import load_image_from_url, load_label_bitmap_from_url, get_semantic_bitmap
# Load the labels as numpy arrays
image = load_image_from_url(sample.attributes.image.url)
instance_bitmap = load_label_bitmap_from_url(label.attributes.segmentation_bitmap.url)
semantic_bitmap = get_semantic_bitmap(instance_bitmap, label.attributes.annotations)
# Visualize
plt.imshow(image)
plt.title('Image')
plt.show()
plt.imshow(instance_bitmap)
plt.title(f'Instance bitmap. Values represent instance ids: {np.unique(instance_bitmap)}')
plt.show()
plt.imshow(semantic_bitmap)
plt.title(f'Semantic bitmap. Values represent category ids: {np.unique(semantic_bitmap)}')
plt.show()
```
## What's next?
The Python SDK offers many more functions besides the ones that were shown here. Have a look at the [reference](https://sdkdocs.segments.ai/en/latest/client.html) for the full list.
The Python SDK can also be used to upload labels into Segments.ai. This is particularly useful for setting up [model-assisted labeling](/tutorials/model-assisted-labeling) workflows, where you verify and correct model predictions instead of labeling from scratch.
# Model-assisted labeling
One way to drastically speed up data labeling is by leveraging your machine learning models from the start. Instead of labeling the entire dataset manually, you can use a model to help you by iterating between image labeling and model training.
Segments.ai makes it easy to set up such model-assisted workflows.
We have two interactive tutorials in the form of a Google Colab notebook:
* [Speed up your image segmentation workflow with model-assisted labeling](https://colab.research.google.com/github/segments-ai/fast-labeling-workflow/blob/master/demo.ipynb)
* [Fast point cloud labeling with model predictions](https://colab.research.google.com/drive/1ZgHiMeHNlEB5WFyw93EiRkMd-0jmb7f4)
# Label images
This section contains how-to guides for using the image labeling interfaces.
* [Image segmentation interface (bitmap)](/how-to-annotate/label-images/image-segmentation-interface)
* [Image vector interface (bounding box, polygon, polyline, keypoint)](/how-to-annotate/label-images/image-vector-interface)
# View and navigate in the image interfaces
## Pan the image
Hold `Ctrl/cmd` and click and drag to pan the image.
## Zoom in/out
Hold `Ctrl/cmd` while scrolling the mouse wheel to zoom in or out.
## Zoom to an object
1. Select the object
2. Press the hotkey (`t` by default)
## Reset the pan and zoom level
Press hotkey `f` to reset to the original pan and zoom level.
## Show only the image or only the label
Hover over the  buttons in the top right of the screen to show only the image, label or both (default). You can also hold the hotkeys `i` or `l` to show only the image or only the label.
# Image interface settings
## Open the 2D interface settings
Click on the "Settings" tab in the sidebar on the right to open the 2D interface settings.
## Object coloring mode
The object coloring toggle allows you to change how different objects in the sample are displayed.
Click on a mode to change the object coloring mode.
## Image display
### Brightness
Drag the slider to adjust the image brightness.
### Contrast
Drag the slider to adjust the image contrast.
### Equalize histogram
Toggling this checkbox will equalize the image's histogram based on the luminance. Useful for labelling dark images.
## Label display
### Opacity
Drag the slider to adjust the label opacity.
# Image segmentation interface
{% hint style="info" %}
When [creating a dataset](https://sdkdocs.segments.ai/en/latest/client.html#create-a-dataset) through the Python SDK, choose `segmentation-bitmap` as the `task_type` to use this labeling interface.
{% endhint %}
{% hint style="info" %}
The image segmentation interface is a bitmap interface: the resulting label is an image containing object segmentation masks. If you need vector labels containing polygon, polyline, bounding box or keypoint coordinates, use the [image vector interface](/how-to-annotate/label-images/image-vector-interface) instead.
{% endhint %}
{% embed url="" %}
## Add a new object
To add a new object:
1. Select a drawing tool (see [#drawing-tools](#drawing-tools "mention"))
2. Label the object by drawing on the image
3. Press `Space` to confirm the object once you finished labeling
## Select an object
1. Press `Space` or `Escape` or click the  button in the left toolbar to enter view-and-select mode
2. Click on a labeled object to select it. You can also select an object by selecting it in the right sidebar
## Remove an object
Select an object and press `Backspace` to remove it. You can also click the  icon in the sidebar to remove an object.
## Change the category of an object
Select an object and click on its name in the sidebar to select a different category. You can also use hotkey `c` to change the category of the active object.
## Drawing tools
{% hint style="info" %}
You can switch seamlessly between drawing tools while labeling
{% endhint %}
### Brush
1. Click the button in the left toolbar or press `b` to select the brush tool
2. Change the brush size by adjusting the slider in the top left of the screen, or by scrolling your mouse wheel
3. Click and drag on the image to draw
### Polygon
1. Click the button in the left toolbar or press `g` to select the polygon tool
2. Click on the image to draw polygon points. You can also click and drag to draw free-form lines
3. Double click or click the first point of the polygon to finish it. The region within the polygon will now be annotated
### Superpixels
1. Click thebutton in the left toolbar or press `s` to select the superpixel tool
2. Change the superpixel granularity by adjusting the slider in the top left of the screen, or by scrolling your mouse wheel
3. Click on a superpixel to select the corresponding region, or drag across multiple superpixels to select more than one at a time
4. Click a superpixel again to unselect the region
More info in [this blog post](https://segments.ai/blog/superpixels).
### Autosegment
1. Click thebutton in the left toolbar or press `a` to select the autosegment tool
2. Draw a box around an object in the image through click and drag
3. The object within the box will be automatically segmented
More info in [this blog post](https://segments.ai/blog/autosegment).
### Hover-and-click
* Click thebutton in the left toolbar or press `d` to select the hover-and-click tool
* Hover with your mouse over the image to see mask suggestions appear
* Click to select a suggested mask
More info in [this blog post](https://segments.ai/blog/faster-labeling-with-meta-segment-anything-model).
## Erase and protect mode
* The object within the box will be automatically segmented
### Erase mode
When labeling in brush, polygon or autosegment mode, an eraser button  will appear in the left toolbar. Click it or hold hotkey `e` to enable the erase mode.
### Protect mode
Click the button in the left toolbar or press hotkey `r` to toggle the protect mode. When protect mode is enabled, existing objects will not be affected when drawing over them.
## Ruler mode
1. Click the button in the right toolbar or press hotkey `z` to toggle the ruler mode
2. Click in the interface to set the first point
3. Move your cursor to another point and it will display the measurement in pixels
4. Click again to go back to select-and-view mode or press `esc`
### Lock a track
See [Object locking](/how-to-annotate/object-locking)
# Image vector interface
{% hint style="info" %}
When [creating a dataset](https://sdkdocs.segments.ai/en/latest/client.html#create-a-dataset) through the Python SDK, choose `vector` or `image-vector-sequence` as the `task_type` to use this labeling interface.
{% endhint %}
## Add a new object
To add a new object:
1. Select an annotation type (see [#annotation-types](#annotation-types "mention"))
2. Label the object by drawing on the image
## Select an object
1. Press `Space` or `Escape` or click the  button in the left toolbar to enter view-and-select mode
2. Click on a labeled object to select it. You can also select an object by selecting it in the right sidebar
## Remove an object
Select an object and press `Backspace` to remove it. You can also click the  icon in the sidebar to remove an object.
## Change the category of an object
Select an object and click on its name in the sidebar to select a different category. You can also use hotkey `c` to change the category of the active object.
## Intersect bounding boxes / polygons
To make it easy to draw shapes closely together you can just draw overlapping bounding boxes and polygons. If you then select one of the overlapping shapes you can press the hotkey (default `r` ) to intersect the active shape with the other shapes it overlaps with.
## Annotation types
### Bounding box
Click thebutton in the left toolbar to draw a bounding box. Click and drag to draw the box.
### Polygon
Click thebutton in the left toolbar to draw a polygon. Click on the image to draw polygon points. Double click or click the first point of the polygon to finish it.
After finishing a polygon, you can add a point to it with `Shift + left click`. This will add a point to the edge closest of the cursor. You will see a preview of where the point will be added as soon as you press `Shift`. You can add points to a selected polygon in *view-and-select* mode, or while still in *draw polygon* mode after finishing it. \
You can remove a point by `Shift + left click`. A polygon needs at least 3 points.
When holding `Shift + F`node snapping is activated. This will snap the point you want to add to the nearest handle of an existing vector, within a certain distance threshold.
### Polyline
Click thebutton in the left toolbar to draw a polyline. Click on the image to draw polyline points. Double click to finish it.
After finishing a polyline, you can add a point to it with `Shift + left click`. This will add a point to the edge closest of the cursor. You will see a preview of where the point will be added as soon as you press `Shift`. You can add points to a selected polyline in *view-and-select* mode, or while still in *draw polygon* mode after finishing it.\
You can remove a point by `Shift + left click`. A polyline needs at least 2 points.
When holding `Shift + F`node snapping is activated. This will snap the point you want to add to the nearest handle of an existing vector, within a certain distance threshold.
### Keypoint
Click thebutton in the left toolbar to draw a keypoint.
## Ruler mode
1. Click the button in the right toolbar or press hotkey `z` to toggle the ruler mode
2. Click in the interface to set the first point
3. Move your cursor to another point and it will display the measurement in pixels
4. Click again to go back to select-and-view mode or press `esc`
# Label 3D point clouds
This section contains how-to guides for using the 3D point cloud labeling interfaces.
* [View and navigate in the 3D interface](/how-to-annotate/label-3d-point-clouds/view-and-navigate-in-the-3d-interface)
* [Upload, view, and overlay images](/how-to-annotate/label-3d-point-clouds/upload-view-and-overlay-images)
* [3D point cloud segmentation interface](/how-to-annotate/label-3d-point-clouds/3d-point-cloud-segmentation-interface)
* [3D point cloud cuboid interface](/how-to-annotate/label-3d-point-clouds/3d-point-cloud-cuboid-interface)
* [3D point cloud vector interface (polylines, polygons, keypoints)](/how-to-annotate/label-3d-point-clouds/3d-point-cloud-vector-interface)
* [Batch mode (for dynamic objects)](/how-to-annotate/label-3d-point-clouds/batch-mode-for-dynamic-objects)
* [Merged point cloud view](/how-to-annotate/label-3d-point-clouds/merged-point-cloud-view-for-static-objects)
* [Tips for labeling cuboid sequences](/how-to-annotate/label-3d-point-clouds/tips-for-labeling-cuboid-sequences)
{% hint style="info" %}
Tip: [use your GPU in Chrome](https://segmentsai.notion.site/How-to-use-your-GPU-in-Chrome-2b95e19fb77c456c87f798013769a98a) to make sure the 3D point cloud interface runs smoothly.
{% endhint %}
# View and navigate in the 3D interface
{% hint style="info" %}
Tip: [use your GPU in Chrome](https://segmentsai.notion.site/How-to-use-your-GPU-in-Chrome-2b95e19fb77c456c87f798013769a98a) to make sure the 3D point cloud interface runs smoothly.
{% endhint %}
## Pan/move the camera
Hold the hotkey (`Ctrl/cmd` by default) and click and drag, or simply click and drag the right mouse button.
### Key panning (only in the perspective view)
Use hotkeys (`w`, `a`, `s`, `d`, `Shift + W`, `Shift + S` by default) to move the camera.
Change the key pan speed in the [3D interface settings](/how-to-annotate/label-3d-point-clouds/3d-interface-settings#change-the-key-pan-speed).
### Panning modes
By default, **map panning** is enabled, which means the camera pans along the ground plane. E.g. when you pan up by clicking and dragging, or press `w`, the camera moves *forwards*.
When **screen panning** is enabled, the camera moves along the screen (i.e. along the camera plane). E.g. when you pan up by clicking and dragging, or press `w`, the camera moves *up*.
You can change the panning mode in the [3D interface settings](/how-to-annotate/label-3d-point-clouds/3d-interface-settings#panning-modes).
## Rotate/orbit the camera in the perspective view
Hold the hotkey (`Shift` by default) and click and drag.
### Change the rotation center point
* The rotation point is initialized at the center of the point cloud.
* If you [zoom to an object](#zoom-to-an-object), the rotation point will move to the center of that object.
* You can manually set the rotation point by middle-clicking on any point of the point cloud. The changed rotation point will then be visible for a short time.
## Zoom in/out
#### Mouse wheel
Scroll using the mouse wheel to zoom in/out.
#### Double-click (only in orthographic views)
* Double-click in the viewer to zoom in.
* Press the hotkey (`Shift` by default) + double-click in the viewer to zoom out.
## Zoom to an object
1. Select the object.
2. Press the hotkey (`t` by default).
## Hide objects
### Hide/show a single object
1. Hover over the object you want to hide/show in the objects sidebar.
2. Press the eye icon.
### Hide/show all unselected instances
Press the hotkey (`h` by default).
## Pan to a bird's eye view in the perspective view
Press the hotkey (`Ctrl/cmd + b` by default).
## Pan to initial camera view
Press the hotkey (`Ctrl/cmd + i` by default).
## Switch the main view to an orthographic top view
Click on the camera icon () in the toolbar on the right, or press the hotkey (`v` by default).
{% hint style="info" %}
This feature is only available in the 3D point cloud cuboid and vector interface.
{% endhint %}
## Change a side view
1. Click the dropdown in the top right corner of the side view.
2. Select the new view you want to change to.
{% hint style="info" %}
This feature is only available in the 3D point cloud segmentation interface.
{% endhint %}
## Crop mode
1. Click on the () icon in the right toolbar. The view switches to ortho mode.
2. Click + drag to draw a 2D bounding box around a subregion of the point cloud.
3. The crop mode is now active and only points within the selected region are shown.
4. Click the () icon again to reset and show the full point cloud again.
## Ruler mode
1. Click on the ruler icon () in the right toolbar, or press the hotkey (`z` by default).
2. Click anywhere in the point cloud to start measuring.
3. When you move your cursor to another point the distance measurement along the ground plane will be shown in the bottom right panel of the sidebar. This measurement is in meters if the scale of your point cloud is true to life.
4. Click again to "lock" the measurement. Both ruler ends can now be moved, including in the vertical direction - via the sideviews available in Layout 2.
5. Click the "Reset ruler" button at the top of the screen to start over or press `Esc` to exit the ruler tool.
## Focus beam
The focus beam tool helps you to locate objects in the 3D point cloud via the calibrated camera images. It renders a beam in the 3D space corresponding to a specific location in an image.
1. Open a specific camera image by clicking on the thumbnail or open the image grid.
2. Move your cursor to a point of interest in the image and press the focus beam hotkey (`Shift + z` by default).
3. Go back to the 3D point cloud view. A focus beam should have appeared.
4. To remove the focus beam, press the hotkey again while not hovering over an image.
# Upload, view, and overlay images
See the [View and navigate in the 3D interface](/how-to-annotate/label-3d-point-clouds/view-and-navigate-in-the-3d-interface) page for basic 3D navigation principles.
## Upload images along with point clouds
Segments.ai allows you to upload images along with point clouds. Additionally, you can also specify the camera calibration parameters, which allows you to view the 3D point cloud and objects on the image. See [Sample formats](/reference/sample-types#camera-image) for information on how to add camera images and calibration parameters to a sample.
## Open/close the image viewer
When you open a sample with images, you can see the thumbnails of the images at the bottom of the main labeling window.
#### Open the image viewer
* Click on the thumbnail of the image
* Or press the number key (`1` - `9`) corresponding to the image. The images are numbered from left to right.
{% hint style="info" %}
The 3D overlay (of objects and the point cloud) is only visible if the camera parameters of the image are specified.
{% endhint %}
#### Close the image viewer
* Click on the thumbnail of the image again
* Or press `Esc`
* Or press the number key (`1` - `9`) corresponding to the image again
## Zoom/pan in the image viewer
### Zoom in/out
To zoom in or out, scroll using the mouse wheel.
### Pan
Hold the hotkey (`Ctrl/cmd` by default) and click and drag.
{% embed url="" %}
## Toggle the point cloud overlay
To toggle the point cloud overlay on and off, press the hotkey (`Ctrl + q` by default) or toggle it in the [3D interface settings](/how-to-annotate/label-3d-point-clouds/3d-interface-settings#toggle-point-cloud-overlay-on-images). Disabling the point cloud overlay allows you to see the camera image in more detail, while enabling it allows you to see if the camera and lidar are well calibrated.
{% embed url="" %}
## Jump to the corresponding 3D location
When you've opened an image by clicking a thumbnail, double click anywhere in the image to jump to the corresponding location in 3D space. A 3D "ray" from the camera to that location is briefly displayed.
Note that this feature only works well if the lidar and camera are well calibrated.
## Use the **synchronized** image viewer
When one or more **calibrated** images are available, a synchronized image viewer appears in the bottom right of the screen. When hovering in the perspective or top view this viewer automatically zooms to the location of the pointer in the closest camera image.
{% embed url="" %}
## View the active cuboid camera images in the synchronized viewer
When a cuboid is selected, the synchronized image viewer shows the active cuboid in every camera image where it appears.
To switch between the camera images, press the arrow buttons on the left/right of the synchronized image viewer, or press a rectangle at the bottom of the synchronized image viewer.
You can also zoom/pan in the synchronized image viewer in the same way as in the image viewer (see [above](#zoom-pan-in-the-image-viewer)).
## View the image grid in a new tab
When adding images to a sample, you have to specify a row and column for each image. This information is used to display a grid of your images. You can view this grid in a separate browser tab (which you could move to a second screen).
* To open the image grid in a new tab, press the "Open grid" rectangle next to the camera images at the bottom of the main viewer.\
You can interact with the images in the grid as follows:
* To zoom in/out on an image, scroll using the mousewheel or double-click on the image.
* To pan an image, click and drag on the image.
* To open the image viewer in the main labeling window, click the camera icon in the top right corner of the image.
# 3D interface settings
## Open the 3D interface settings
Click on the "Settings" tab in the sidebar on the right to open the 3D interface settings.
## View and customize hotkeys
Press the "Hotkeys" button to open the hotkey settings. See [Customize hotkeys](/how-to-annotate/customize-hotkeys) for more information on how to customize the hotkeys.
## Interface layout presets
The interface layout presets allow you to change the layout of the point cloud and image viewers. The preset you picked will be remembered within the dataset type.
### Description of the layout presets
1. This layout contains a large main viewer and below it a top view and a synced image viewer (when calibrated images are present). When selecting a cuboid or vector, the top view splits into a side, top, and back view which are placed in a row. *Default for cuboid datasets.*
2. This layout contains a large main viewer and below it a top view, back view, and a synced image viewer (when calibrated images are present). In segmentation datasets, the top and back view can be switched to other perspectives.
3. This layout contains a large main viewer and a synced image viewer in the bottom right (when calibrated images are present). *Default for vector and segmentation datasets.*
4. This layout contains a large main viewer, a top view in the bottom left, and a synced image viewer in the bottom right (when calibrated images are present).
5. This layout contains a large main viewer and right of it a top view and a synced image viewer (when calibrated images are present). When selecting a cuboid or vector, the top view splits into a side, top, and back view which are placed in a column.
## Object coloring mode
The object coloring toggle allows you to change how different objects in the sample are displayed.
Click on a mode to change the object coloring mode.
#### **By instance**
Each object is displayed in a different color.
*E.g. two cars each get a different color.*
#### By category
Each category has a color defined in the dataset settings (see [Categories and attributes](/reference/categories-and-attributes)). Every object of the same category is displayed in that category color.
*E.g. two cars are displayed in the same category color (defined in the dataset settings).*
## Point cloud display
### Change the size of the points
Drag the slider to adjust the point size.
### Toggle point cloud overlay on images
If you have uploaded one or more calibrated images along your point cloud, you can check the checkbox to choose whether the point cloud should be overlaid on the image in the image viewers.
See [Upload, view, and overlay images](/how-to-annotate/label-3d-point-clouds/upload-view-and-overlay-images) for more information on images in the 3D interfaces.
### Invert selection point color (segmentation datasets only)
If the default color doesn't contrast enough with the background while selecting points, you can toggle this checkbox to invert the color.
### Gradient coloring
Gradient coloring allows you to view your point cloud more clearly. Gradient coloring can be used to visualize the height of each point, or the intensity value of each point.
#### Enable height coloring
1. Check the "Gradient coloring" checkbox.
2. Choose "Height" in the "Gradient attribute" dropdown.
3. *Optional:* Check the "Set min/max automatically" checkbox to automatically adjust the height range to the bounds of the current point cloud. Or unselect it and set the minimum and maximum height manually.
4. *Optional:* Toggle option to hide points outside of height range.
5. *Optional:* Choose a gradient by clicking on the gradient and selecting one from the list. You can also reverse the gradient by checking the box under "Reverse".
#### Enable intensity coloring
1. Upload a point cloud with intensity values. See [Supported file formats](/reference/sample-types/supported-file-formats).
2. Check the "Gradient coloring" checkbox.
3. Choose "Intensity" in the "Gradient attribute" dropdown.
4. *Optional:* Set the minimum and maximum intensity manually.
5. *Optional:* Toggle option to hide points outside of intensity range.
6. *Optional:* Choose a gradient by clicking on the gradient and selecting one from the list. You can also reverse the gradient by checking the box under "Reverse".
### Change the point budget
{% hint style="warning" %}
This feature is deprecated. It is only available in the 3D point cloud vector interfaces for point clouds that do not use 3D tiling.
{% endhint %}
The point budget controls the maximum number of points that are displayed. If the point cloud contains more points than the point budget, the point cloud will be randomly subsampled to limit the number of points.
1. Click/hover over the dot icon in the toolbar on the right.
2. Enter the maximum amount of points in the point budget input box. If you want to remove the point budget, simply remove the value in the input box.
### Limit the amount of frames around the current frame in merged point cloud mode
With this limiter you can choose to limit the frames around the current selected frame in the merged point cloud mode. If you leave before or after empty, it will take all the frames before or after the current frame. So leaving both fields empty (or unchecking the checkbox) will merge all the point clouds in the sequence. The number of frames that are merged, is shown in the timeline with a red indicator whenever the merged point cloud is active.
1. Check the "Limit merged frames around current frame" checkbox
2. Fill in before and/or after
## Scene helpers
### Follow active object
Check the "Follow active object" checkbox to make your camera follow the active object through the sequence.
### Change the key pan speed
Drag the slider to adjust the key pan speed, i.e. the speed the camera moves when using keyboard panning.
### Toggle calibrated camera helpers
Check the "Show calibrated camera helpers" option to show a camera helper for each calibrated camera in the scene. Click a camera helper to open its sensor's corresponding image view.
Hovering over an image thumbnail will also always highlight the corresponding camera helper in the scene.
### Panning modes
Click on a panning mode to change how the camera pans in the 3D perspective view.
#### Map panning
The camera pans along the ground plane.
*E.g. when you pan up by clicking and dragging, or press `w`, the camera moves forwards.*
#### Screen panning
The camera pans along the screen, i.e. along the camera plane.
*E.g. when you pan up by clicking and dragging, or press `w`, the camera moves up.*
### Square grid
Check the "Show square grid" checkbox to show a helper grid that is fixed to the [ego pose](/reference/sample-types#ego-pose) (tracks position but not rotation).
### Concentric circles grid
Check the "Show concentric circles grid" checkbox to show a circular helper grid that is attached to the [ego pose](/reference/sample-types#ego-pose) (tracks position and rotation).
### Show ROI
Check the "Show ROI" checkbox to show the region of interest. To adjust the shape and the dimensions, go to Dataset Settings > Labeling tab > Enable region of interest. More info [here](https://app.gitbook.com/o/nAEpl593xwZqipzOfJti/s/HczBG7NHgAtEe4ql1rXH/~/changes/341/guides/configure-label-editor/region-of-interest).
## Label display
### Show all cuboids in active track
Check the "Show all cuboids in active track" checkbox to show all cuboids across all frames in the sequence of the active track. On a keyframe, the cuboid box is orange, while on other frames, it is grey.
Click on a cuboid to jump to the frame of that cuboid.
### Show trajectory line of active track
Check the "Show trajectory line in active track" checkbox to show the trajectory line of the active track across all frames in the sequence. The position of the cuboid at a certain frame is visualized as a circle, which is orange when the frame is a keyframe, and grey if it is not.
Click on the circle to jump to the frame where the cuboid is positioned at the circle's position.
### Cuboid opacity
Drag the slider to change the opacity of the cuboids in the view. This will change the opacity for all cuboids in the scene, including the active cuboid.
# 3D point cloud cuboid interface
{% hint style="info" %}
When [creating a dataset](https://sdkdocs.segments.ai/en/latest/client.html#create-a-dataset) through the Python SDK, choose `pointcloud-cuboid` or `pointcloud-cuboid-sequence` as the `task_type` to use this labeling interface
{% endhint %}
{% hint style="info" %}
Tip: [use your GPU in Chrome](https://segmentsai.notion.site/How-to-use-your-GPU-in-Chrome-2b95e19fb77c456c87f798013769a98a) to make sure the 3D point cloud interface runs smoothly.
{% endhint %}
{% embed url="" %}
This video shows you how to label cuboids in 3D point cloud sequences on Segments.ai. Specifically, we'll use the [merged point cloud view](/how-to-annotate/label-3d-point-clouds/merged-point-cloud-view-for-static-objects) to easily label static objects, and [batch mode](/how-to-annotate/label-3d-point-clouds/batch-mode-for-dynamic-objects) to label dynamic objects.
{% endembed %}
## Create a new cuboid
### Create a cuboid with default dimensions
Set default cuboid dimensions while editing each category in **Dataset → Settings → Labeling**.
Alternatively, edit the raw JSON in **Dataset → Settings → Labeling → Raw** (see [Categories and attributes](/reference/categories-and-attributes)).
You can then create a cuboid with the current category's default dimensions by double-clicking.
1. Select the "Create cuboid" tool by clicking on the box icon in the toolbar on the left, or by pressing the hotkey (`b` by default).
2. Double click in the perspective view or in the top view to create a new cuboid. This cuboid will have the default dimensions of the current category. When you change the category, the dimensions of the cuboid will change as well, given that you have not altered the dimensions of the cuboid yet.
### Draw a cuboid with custom dimensions
You can quickly draw a cuboid with an arbitrary rotation using our 3-click cuboid drawing method.
1. Select the "Create cuboid" tool by clicking on the box icon in the toolbar on the left, or by pressing the hotkey (`b` by default).
2. Click in the perspective view or in the top view to set the back corner of the new cuboid.
3. When you move your mouse, the back side of the cuboid will follow your mouse position. Click to set the second corner of the cuboid.
4. Next, you can extend the cuboid by moving your mouse. Click to complete the cuboid. After clicking, the new cuboid is added to the objects sidebar on the right.
### Smart cuboid initializaition
Instead of manually placing a cuboid, you can automatically create one from a point selection:
1. Switch to cuboid mode
2. Hold Alt and drag to draw a selection box over the point cloud
3. On release, the system analyzes the selected points and creates a fitted 3D cuboid with the correct position, dimensions, and rotation
A `"Creating cuboid..."` indicator appears while the cuboid is being computed.
{% hint style="info" %}
This works best when you select a cluster of points belonging to a single object.
{% endhint %}
#### Initialization modes
Three modes are available, configurable in the interface settings:
* **Simple.** Fits the smallest cuboid from the selected points, using the direction you dragged as the cuboid's yaw. Fastest option, but it does not filter out ground points nor handle outliers.
* **Smart (default).** Fits the smallest cuboid from the selected points, trying to predict the orientation of the object. Filters out ground points and outliers before fitting the cuboid. An outlier tolerance slider lets you control how aggressively noise is removed.
* **AI.** Uses a deep learning detection model to analyze the selection and fit the best (i.e. not necessarily the smallest) cuboid. This methods automatically predicts the rotation and filters out noise. Best use when the points depict only part of the object (e.g. only one side of a vehicle).
{% hint style="info" %}
Notice the AI mode is still in beta, it can produce unexpected results.
{% endhint %}
{% embed url="" %}
### Auto-adjust cuboids
Press hotkey `q` to auto-adjust an existing cuboid, to make it fit better to the point cloud points.
{% hint style="info" %}
The auto-adjust feature snaps the cuboid to fit the points inside it, while keeping its orientation as it is. It does not filter out ground points or outliers.
{% endhint %}
{% embed url="" %}
## Remove a cuboid
1. Select the cuboid you want to remove.
2. Press the hotkey (`Backspace` by default) or click the trash icon next to the object in the objects sidebar on the right.
## Select a cuboid
### Using the "Select and edit" tool
1. Select the "Select and edit" tool by clicking on the pointer icon in the toolbar on the left, or by pressing the hotkey (`Esc` by default).
2. Click on the cuboid in any view.
**Side view auto-centering**
* The side views auto-center on the selected annotation
* Panning or rotating a side view automatically disables its auto-centering behavior on the active object. Other views will continue to follow the active object normally.
* A "*Re-cente*r" button appears on the modified view to manually re-enable auto-centering.
* Auto-centering resets automatically on track or frame change
### Using the objects sidebar
1. Click on the object in the objects sidebar to select it.
## Change the dimensions of a cuboid
1. Select the "Select and edit" tool by clicking on the pointer icon in the toolbar on the left, or by pressing the hotkey (`Esc` by default).
2. Select the cuboid you want to change.
3. Change the cuboid's dimensions by using the hotkeys (`i`, `j`, `k`, `l`, `u`, `o` by default) or by using your mouse:
* In the perspective view: click and drag a face of the cuboid.
* In a side view: click and drag one of the edges of the cuboid.
### Auto-adjust cuboid
To automatically refine a cuboid so it fits the enclosed points more tightly:
1. Select the cuboid you want to adjust
2. Press Q
The system collects the point cloud points inside the cuboid and snaps it to a tighter fit — position, dimensions, and rotation are all updated.
{% hint style="info" %}
Useful after manually placing a cuboid or after propagation — press Q to quickly tighten the fit without dragging faces.
{% endhint %}
## Translate a cuboid
1. Select the "Select and edit" tool by clicking on the pointer icon in the toolbar on the left, or by pressing the hotkey (`Esc` by default).
2. Select the cuboid you want to translate.
3. Translate the cuboid by using the hotkeys (`w`, `a`, `s`, `d`, `Shift + W`, `Shift + S` by default) or by using your mouse:
* Click and drag the yellow cube in the middle of the cuboid to translate the cuboid freely.
* Hover over the yellow cube in the middle to see the translation axes. Click and drag on an axis to translate the cuboid along the selected axis.
## Rotate a cuboid
{% hint style="info" %}
By default, only yaw rotation (around the z-axis) is enabled. To enable full 3D rotation, visit the dataset settings, select the "Labeling" tab, tick the "Enable 3D cuboid rotation" checkbox, and press "Save".
{% endhint %}
1. Select the "Select and edit" tool by clicking on the pointer icon in the toolbar on the left, or by pressing the hotkey (`Esc` by default).
2. Select the cuboid you want to rotate.
3. Click and drag the red rotation plane attached to the cuboid to rotate the cuboid or use the hotkeys:
* Rotate 1° counterclockwise (yaw): `f` by default
* Rotate 1° clockwise (yaw): `g` by default
* Rotate 45° counterclockwise (yaw): `Shift + F` by default
* Rotate 45° clockwise (yaw): `Shift + G` by default
## Change the heading of a cuboid
1. Select the "Select and edit" tool by clicking on the pointer icon in the toolbar on the left, or by pressing the hotkey (`Esc` by default).
2. Select the cuboid you want to change.
3. Change the heading of the cuboid (indicated by a red arrow):
* In the perspective view: double-click the face where you want the heading to be.
* In a side view: double-click the edge where you want the heading to be.
## Edit cuboid information
When a cuboid is selected you'll see the "Cuboid info" pane pop up at the bottom right. In here you can directly change the numeric inputs of the cuboid for more fine-grained control. In here it's also possible to propagate a certain value to the next and previous frames by clicking on the arrow buttons next to the input field. These buttons are disabled for properties that are already synced across frames.
Cuboid information pane
## Change the category of a cuboid
{% hint style="info" %}
If the selected cuboid was created with default dimensions, the dimensions of the cuboid will change to the default dimensions of the new category.
{% endhint %}
### Using the category dropdown
1. Select the cuboid.
2. Open the category dropdown by clicking on the category name in the objects sidebar on the right, or by pressing the hotkey (`c` by default).
3. Select a category by clicking on the desired category, or by using the arrow keys and pressing `Enter` to confirm.
### Using the category hotkey
1. Select the cuboid.
2. Press the hotkey of the desired category (`1` - `9`).
## Cycle between cuboids / select the next cuboid
Press the hotkey (`Tab` by default).
## Copy/paste a cuboid
### Copy a cuboid
1. Select the cuboid.
2. Press the copy hotkey of your operating system (`Ctrl/cmd + c`).
### Paste a cuboid
Press the paste hotkey of your operating system (`Ctrl/cmd + v`).
* If you paste a cuboid and the same cuboid already exists (e.g. because you pasted it in the same frame), a **duplicate** cuboid with a new [track ID](/how-to-annotate/label-sequences-of-data/use-track-ids-in-sequences) will be created and its position will be offset from the original cuboid.
* If you paste a cuboid and the cuboid does not exist yet (e.g because you pasted it in another frame which doesn't have a cuboid with that track ID yet), the cuboid will keep its track ID and will not be offset.
### Lock a track
See [Object locking](/how-to-annotate/object-locking)
# 3D point cloud vector interface
{% hint style="info" %}
When [creating a dataset](https://sdkdocs.segments.ai/en/latest/client.html#create-a-dataset) through the Python SDK, choose `pointcloud-vector` or `pointcloud-vector-sequence` as the `task_type` to use this labeling interface
{% endhint %}
{% hint style="info" %}
Tip: [use your GPU in Chrome](https://segmentsai.notion.site/How-to-use-your-GPU-in-Chrome-2b95e19fb77c456c87f798013769a98a) to make sure the 3D point cloud interface runs smoothly.
{% endhint %}
## Create a new vector
You can create three types of vectors: a polygon, polyline, and a keypoint.
### Create a polygon
1. Select the "Create polygon" tool by clicking on the polygon icon in the toolbar on the left, or by pressing the hotkey (`g` by default).
2. Click in the perspective or orthographic view to add a point to the polygon (each click adds a new point).
3. Close the polygon by
1. double clicking (this adds a final point and closes the polygon) or by
2. clicking on the first polygon point (this closes the polygon without adding a final point) or by
3. confirming the object (hotkey `space` by default) - this closes the polygon without adding a final point.
To reopen a completed polygon for continued drawing, double-click any of its points.
### Create a polyline
1. Select the "Create polyline" tool by clicking on the line icon in the toolbar on the left, or by pressing the hotkey (`n` by default).
2. Click in the perspective or orthographic view to add a point to the polyline (each click adds a new point).
3. Close the polyline
1. by double clicking (this adds a final point and finishes the polyline) or by
2. confirming the object (press `space` hotkey by default) - this closes the polyline without adding a final point.
To reopen a completed polyline, double-click its first or last endpoint. Double-clicking a middle point does not reopen the polyline.
{% hint style="info" %}
It's possible to view the direction of a polyline on orthographic top views. To enable this, open the "Settings" tab, go to the "Helper objects" section and activate "Show polyline arrows".
{% endhint %}
#### Connect polylines (Lane Connection tool)
Use the lane connection tool to instantly create a smooth transition between two existing polylines.
1. Select the Create polyline tool from the toolbar.
2. Select the Lane Connection tool button that appears right below it.
3. Click on the first polyline, then click on the second polyline.
A new polyline will automatically be generated following a smooth cubic Bezier curve that connects the two closest endpoints of the selected polylines.
### Create a point
* Select the "Create point" tool by clicking on the point icon in the toolbar on the left, or by pressing the hotkey (`p` by default).
* Click in the viewer to add a point.
## Remove a vector
1. Select the vector you want to remove.
2. Press the hotkey (`Backspace` by default) or click the trash icon next to the object in the objects sidebar on the right.
## Select a vector
### Using the "Select and edit" tool
1. Select the "Select and edit" tool by clicking on the pointer icon in the toolbar on the left, or by pressing the hotkey (`Esc` by default).
2. Click on the vector in any view.
### Using the objects sidebar
1. Click on the object in the objects sidebar to select it.
## Translate a vector
* **Polygon.** Hover over a polygon (the cursor changes to a "move cursor"). Click, hold and drag to move the polygon.
* **Polyline.** Hover over a polyline (do not hover over the middle of a line segment, this adds a preview of a new polyline point - see [#add-a-point-to-a-vector](#add-a-point-to-a-vector "mention")). The cursor changes to a "move cursor". Click, hold and drag to move the polyline.
* **Point.** Hover over a point (the cursor changes to a "move cursor"). Click, hold and drag to move the point.
## Translate a vector point
Hover over a polygon point, a polyline point or a key point. The cursor changes to a "grab cursor". Click, hold and drag to move this point.
## Add a point to a vector
1. Select a polygon or polyline annotation.
2. Press `Shift` and move the mouse close to a line segment.
3. Click to add this point to the polygon or polyline. Note that the new point gets initialized at a height that is linearly interpolated from the height of its two neighboring nodes.
4. Release `Shift` and move the point if necessary.
## Remove a point from a vector
1. Hover over keypoint or a polygon or polyline point.
2. Press `Shift` and click to remove the point.
## Node snapping
When creating or editing a polygon or polyline, hold `Shift + F` to activate node snapping. This snaps the point you add to the nearest handle of an existing vector, within a certain distance threshold.
Points snapped this way share exact coordinates with the original handle, making them co-located. Co-located points stay linked across tracks:
* Dragging a co-located point moves all points at the same coordinates together
* Undo/redo applies atomically across all affected tracks
* Unsnapping: hold `Shift` and click a co-located point to break the link — the point will move independently afterward
## Annotation placement
In the Settings sidebar, go to the "Annotation placement" section to change how the height of newly created annotations (cuboids and polygon/polyline nodes) is determined:
* **Fixed (default\_z)**: all annotations are initialized at a fixed height, determined by the configurable `default_z` parameter which can be provided when uploading a sample (defaults to 0).
* **Snap to point cloud**: the height of new annotations is determined dynamically based on the surrounding point cloud points.
## Change the category of a vector
### Using the category dropdown
1. Select the vector.
2. Open the category dropdown by clicking on the category name in the objects sidebar on the right, or by pressing the hotkey (`c` by default).
3. Select a category by clicking on the desired category, or by using the arrow keys and pressing `Enter` to confirm.
### Using the category hotkey
1. Select the vector.
2. Press the hotkey of the desired category (`1` - `9`).
## Cycle between vectors / select the next vector
Press the hotkey (`Tab` by default).
## Customize the vector visualization
In the Settings sidebar, go to the "Vector" section to change the node size and line width of vector annotations.
### Lock a track
See [Object locking](/how-to-annotate/object-locking)
# 3D point cloud segmentation interface
{% hint style="info" %}
When [creating a dataset](https://sdkdocs.segments.ai/en/latest/client.html#create-a-dataset) through the Python SDK, choose `pointcloud-segmentation` or `pointcloud-segmentation-sequence` as the `task_type` to use this labeling interface.
{% endhint %}
{% hint style="info" %}
Tip: [use your GPU in Chrome](https://segmentsai.notion.site/How-to-use-your-GPU-in-Chrome-2b95e19fb77c456c87f798013769a98a) to make sure the 3D point cloud interface runs smoothly.
{% endhint %}
## Create a new object
1. If another object is currently selected, first confirm the active object.
2. Select some points. Instantly, a new object is created and appears in the objects sidebar.
## Select points
### Select points using the brush tool
1. Select the brush tool by clicking on the brush icon in the toolbar on the left, or by pressing the hotkey (`b` by default).
2. Change the brush size by using the slider in the toolbar on the top left, or use the hotkey (`Ctrl/cmd + scroll` by default).
3. Click or click and drag in any view to select the points within the radius of the brush.
### Select points using the polygon tool
1. Select the polygon tool by clicking on the polygon icon in the toolbar on the left, or by pressing the hotkey (`g` by default).
2. Click in one of the views to place the first polygon point and start drawing the polygon.
3. Click to add additional points (connected by a straight line) or click and drag to add a line that follows your mouse path.
4. To close the polygon, either:
* Double-click the last point
* Click on the first point (indicated by a square)
* Press the hotkey (`Enter` by default)
### Select points using the box tool
1. Select the box tool by clicking on the rectangle icon in the toolbar on the left, or by pressing the hotkey ( `m` by default).
2. Click and drag in one of the views to draw a box. When you release your mouse click, the points in the box will be selected.
## Deselect points
1. Activate one of the selection tools.
2. To deselect points, either:
* Click on the eraser icon in the left toolbar, and then use the selection tool. The newly selected points will now be removed from the active object.
* Use the selection tool, and hold the shortcut (`e` by default) to remove points from the active object.
## Remove an object
1. Select the object you want to remove.
2. Press the hotkey (`Backspace` by default) or click the trash icon next to the object in the objects sidebar on the right.
## Select an object
#### Using the "View and select" tool
1. Select the "View and select" tool by clicking on the cursor icon in the toolbar on the left, or use the hotkey (`Esc` by default).
2. Click on a point of the object in any view.
#### Using the objects sidebar
1. Click on the object in the objects sidebar to select it.
## Confirm the active object
Press the hotkey (`Space` by default), or select another object.
## Change the category of a**n object**
#### Using the category dropdown
1. Select the object.
2. Open the category dropdown by clicking on the category name in the objects sidebar on the right, or by pressing the hotkey (`c` by default).
3. Select a category by clicking on the desired category, or by using the arrow keys and pressing `Enter` to confirm.
#### Using the category hotkey
1. Select the object.
2. Press the hotkey of the desired category (`1` - `9`).
## Use the protect mode
When the protect mode is enabled, you cannot select points that are already part of another object.
To toggle the protect mode:
1. Activate one of the selection tools.
2. Click the icon in the toolbar on the left, or use the shortcut (`r` by default) to toggle the protect mode.
## Cycle between objects/ select the next object
Press the hotkey (`Tab` by default).
## Use the limiting cuboid
The limiting cuboid can be used to limit the point cloud to an (oriented) bounding box. Only points in the cuboid can be edited.
The limiting cuboid is shown as a yellow cuboid with red outlines and includes the whole point cloud by default.
Adjust the limiting cuboid to determine which part of the point cloud will be shown.
The resulting point cloud view. Only the visible points can be edited.
### Limit the point cloud to an oriented bounding box
1. Show the limiting cuboid by clicking on the cuboid icon () in the toolbar on the right.
2. Adjust the cuboid:
* Adjust the cuboid dimensions by clicking and dragging a face of the cuboid.
* Rotate the cuboid by clicking and dragging the red rotation plane attached to the cuboid.
* Translate the cuboid freely by clicking and dragging the yellow cube in the middle of the cuboid. \
Hover over the yellow cube in the middle to see the translation axes. Click and drag on an axis to translate the cuboid along the selected axis.
3. Apply the changes by clicking on the cuboid icon again or by selecting any of the segmentation tools in the toolbar on the left.
{% hint style="info" %}
In [interface layout presets 1 and 5](/how-to-annotate/label-3d-point-clouds/3d-interface-settings#description-of-the-layout-presets), you can also adjust the limiting cuboid in the side, top, and back views.
{% endhint %}
### Limit the height of the point cloud
1. Show the limiting cuboid by clicking on the cuboid icon () in the toolbar on the right.
2. Adjust the min/max values in the sidebar on the right.
3. Apply the changes by clicking on the cuboid icon again or by selecting any of the segmentation tools in the toolbar on the left.
### Reset the limiting cuboid
1. Show the limiting cuboid by clicking on the cuboid icon () in the toolbar on the right.
2. Click on the "Reset limiting cuboid" button located at the center-top of the interface.
3. Apply the changes by clicking on the cuboid icon again or by selecting any of the segmentation tools in the toolbar on the left.
### Lock a track
See [Object locking](/how-to-annotate/object-locking)
# Merged point cloud view (for static objects)
When labeling a static object in 3D point cloud sequences, a single frame might not show the object in full detail. It can thus be difficult to assess the full dimensions of an object.
To solve this issue, you can toggle a merged point cloud view.
{% embed url="" %}
This video shows how you can use the merged point cloud view to draw a perfect cuboid around a static object.
{% endembed %}
## Toggle the merged point cloud view
Click on the merged point cloud icon () in the right toolbar to toggle the merged point cloud view on/off.
{% hint style="info" %}
Tip: [use your GPU in Chrome](https://segmentsai.notion.site/How-to-use-your-GPU-in-Chrome-2b95e19fb77c456c87f798013769a98a) to make sure the 3D point cloud interface runs smoothly.
{% endhint %}
# Batch mode (for dynamic objects)
{% hint style="info" %}
Batch mode is only available for 3D cuboid labeling.
{% endhint %}
{% hint style="info" %}
Tip: [use your GPU in Chrome](https://segmentsai.notion.site/How-to-use-your-GPU-in-Chrome-2b95e19fb77c456c87f798013769a98a) to make sure the 3D point cloud interface runs smoothly.
{% endhint %}
When labeling or reviewing dynamic objects in 3D point cloud sequences, working object by object is usually the most efficient way to obtain precise annotations. However, this means navigating through the different frames in a sequence constantly. To make this process more efficient, we developed batch mode. Batch mode shows you an object in multiple frames at the same time. This way, you can easily choose where to make adjustments to label an object, and easily review an object by simply scrolling through the frames.
{% embed url="" %}
This video shows how you can use batch mode to quickly adjust a cuboid.
{% endembed %}
## Toggle the batch mode
1. Select the object you want to adjust/review.
2. Click on the batch mode icon () in the right toolbar to toggle batch mode on/off.
You can also exit batch mode by pressing the hotkey (`Escape` by default).
## Select a frame
Clicking somewhere in a frame to select the frame. If you want to select the previous or next frame, press the back or forward arrow, respectively.
## Zoom in/out
* Double-click a view to zoom in on that view.
* Press the hotkey (`Shift` by default) + double-click a view to zoom out on that view.
## Edit the object
Object editing works exactly in the same way as in the single frame interface, so you can still use your mouse or the hotkeys. See the following pages:
{% content-ref url="/pages/STCGOim5NnMFMZG2yqMP" %}
[3D point cloud cuboid interface](/how-to-annotate/label-3d-point-clouds/3d-point-cloud-cuboid-interface)
{% endcontent-ref %}
## Remove the object
1. Select the frame where you want to remove the object.
2. Press the trash icon on the left of the frame.
## Add the object to an empty frame
1. Select the empty frame where you want to add the object.
2. Press the plus icon on the left of the frame.
## Use keyframes
Keyframes work in the same way as in the single frame interface, see [Use keyframe interpolation](/how-to-annotate/label-sequences-of-data/use-keyframe-interpolation). The only difference is that keyframes are now displayed directly next to the frame.
# Smart cuboid propagation
Smart cuboid propagation automatically propagates a cuboid frame by frame. This can help you save time when annotating dynamic objects that do not move linearly. For linear movement, simply use [keyframe interpolation](/how-to-annotate/label-sequences-of-data/use-keyframe-interpolation).
## Automatically propagate a cuboid
1. Select the cuboid you want to propagate.
2. Open the [batch mode](/how-to-annotate/label-3d-point-clouds/batch-mode-for-dynamic-objects) by clicking on the batch mode icon () in the right toolbar.
3. Label the cuboid in the first frame(s) where the object is visible. Label the object in two consecutive frames to improve the smart propagation results.
4. Select the last labeled frame.
5. Press the "step forward" button () to propagate the cuboid forward one frame, or press the "forward" button () to propagate the cuboid forward until the end of the sequence.
6. If you see the propagation going wrong, you can press the stop button () to stop the propagation. You can then correct the label and continue the propagation again.
{% hint style="warning" %}
**Limitations**
* Objects need to contain enough points (\~ hundreds of points) to be propagated reliably.
* Objects that move erratically might not be tracked accurately. Specifically, their heading might move erratically as well.
* You cannot switch browser tabs while smart cuboid propagation is propagating the object
{% endhint %}
# Tips for labeling cuboid sequences
This page describes some tips and tricks to label cuboids in point cloud sequences faster.
## General tips
### Use your GPU in Chrome for better performance
[Use your GPU in Chrome](https://segmentsai.notion.site/How-to-use-your-GPU-in-Chrome-2b95e19fb77c456c87f798013769a98a) to make sure the 3D point cloud interface runs smoothly.
### Label object by object
By labeling object by object, we mean first labeling an object in the whole sequence, before moving on to the next object. This is typically easier and faster than labeling all objects on the first frame, then all objects on the second frame, and so on.
### Use hotkeys for fine adjustments
Your mouse is perfectly suitable for quickly drawing a cuboid or for moving it to roughly the right position. However, for fine adjustments, you’re probably better off using your keyboard. Segments.ai has hotkeys for almost everything, including for [moving a cuboid](/how-to-annotate/label-3d-point-clouds/3d-point-cloud-cuboid-interface#translate-a-cuboid) and for [changing its dimensions](/how-to-annotate/label-3d-point-clouds/3d-point-cloud-cuboid-interface#change-the-dimensions-of-a-cuboid). Better still, you can customize all the hotkeys to your liking:
{% content-ref url="/pages/OzaK9t5jhQAdX9Jt2j39" %}
[Customize hotkeys](/how-to-annotate/customize-hotkeys)
{% endcontent-ref %}
### Learn about keyframes and remove-keyframes
[Keyframe interpolation](/background/sequences#keyframes-and-remove-keyframes) is enabled by default on point cloud sequences with cuboid annotations. It allows you to label a sequence perfectly by only labeling a few “key” frames, while the rest of the frames are completed automatically. On Segments.ai, there are also remove-keyframes that indicate when an object disappears. More precisely, a remove-keyframe means that the object is not present in that frame and all frames before the next normal keyframe.
You can learn about adding, moving, and removing keyframes on this page:
{% content-ref url="/pages/t0Fgqnxp0HukTUNTWf3p" %}
[Use keyframe interpolation](/how-to-annotate/label-sequences-of-data/use-keyframe-interpolation)
{% endcontent-ref %}
An example of keyframes (blue diamonds) and a remove-keyframe (grey circle with cross).
## How to find objects
In order to find objects to label, you can use the main point cloud viewer. You can move through the interface either by using your mouse or the hotkeys (`w`, `a`, `s`, `d` by default). Learn more about navigating the 3D scene on the following page:
{% content-ref url="/pages/H6xRAfMbQoZ2XZptIhtt" %}
[View and navigate in the 3D interface](/how-to-annotate/label-3d-point-clouds/view-and-navigate-in-the-3d-interface)
{% endcontent-ref %}
If the point cloud sequence has images associated with it, they can often be helpful for locating objects that might otherwise be hard to spot in the point clouds. Learn more about using images in the 3D interface on the following page:
{% content-ref url="/pages/3qyBrOtYnqZoxYNKfln9" %}
[Upload, view, and overlay images](/how-to-annotate/label-3d-point-clouds/upload-view-and-overlay-images)
{% endcontent-ref %}
## How to label a static object
1. Go to the first frame where the static object is visible.
2. [Toggle the merged point cloud](/how-to-annotate/label-3d-point-clouds/merged-point-cloud-view-for-static-objects#toggle-the-merged-point-cloud-view) view if it is available.
3. [Draw a cuboid](/how-to-annotate/label-3d-point-clouds/3d-point-cloud-cuboid-interface#create-a-new-cuboid) around the object.
4. Disable the merged point cloud view.
5. Go through the sequence to find the frame where the object disappears.
Tip: you can use the [batch mode](/how-to-annotate/label-3d-point-clouds/batch-mode-for-dynamic-objects) for navigating through a sequence faster.
6. [Remove the cuboid](/how-to-annotate/label-3d-point-clouds/3d-point-cloud-cuboid-interface#remove-a-cuboid) in the frame where the object disappears.
### Use an `is_static` attribute
Add an `is_static` track-level attribute to the categories that can be static (see [Dataset configuration](/guides/configure-label-editor)). When this attribute is checked, you can edit the position/dimensions/rotation of a cuboid in one frame, and these changes will be applied to all other frames automatically.
Use the merged point cloud view to see the full dimensions of a static object.
## How to label a dynamic object
1. Go to the first frame where the static object is visible.
2. [Draw a cuboid](/how-to-annotate/label-3d-point-clouds/3d-point-cloud-cuboid-interface#create-a-new-cuboid) around the object.
3. Go through the sequence to find the last frame where the object is visible.
4. [Move the cuboid](/how-to-annotate/label-3d-point-clouds/3d-point-cloud-cuboid-interface#translate-a-cuboid) to the correct position.
5. [Remove the cuboid](/how-to-annotate/label-3d-point-clouds/3d-point-cloud-cuboid-interface#remove-a-cuboid) in the frame where the object disappears.
6. [Open the batch mode](/how-to-annotate/label-3d-point-clouds/batch-mode-for-dynamic-objects#switch-to-the-batch-mode).
7. Adjust the object in the frames where the cuboid’s position is the farthest away from the real object’s position. Keyframe interpolation will interpolate the object linearly in the frames in between.\
If the object does not move linearly, use [smart propagation](/how-to-annotate/label-3d-point-clouds/smart-cuboid-propagation) to propagate the cuboid automatically.
8. When the sequence is roughly labeled, go through the sequence by using the arrow keys and make fine adjustments using keyboard hotkeys.
9. [Close the batch mode](/how-to-annotate/label-3d-point-clouds/batch-mode-for-dynamic-objects#exit-batch-mode).
Use batch mode to quickly adjust a cuboid in a point cloud sequence.
## How to label a static object that starts moving
Labeling a static object that starts moving is basically the same as labeling a dynamic object. To label the static part:
1. Place a keyframe on the first frame where the object is static.
2. Add a keyframe to the last frame where the object is static ([by double-clicking under the frame](/how-to-annotate/label-sequences-of-data/use-keyframe-interpolation#in-a-different-frame)).
This way, you make sure that the object remains static between the keyframes.
# Label sequences of data
Segments.ai allows you to upload [a sequence](/background/sequences) of data as one [sample](/background/main-concepts#sample). This way, a single labeler can efficiently label a whole sequence of data. This workflow also allows you to obtain consistent IDs across frames, which is useful for tracking applications. To speed up labeling, you can use keyframe interpolation.
This section contains how-to guides explaining these features in more detail:
* [Use track IDs in sequences](/how-to-annotate/label-sequences-of-data/use-track-ids-in-sequences)
* [Use keyframe interpolation](/how-to-annotate/label-sequences-of-data/use-keyframe-interpolation)
# Track timeline
The track timeline is the main interface for managing annotations across frames in a sequence. It's always visible at the bottom of the editor.
## Overview
The timeline displays your annotations as horizontal rows (tracks) across time (frames), with each track showing:
* Track ID and category
* Where the object is present (colored bars)
* Keyframe locations (diamond icons)
* Track attributes (when selected)
## Timeline states
The timeline can be displayed in two states:
**Collapsed view:**
* The timeline appears in a compact form by default
* Shows basic track information and frame columns
**Expanded view:**
* Click the **Details** button to expand the timeline
* Reveals additional information and controls
* Provides more space for viewing tracks and attributes
{% hint style="info" %}
Use the expanded view when working intensively with keyframes or attributes. Use the collapsed view to maximize space for the main viewer.
{% endhint %}
## Timeline views
The timeline adapts its display based on your current selection:
### Tracks view
**Access:** No track selected + View dropdown = "Tracks"
Shows all tracks in your sequence with:
* One row per track
* Track ID and category name on the left
* Colored bars showing track visibility
* Keyframes marked with white diamonds (◆)
* Remove-keyframes marked with white crosses (×)
**Use this view to:**
* Get an overview of all objects in the sequence
* Compare timing between different tracks
* Select a track to work on
### Scene attributes view
**Access:** No track selected + View dropdown = "Scene attributes"
Shows scene-level attributes with:
* One row per scene attribute
* Attribute name on the left
* Editable cells for each frame
**Use this view to:**
* Label environmental conditions (weather, lighting, etc.)
* Add sequence-level metadata (recording location, camera settings, etc.)
### Single track view
**Access:** Select any track in the viewer (automatic)
Shows only the selected track with:
* The track row at the top
* Track attribute rows below (if attributes exist)
* View dropdown is disabled
**Use this view to:**
* Focus on labeling a specific object
* Edit track-specific attributes
* Work with keyframes for that track
{% hint style="info" %}
The timeline automatically switches to single track view when you select an object. Deselect the object (click in empty space) to return to the previous view. The track list is synchronized with the sidebar panel.
{% endhint %}
## Navigate between frames
### Navigation controls
The timeline header provides several ways to move between frames:
**Navigation buttons:**
* `◄`: Previous frame
* `►`: Next frame
* `◄◄`: Jump 5 frames backward
* `►►`: Jump 5 frames forward
**Direct selection:**
* Click on any frame column to jump directly to it
**Keyboard shortcuts:**
* Arrow keys: Move frame-by-frame
* Shift + Arrow keys: Jump multiple frames
**Visual feedback:**
* Active frame: Highlighted vertical column
* Hovered frame: Gray highlight when you hover over a frame
## Create tracks
When you add a new object in the viewer:
1. A track is automatically created with a unique track ID
2. A new row appears in the timeline (Tracks view)
3. A keyframe is created at the current frame
4. The track bar displays the track's visibility range using the same color as the object in the viewer
{% hint style="info" %}
Track IDs increment sequentially, starting from the highest existing track ID.
{% endhint %}
## Manage tracks
Right-click on any track row to open the context menu with these options:
### Update track ID
Change the track's ID across all frames. Useful for:
* Correcting incorrectly assigned IDs
* Aligning with external tracking data
See Use track IDs - Update a track ID for details.
### Merge with track
Combine two tracks into one. Useful for:
* Fixing tracking errors where one object was assigned multiple IDs
* Correcting splits that shouldn't have happened
{% hint style="warning" %}
Merging is only possible when both tracks have no overlapping keyframes.
{% endhint %}
See Use track IDs - Merge tracks for details.
### Hide and lock tracks
Click the **eye icon** on a track row to hide it — the track remains saved but invisible in the viewer.
Click the **lock icon** on a track row (or via the right-click context menu) to lock it. A locked track cannot be moved, resized, edited, or have its category changed. The cursor changes to a lock icon when hovering over a locked object in the viewer.
Use **Lock all / Unlock all** in the global timeline menu \[⋮] to lock or unlock all tracks at once.
Locking is local and temporary — not saved to the server, resets on reload.
Shortcut: `Shift + R` — toggle lock on the active track.
### Remove track
Delete the entire track and all its annotations. This action:
* Removes all keyframes
* Deletes all attribute values
* Can be undone via undo history
{% hint style="info" %}
When a track is removed, the highest track ID becomes available for new objects.
{% endhint %}
### Split track
Divide a track into two separate tracks at the current frame. Useful for:
* Separating objects that were tracked as one
* Handling scenarios where one object becomes two
{% hint style="warning" %}
Splitting is only possible if there are keyframes before the current frame.
{% endhint %}
{% hint style="info" %}
Split track is only available in single-sensor sequences.
{% endhint %}
See Use track IDs - Split a track for details.
## Work with keyframes
### View keyframes
Keyframes appear as **white diamond icons (◆)** on the track bar. Each diamond marks a frame where you've defined the object's state.
Remove-keyframes appear as **white cross icons (×)** on blank spaces, marking frames where you've explicitly removed the object.
### Keyframe operations
The timeline supports several keyframe operations:
**Select keyframes:**
* Click on a keyframe to select it
* The viewer jumps to that frame automatically
**Move keyframes:**
* Drag keyframe diamonds to different frames
* Interpolation recalculates automatically
* Movement is constrained to segments where the object exists (within the colored bar)
**Delete keyframes:**
* Select a keyframe and press `Backspace`
* The keyframe is removed and interpolation adjusts
**Copy and paste keyframes:**
* Copy a keyframe's state and paste it to another frame
* Useful for duplicating complex object states
{% hint style="info" %}
Keyframe movement is constrained to preserve data integrity. You can only move keyframes within the track's visibility range.
{% endhint %}
{% hint style="info" %}
For detailed keyframe operations, see Use keyframe interpolation.
{% endhint %}
## Edit attributes in timeline
The timeline allows direct editing of attribute values without leaving the labeling interface.
### Track attributes
**When to see them:** Select a track
Track attribute rows appear below the track row, showing:
* Attribute name on the left
* One cell per frame (or one cell for sequence-level attributes)
* Cell type matches the attribute type (text, number, dropdown, checkbox, etc.)
**How to edit:**
1. Click on any cell
2. Enter or select the value
3. The value is saved automatically
**Synced across frames:**
* If enabled: Editing one cell updates all frames
* If disabled: Each frame can have a different value
### Scene attributes
**When to see them:** Deselect all tracks + View dropdown = "Scene attributes"
Scene attribute rows appear, showing:
* Attribute name on the left
* One cell per frame (frame-level) or one cell total (sequence-level)
**How to edit:** Same as track attributes — click and edit.
{% hint style="info" %}
For configuring attributes (synced vs not synced, frame-level vs sequence-level), see Configure the label editor.
{% endhint %}
## Timeline layout
### Sticky elements
As you scroll through the timeline:
* **Name column** stays visible on the left, so you always see track/attribute names
* **Navigation header** stays visible at the top, so controls are always accessible
### Scrolling
* **Horizontal scrolling:** View frames beyond the visible area (useful for long sequences)
* **Vertical scrolling:** View tracks beyond the visible area (only in all tracks view)
### Active frame indicator
The active frame is marked with a highlighted vertical column spanning the entire timeline height. This makes it easy to see your current position, especially when scrolling.
# Use track IDs in sequences
## Use track IDs in sequences
A track ID uniquely identifies an object across multiple frames in a sequence. Track IDs are essential for tracking applications where you need to follow the same object over time.
## Understanding track IDs
When you create an object in a sequence:
* A unique track ID is automatically assigned
* The track ID remains constant across all frames where the object exists
* Track IDs increment sequentially, starting from the highest existing ID
Track IDs are visible:
* In the **sidebar** next to the object category
* In the **timeline** at the left of each track row
{% hint style="info" %}
The timeline interface is described in detail in Track timeline.
{% endhint %}
## Update a track ID
Change a track ID across all frames:
1. In the timeline, right-click on the track row
2. Select **Update track ID**
3. Enter the new track ID in the dialog
4. Click **Update**
{% hint style="info" %}
Track IDs must be unique within the sequence. If you enter an existing ID, you'll be prompted to merge the tracks instead.
{% endhint %}
**Use case:** Fix incorrectly assigned track IDs or align with external tracking data.
## Merge tracks
Combine two tracks that represent the same object:
1. Right-click on the source track row in the timeline
2. Select **Merge with track**
3. Choose the target track ID from the dropdown
4. Confirm the merge
All keyframes from the source track are transferred to the target track, and the source track is removed.
{% hint style="warning" %}
**Important constraint:** Merging tracks is only possible when both tracks have no overlapping keyframes.
{% endhint %}
**Use case:** Correct tracking errors where the same object was accidentally assigned multiple track IDs.
## Split a track
Divide a track into two separate tracks at a specific frame:
1. Navigate to the frame where you want to split
2. Right-click on the track row in the timeline
3. Select **Split track**
The track is divided:
* Frames before the split point keep the original track ID
* Frames from the split point onward receive a new track ID
{% hint style="warning" %}
**Limitation:** Splitting a track is only possible if there are keyframes before the current frame.
{% endhint %}
{% hint style="info" %}
The **Split track** action is only available in single-sensor sequences.
{% endhint %}
**Use case:** Separate objects that were incorrectly tracked as one, or handle scenarios where one object becomes two (e.g., a vehicle splitting into two paths).
## Remove a track
Delete an entire track from the sequence:
1. Right-click on the track row in the timeline
2. Select **Remove track**
3. Confirm the deletion
{% hint style="warning" %}
This deletes all keyframes and annotations associated with the track. Use undo if you need to recover it.
{% endhint %}
{% hint style="info" %}
When a track is removed, the highest track ID becomes available for new objects. Lower IDs require manual adjustment if you want to reuse them.
{% endhint %}
## Same-dimensions constraint
For 3D cuboid sequences, you can enforce consistent object dimensions across all frames:
1. Go to **Settings** > **Labeling**
2. Enable the **Same-dimensions track constraint** checkbox
When enabled, all keyframes in a track will maintain the same object dimensions (width, height, depth), allowing only position and rotation changes between frames.
{% hint style="info" %}
**Note:** The same-dimensions track constraint is only available for cuboid sequences.
{% endhint %}
**Use case:** Label objects with known fixed dimensions (e.g., standard shipping containers, vehicles with known specifications).
{% hint style="info" %}
For more details on the timeline context menu and interface, see Track timeline.
{% endhint %}
# Use keyframe interpolation
Keyframe interpolation is the process of creating intermediate values between a set of keyframes. In Segments.ai, this technique applies to object labels that change across video frames, such as a moving car's bounding box. Users set labels at key moments, and the system automatically generates intermediate values.
{% hint style="info" %}
Keyframe interpolation is available for 2D bounding box labels, and for 3D cuboid labels.
{% endhint %}
## Understanding keyframes
A **keyframe** marks a frame where you've explicitly defined an object's state (position, size, rotation, etc.). Between keyframes, the system automatically interpolates the object's properties.
A **remove-keyframe** marks a frame where you've explicitly removed an object. It indicates the object is absent from that point until the next regular keyframe.
{% hint style="info" %}
Keyframes are managed through the timeline interface at the bottom of the editor. See Track timeline for details on the timeline interface.
{% endhint %}
## Object attributes and interpolation
{% hint style="warning" %}
Object-level attribute edits create new keyframes, with values copying between them. To modify attributes on objects with existing keyframes, you must update the value across **all necessary keyframes**.
{% endhint %}
## View the keyframes of an object
The **timeline** is always visible at the bottom of the editor and displays tracks across frames.
### Timeline views
The timeline content changes based on your selection and the active view:
**No track selected:**
* Use the **View dropdown** in the timeline header to switch between:
* **Tracks view**: Displays all tracks with their visibility across frames
* **Scene attributes view**: Displays scene-level attributes
**Track selected:**
* The timeline automatically shows the selected track with its track attributes below
### Keyframe indicators
Select a track in the viewer to see its keyframes in the timeline:
* **White diamond icons (◆)**: Mark frames with keyframes
* **White cross icons (×)**: Mark remove-keyframes
* **Colored bar**: Shows where the track is visible
{% hint style="info" %}
When you select a track in the viewer, the timeline automatically switches to show only that track with its attributes. The View dropdown is disabled while a track is selected.
{% endhint %}
## Add a keyframe
Keyframes are created automatically when you modify an object:
1. Select a track in the viewer
2. Navigate to a frame using the timeline navigation buttons or arrow keys
3. Modify the object (move, resize, rotate, change shape)
4. A keyframe is automatically created at that frame
{% hint style="info" %}
You don't need to manually create keyframes — they're added whenever you make changes to an object.
{% endhint %}
## Move a keyframe
Drag keyframes to different frames in the timeline:
1. Select a track in the viewer (timeline switches to show that track)
2. In the timeline, click and hold on a keyframe diamond (◆)
3. Drag it to the desired frame
4. Release to confirm — interpolation is automatically recalculated
{% hint style="info" %}
Keyframes can only be moved within the track's visibility range (where the colored bar appears).
{% endhint %}
## Remove a keyframe
Delete a keyframe to simplify your annotation:
1. Select a track in the viewer
2. Navigate to a frame with a keyframe
3. Press `Backspace`
The keyframe is removed and interpolation is recalculated based on the remaining keyframes.
{% hint style="warning" %}
A track must have at least one keyframe. When there exists only one keyframe, this keyframe cannot be removed.
{% endhint %}
## View the deleted-keyframes of an object
Remove-keyframes appear as white cross icons (×) on blank spaces in the timeline. They indicate frames where you've explicitly removed the object.
## Add a deleted-keyframe
Create a remove-keyframe to mark where an object disappears:
1. Select a track in the viewer
2. Navigate to the frame where the object should disappear
3. Press `Backspace`
The object is removed from that frame onward until the next keyframe.
## Move a deleted-keyframe
Drag remove-keyframes just like regular keyframes:
1. Select a track in the viewer
2. In the timeline, click and hold on a remove-keyframe cross (×)
3. Drag it to the desired frame
4. Release to confirm
## Remove a deleted-keyframe
Delete a remove-keyframe to restore object visibility:
1. Select a track in the viewer
2. Navigate to a frame with a remove-keyframe
3. Press `Backspace`
The object becomes visible again from that frame onward.
## Label occluded objects
Two approaches exist for objects that disappear and reappear:
1. **Adjust on reappearance:** Adjust the object on its reappearance frame (creating a keyframe), then remove it where occlusion begins.
2. **Remove and merge:** Remove the object where occlusion starts, label a new object where it reappears, then merge them by matching their track IDs. See Use track IDs - Merge tracks for details.
{% hint style="info" %}
For more information on managing tracks and navigating the timeline, see Track timeline.
{% endhint %}
# Playback
In some cases it can be useful to play through a sequence to help visualize how the scene and the annotations change. This playback can be triggered using the *Play* button in the timeline.
A prerequisite for a smooth playback is that all assets are available and loaded. To facilitate this, the play functionality will run a [prefetch](/reference/sample-and-label-types) of the needed assets for each frame.
Playback speed can be modified from the settings sidebar.
# Prefetch
When using the labeling interface, only the necessary assets (e.g. point cloud files, image files) are fetched matching the current active frame. In many cases, when working on a sample extensively, many frames will be accessed eventually.
To improve the experience and reduce waiting time for fetching and loading assets, the assets can be prefetched proactively. (In multisensor datasets, prefetching happens on a per sensor level.)
Prefetching can be done by clicking the prefetch button in the timeline.
In the settings in the sidebar, the prefetching can be configured to activate automatically on sample load (and on sensor switch in multisensor datasets).
Prefetched assets are [cached on disk](/guides/caching-assets) so that repeated access to frequently used assets loads faster.
# Label multi-sensor data
Multi-sensor labeling interface combines 3D point cloud data and 2D image data in the same task, to make it easier to annotate and categorize the point clouds.
This section contains additional information for using the multi-sensor labeling interfaces.
# 3D to 2D Projection
For multi-sensor labeling, utilizing the ‘Project cuboids to camera sensors’ feature will automatically project 2D bounding boxes on the camera sensor frames from the 3D cuboids in the lidar point cloud. The projected 2D bounding boxes will maintain the object’s track ID throughout all sensors. This will help speed up the labeling for the 2D camera frames.
{% hint style="info" %}
The sensors list has been moved to a dropdown in the top part of the right sidebar.
{% endhint %}
Open the sensors dropdown to access ‘Project cuboids to camera sensors’.
### **Overwrite existing tracks**
* If this option is checked, it will use the 3D cuboids to overwrite existing 2D bounding box objects.
* If you made adjustments to the 2D bounding boxes and want to project additional objects from 3D to 2D, make sure this option is unchecked.
# Sensors list
For multi-sensor datasets, the sensors list can be opened and closed with the backtick key (\`) located to the left of the "1" key on a standard US or UK keyboard. This dropdown now also reflects the presence of the active object on a sensor. If the active object is present, a filled circle appears before the sensor name; an empty circle means that the object is not present. \
\
Additionally, hotkeys for navigating through the sensors are displayed on the right side of the sensor name. For example, pressing key **7** will change the view to **Cam back right**. These hotkeys now extend to the second row of the keyboard rather than only the digits. If there are too many sensors, it will not display a hotkey for it.
# Annotate object links
With the object linking feature you can annotate *relations* or *links* between objects in the scene. These links can also contain *link attributes*.
The format of the link annotations in the exported label attributes is documented [here](/reference/label-types#links).
## Enable the object linking feature
Navigate to the dataset settings and go to the "Labeling" section. At the bottom of the page, check the option "Use object linking". If the option is not present, it means that the dataset is not compatible yet (see limitations above).
#### Enable category restrictions
With the "Enable category restrictions" flag disabled, links can be created between any two objects. With the flag enabled (default), links can only be created between whitelisted categories. These categories need to be configured in the categories editor.
## Configure link attributes and category restrictions
Link attributes and category restrictions can be configured in the category editor.
#### Category restrictions
Configure which categories can be linked to the current category by adding them in the "Allow linking with these categories" input field. If a category can be linked to itself, it should also be added in the list. Since links have a direction, pay attention to which categories are added to each configuration.
#### Link attributes
Once the feature is activated, it's possible (but not required) to configure link attributes for each category.
Note that a link has a direction, *from* one object *to* another one. The attributes that will appear when annotating a link from one object to another one are determined by the *from* category.
Link attributes can also be configured [programmatically](/reference/categories-and-attributes#category).
## Link two objects and edit link attributes
The tool bar has a new button to activate the object linking tool.
The toolbar with the object linking tool activated
The below video shows how to link two objects, in this case a traffic light and a lane:
1. Select an object
2. Activate the object linking tool
3. Click on the other object you want to link it to
4. The link is now shown in the right sidebar. You can also set the link attributes here.
{% embed url="" %}
## Unlink two objects
The following video shows how to delete a link:
1. Select the object
2. Select the object linking tool
3. Click on the other object to break the link
Alternatively a link can be deleted by clicking the trash bin icon in the sidebar next to the link.
{% embed url="" %}
# Use the warning system
{% hint style="info" %}
[More info about how to configure rules for a dataset](/guides/configure-label-editor/configure-rules)
{% endhint %}
When in a labeling / reviewing interface, you can open the warnings sidebar from the navigation bar by clicking the warnings icon.
When the warnings sidebar is opened, you can click "Check sample" to verify the annotations with the configured rules. The resulting warnings will be displayed in the overview.
Triggering the check to run again can be done using the button at the bottom of the sidebar.
In the warning list, you can see an overview of the potential issues.
Individual warnings can be dismissed by clicking the 'x' icon on the top right of a warning. Re-running the check will re-add any dismissed warnings if they are still present in the sample.
To quickly jump to the context of a warning, you can click on the last pill button (e.g. "Track 1 & 2" in the image above). This will update the current sensor, frame and track to match the warning.
To facilitate going through the set of warnings and treating them in an organised manner, you can filter the list to only display warnings matching the current sensor / frame / active object, or filter to display only a specific warning type.
In addition to this, you can also group the warnings by type, sensor, or track. The grouping can be combined with an active filter to fully tailor the warning list to your needs.
## Save flow
The save / submit / review flow now allows you to run the check before saving.
If warnings are found, a prompt will ask you whether you want to proceed anyway, or if you want to check the warnings by opening the sidebar.
# Customize hotkeys
To speed up labeling, you can use the hotkeys/shortcuts built into our labeling interfaces. You can also customize the hotkeys to your liking.
The hotkey menu of the 3D pointcloud vector interface.
### Open the hotkey menu
1. Open a labeling interface.
2. Press the keyboard icon () in the floating toolbar on the right.
### Change a hotkey
1. Open the hotkey menu.
2. Click in the black box of the hotkey you want to change. Greyed-out hotkeys cannot be edited.
3. Enter the new key combination you want to use as the hotkey.
4. Press the "Save" button at the top right of the hotkey menu.
{% hint style="warning" %}
Duplicate assignments may cause the labeling interface to behave unexpectedly.
{% endhint %}
### Delete a hotkey
1. Open the hotkey menu.
2. Click the trash can icon next to the hotkey.
3. Press the "Save" button at the top right of the hotkey menu.
### Choose a hotkey preset
1. Open the hotkey menu.
2. Click the "Preset" dropdown at the top left of the hotkey menu.
3. Select the hotkey preset you want to use.
### Reset the hotkeys
1. Open the hotkey menu.
2. Press the "Reset" button at the top right of the hotkey menu.
{% hint style="danger" %}
Resetting the hotkeys removes your changes for a certain preset. This cannot be undone.
{% endhint %}
# Object filtering
Across all task types you can use the filter box in the sidebar to isolate specific annotations based on their *category* and/or *attributes* (e.g., showing only *stationary vehicles*). It also allows for batch hide/show all filtered annotations simultaneously with a single click to instantly clear background clutter.
The filter is located in the right sidebar, under **Objects** or **Tracks** section (for sequence datasets).
The active filter state is automatically appended to the browser's URL (e.g., `?object-filter=[{"type":"category","value":"human.pedestrian.adult"},{"type":"visibility","value":"visible"}]`). This enables you to copy and share the URL so teammates or QA reviewers land on the exact same filtered view.
# Object locking
Lock an object to prevent any modification (move, resize, category change, delete). The cursor changes to a lock icon when hovering over a locked object.
* `Shift + R` - toggle lock on the active object
* Or click the lock icon in the right sidebar
Note that
* Locking is **local**, meaning others will not see any locks that you've placed on objects
* Locking is **temporary**, meaning locks are not saved and are reset on reload
Locking can be used for accuracy purposes to avoid unintended modification, or for structured QA processes.
If you want all objects to be locked, you might benefit more from the [read-only](/guides/open-a-sample-in-read-only-mode) mode.
# Add collaborators to a dataset
Go to **Settings -> Collaborators** to add collaborators to your dataset.
### Permissions
Collaborators can have one of 4 roles: viewer, labeler, reviewer, manager and administrator. Each role has a different set of permissions:
Viewer
Labeler
Reviewer
Manager
Administrator
View samples
true
true
true
true
true
Label samples
false
true
true
true
true
Review samples
false
false
true
true
true
Delete and resolve issues
false
false
false
true
true
Add, update, delete collaborators
false
false
false
true
true
Edit sample priority and assigned labeler/reviewer
false
false
false
true
true
Add and delete samples
false
false
false
false
true
View, add, delete releases
false
false
false
false
true
Manage settings
false
false
false
false
true
Viewers can only open a sample in read-only mode with all editing actions disabled, except the creation of issues. On the dataset level, they can see the "Overview", "Samples" and "Issues" tabs.
Labelers and reviewers can only see the "Overview" tab of a dataset, where they can click the "Start labeling" and "Start reviewing" buttons depending on their role. Note that they cannot browse the samples on the Samples tab.
Managers and administrators can add collaborators to a dataset. Note that a manager cannot add a collaborator with an administrator role.
### Adding collaborators programmatically
You can also [add collaborators to a dataset using the Python SDK](https://sdkdocs.segments.ai/en/latest/client.html#add-a-dataset-collaborator).
# Create an organization
Organizations are shared accounts where teams can easily collaborate across many datasets.
Each person that uses Segments signs into a personal account. Multiple personal accounts can collaborate on shared datasets by joining the same organization account, which owns the datasets. Administrators can manage memberships and create datasets within an organization.
## Create an organization
To create a new organization, click the plus-icon in the top navigation bar and select “New organization”. Choose a username for your organization in the next screen.
When clicking the “Create organization” button, you will be taken to the organization account page.
Each member of an organization can quickly access it via the profile button in the top navigation bar.
## Create a dataset within an organization
{% hint style="info" %}
Only organization administrators can create new datasets within the organization
{% endhint %}
To create a new dataset within an organization, go to the organization account page and click the “New dataset” button. In the popup window, you will see that the organization is selected as the owner of the dataset.
Alternatively you can create a new dataset by clicking the plus-icon in the top navigation bar, selecting “New dataset”, and changing the owner from your personal account to your organization account manually.
## Manage organization members
As an administrator, you can add new members to the organization and assign them a role.
From the organization account page, go to the “Settings” tab and click the “Members” subtab. Type the username of the user you want to add, and click “Add member”.
For each member, you can assign an “Organization role” and a “Default dataset role”.
### Organization role
The organization role determines the role of a user within the organization itself.
There are only two organization roles, *member* or *administrator*. Their permissions are as follows.
Member
Administrator
Access the organization account and see other members
true
true
Create new organization datasets
false
true
Manage memberships and settings
false
true
### Default dataset role
When creating a new dataset within an organization, all organization members will be added as collaborators to these datasets according to their *default dataset role*. Dataset collaborator roles are described in [Add collaborators to a dataset](/guides/add-collaborators-to-a-dataset).
Choose “None (do not add)” if you don’t want to add a member as a collaborator to new datasets by default.
## Leave an organization
As a member you can leave an organization by going to the *organizations* tab on your own settings page, and clicking the "Leave organization" button.
Removing a member from an organization only revokes the access rights of that user. The user's previous labeling work is not affected, and their username will still be present in the Insights tab and sample history.
# Dataset configuration
After you create a dataset, you can configure its labeling setup and dataset-specific behavior from **Settings**.
Use the pages in this section to:
* [Configure the label editor](/guides/configure-label-editor/configure-the-label-editor)
* [Configure rules](/guides/configure-label-editor/configure-rules)
* [Configure region of interest](/guides/configure-label-editor/configure-region-of-interest)
* [Dataset tags](/guides/configure-label-editor/dataset-tags)
# Configure the label editor
Once you have created a dataset, you can further configure the labeling interface under **Settings** -> **Labeling** -> **Categories:**
Here you can set the color, name, and description for each category.
## Working with attributes
### Object attributes
You can add one or more object-level attributes or *object attributes* to a category by expanding its row.
Attributes come in 4 types:
* Select box
* Multi-select box
* Checkbox
* Text
* Number
You can optionally configure following settings for each object attribute:
* **Default value**: default value of the attribute.
* **Mandatory**: whether the attribute is mandatory. Mandatory attributes raise a warning when not filled in.
* **Synced across frames ("Sequence-level")**: Only in sequence datasets. Whether an attribute should remain constant across all frames for an object with a certain track ID. If false, the attribute can change on each frame.
* **Synced across sensors ("Across sensors")**: Only in multi-sensor datasets. Whether an attribute should remain constant across all sensors for an object with a certain track ID. If false, the attribute can change on each sensor.
* **Sensors**: Only in multi-sensor datasets. Whether an attribute applies to 2D sensors, 3D sensors, or all sensors.
{% hint style="info" %}
**Checkbox default values**
Checkbox attributes support three possible default values:
* **Checked** (`true`) — the checkbox is on by default
* **Unchecked** (`false`) — the checkbox is off by default
* **Unset** (no default) — the checkbox has no value until the labeler explicitly sets it
When a checkbox attribute is **mandatory** and its value is **unset**, a warning will be raised — just like other mandatory attributes that are left empty. This is useful when you want labelers to make a deliberate choice rather than relying on a pre-filled default.
When no default is configured, the checkbox appears in a neutral "unset" state. The labeler can click the left half to set it to unchecked, or the right half to set it to checked.
{% endhint %}
### Scene attributes
You can add scene attributes by clicking the "Edit image attributes" button. Scene attributes are not linked to a specific object, but rather to a frame or the scene as a whole.
### In the labeling interface
When labeling, the object and scene attributes will be shown in the sidebar on the right. Scene attributes are always visible, while object attributes are only shown when an object is selected which has a category with object attributes.
## Timeline display (sequences only)
For sequence datasets, user-defined attributes also appear in the timeline interface at the bottom of the editor, allowing you to view and edit them directly while navigating through frames.
### How attributes appear in the timeline
The timeline displays attributes differently based on whether they have a category:
**Track attributes** (attributes with a category):
* Visible when you select a track in the viewer
* Display as rows below the track row in the timeline
* Each row shows the attribute values across frames
**Scene attributes** (attributes without a category):
* Visible when you select "Scene attributes" from the View dropdown (when no track is selected)
* Display as rows at the top of the timeline
### Attribute behavior in timeline
Attributes can be configured along two independent dimensions:
**Dimension 1: Frame-level vs Sequence-level** (Synced across frames setting)
* **Frame-level** (Synced across frames = disabled): Each frame can have a different value. The timeline displays one cell per frame that you can edit individually.
* **Sequence-level** (Synced across frames = enabled): One value applies to all frames. The timeline displays a single value that applies to the entire sequence.
**Dimension 2: Sensor-specific vs Synced across sensors**
* **Sensor-specific**: Each sensor can have different values. In multi-sensor sequences, each sensor's timeline shows its own attribute values.
* **Synced across sensors**: The same value applies to all sensors. All sensors share the same attribute value.
**Possible combinations:**
These two dimensions create four possible configurations:
1. **Frame-level + Sensor-specific**: Each frame and each sensor has its own value
2. **Frame-level + Synced across sensors**: Each frame has its own value, but all sensors share the same values
3. **Sequence-level + Sensor-specific**: One value per sensor for the entire sequence
4. **Sequence-level + Synced across sensors**: One value for the entire sequence across all sensors
### Benefits of timeline editing
* Edit attribute values without switching between panels
* See how attribute values change across frames at a glance
* Quickly identify frames where specific attribute values are set
* For frame-level attributes, easily compare values across multiple frames
{% hint style="info" %}
For detailed instructions on editing attributes in the timeline, see Track timeline - Edit attributes.
{% endhint %}
### Editing the configuration file directly
If you click on the "Raw" tab, you can see the configuration in JSON format. You can copy-paste this configuration from one dataset to another, or update it programmatically using the [Python SDK](https://sdkdocs.segments.ai/en/latest/client.html). The configuration should adhere to the format defined [here](/reference/categories-and-attributes).
# Configure rules
In the dataset settings, rules can be configured to help spot potential issues early by raising warnings. Currently, the rules that can be added are only applicable to cuboids.
{% hint style="info" %}
[More info about how to use the warning system while labeling / reviewing](/how-to-annotate/use-the-warning-system)
{% endhint %}
## Configuration
Go to your dataset settings into the labeling tab. Scroll down the page to the warnings section (just after the categories section).
Here you can enable the intersecting cuboids rule and / or the cuboid dimension rules.
### Intersecting cuboids
Excluded groups are groups of categories that should not raise a warning when they intersect with each other. A common example is a person on a bicycle.
Excluded categories are a set of categories that the rule should not be applied to at all. Add categories here that should be ignored during the check for intersecting cuboids.
Here's an example of a configuration.
It's also possible to configure an overlap threshold.
### Cuboid dimensions
Configure rules that can warn users when cuboids exceed certain limits. Only fields that are filled in are checked.
This can be applied on a global scale by adding a rule without setting any categories on it.
You can also configure more granular limits for groups of categories.
# Configure region of interest
In 3D point cloud datasets, you can now define a Region of Interest to focus labeling efforts on a limited zone centered around the ego vehicle. The Region of Interest can be configured in the dataset
## Configuration
Go to your dataset settings into the labeling tab. Scroll down the page to the Region of interest section (just after the warnings section).
Here you can enable the Region of Interest choose the shape (circle or rectangle) and the dimensions.
# Dataset tags
Dataset tags help you organize, categorize, and filter datasets at scale.
## Add tags to a dataset
To add or update dataset tags:
1. Open your dataset.
2. Click **Settings** in the top navigation bar.
3. Scroll to **General**.
4. Find the **Tags** field.
5. Enter a tag such as `Project-A`, `Camera-Front`, or `v1.0`.
6. Press **Enter** to confirm each tag.
7. Press **Save** to save the changes to the dataset.
Dataset tags are visible in dataset list (as chips next to the dataset name) and next to the dataset header.
## Filter datasets by tag
You can filter the datasets list by tag.
1. Open the main datasets page.
2. Open the filter and select the tag(s).
3. The list updates to show matching datasets.
# Customize label queue
You can customize the label queue by setting a sample priority and by assigning samples to a specific labeler or reviewer. An in-depth explanation of how the label queue order is determined is available in [Label queue mechanics](/background/label-queue-mechanics).
## Set sample priority
The priority value of a sample determines its [order in the label queue](/background/label-queue-mechanics): samples with a higher priority value are labeled first. For samples with the same priority, the oldest one is returned first. When a new sample is added to a dataset, its default priority is 0.
{% hint style="info" %}
Note that you can also assign a negative priority to a sample.
{% endhint %}
To change the priority of a group of samples, select the samples you want to update and click the "Edit" button. In the pop-up window, fill in the desired priority value, and press "Update".
### Set the priority programmatically
You can also [set the priority value of a sample using the Python SDK](https://sdkdocs.segments.ai/en/latest/client.html#create-a-sample).
## Assign a specific labeler or reviewer
By default, a sample is not assigned to a specific labeler or reviewer, meaning that the sample can be labeled or reviewed by anyone. See [Label queue mechanics](/background/label-queue-mechanics)for more details.
If you want to ensure that a specific sample is labeled or reviewed by a particular user, you can set an assigned labeler and reviewer for it.
To change the assigned labeler and/or reviewer of a group of samples, select the samples you want to update and click the "Edit" button. In the pop-up window, fill in the username in the appropriate field, and press "Update".
### Assign programmatically
You can also set the `assigned_labeler` and `assigned_reviewer` fields [using the Python SDK](https://sdkdocs.segments.ai/en/latest/client.html#create-a-sample).
# Search within a dataset
With the search bar, you can search for samples by their name, metadata attributes, and label content.
## Search syntax
To search by sample name, just type the full or partial name of a sample.
To search by metadata attribute, use the `key:value` syntax.
To search by number of objects of a category in a labelset, use the `labelset.category:=value` syntax, or `labelset.total_count:=value` to search by total number of objects.
To search by labeler and reviewer of a sample, use the `labeled-by:username` or `reviewed-by:username` syntax.
For string values, use the `:` operator. For numeric values, use the operators `:=`,`:>`, `:>=`, `:<` and `:<=` to search for values that are equal to, greater than, greater than or equal to, less than, and less than or equal to another value. For list values, use the `:|` operator to search for a value contained in the list.
Queries can be combined by separating them with a space. They will be ANDed together.
## Search from label statistics
On the **Overview** tab, you can click a category name in the label statistics to jump to the **Samples** tab.
The search bar is pre-filled with the corresponding category query.
The current search query is also reflected in the page URL through the `?q=` query parameter. For example, `?q=ground-truth.category:>0` opens the **Samples** tab with that filter already applied.
This makes filtered views easy to bookmark and share. The search input and URL stay in sync in both directions: typing in the search bar updates the URL, and opening or editing a URL with `?q=` updates the search field.
## Examples
| Examples |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **image\_grayscale** matches samples with the word "image\_grayscale" in their name. |
| **city:london** matches samples with a metadata attribute "city" set to "london". |
| **vehicle\_id:>3** matches samples with a metadata attribute "vehicle\_id" set to a value larger than 3. |
| **ground-truth.car:>=5** matches samples where the "ground-truth" label contains 5 or more "car" objects. |
| **my-predictions.car:<5** matches samples where the "my-predictions" label contains less than 5 "car" objects. |
| **my-predictions.total\_count:<=20** matches samples where the "my-predictions" label contains 20 or fewer objects in total. |
| **city:london ground-truth.car:>0 my-predictions.car:=0** matches samples where metadata attribute "city" is set to "london" AND the "ground-truth" label contains more than 0 "car" objects AND the "my-predictions" label contains 0 "car" objects. |
| **tags:\|red** matches samples where metadata attribute "tags" is a list of values \["red", "green", "blue"]. |
| **labeled-by:jane** matches samples labeled by the user with username jane. |
# Clone a dataset
### Clone an existing dataset
To copy the contents of a dataset to a new dataset (with possibly a different task type), you can make use of the clone functionality.
{% hint style="info" %}
You can only clone public datasets and datasets in which you are an administrator.
{% endhint %}
1. Open the dataset you want to clone.
2. Press the clone button at the top right.
3. Choose the desired task type of the cloned dataset.
4. Press the Clone button. When the dataset is cloned, you will be redirected to the cloned dataset.
### Clone an example dataset
To try out the different interfaces, a number of different example datasets are provided. These can be cloned into your personal account.
1. Either go to your Segments.ai homepage or to any organization you are an administrator of
2. Click on the dropdown next to 'New dataset' and select 'Clone an example dataset'
3. Select a data type and a task type, and press the Clone button.
# Work with issues
## Creating Issues
Both labelers and reviewers can open a new issue on a sample by clicking the icon in the top navigation bar. For example, a labeler might open an issue when unsure how to label something, or a reviewer might open one after finding a mistake in a labeled sample.
When an issue is created, it automatically becomes an "open" issue, indicated by a red number next to to message icon.
An issue must include a description or at least one tag. You can also optionally add screenshots, assign tags for categorization, and link an issue to specific context like sensor, frame, object, track,\
attributes, or a location. Any available fields will be pre-filled based on your current selection (e.g., the current sensor).
When visualizing an issue, a user can see all the data linked to it. By clicking on the tag, the user is brought directly to the relevant context (e.g., frame, sensor).
### Tags
You can configure issue tags to help categorize and triage issues on your dataset. Click the "Issue Tags" section under Settings → Labeling to manage tags.
Each tag has:
* Name: a short label for the tag (must be unique within the dataset)
* Color: choose from 10 preset colors or pick a custom color
* Description (optional): a longer explanation of when to use the tag
Tags can be edited or deleted at any time. Deleting a tag does not remove it from existing issues — it will appear as an orphaned tag with reduced opacity.
Once configured, tags become available in the issue creation and editing forms within the labeling interface. Tags appear as colored badges on the issue. When more than two tags are assigned, the extra tags are collapsed with a "+N" indicator.
Tags help categorize and triage issues — for example, you might use tags like "unclear boundary", "missing object", or "wrong category" to distinguish different types of problems.
### Managing Issues
Everyone can "resolve" issues by clicking the checkmark icon. A resolved issue is indicated with a green icon.
Issue authors and admins can edit an issue's description or delete an issue by clicking the ellipsis icon and choosing the action from the dropdown menu.
Everyone can add replies to an issue, and only reply authors can delete their own replies. This feature enables back-and-forth communication between labelers and reviewers. Remember that rejected samples always return to the original labeler, while corrected samples go back to the original reviewer.
Reviewers and admins have access to an issues panel with a filter dropdown. You can filter issues by status (open or closed) and by tags. When filtering by tags, issues matching any of the selected tags are shown. An active filter is indicated by a dot on the filter button.
### Anchoring Issues
For more precision, optional tools are available to anchor an issue to specific elements. An issue can be linked to:
1. A track (and optionally a specific track attribute)
2. A specific location in the point cloud or image
3. The scene itself (and optionally a specific scene attribute)
Link an issue to a specific point or pixel
# Bulk change label status
To change the label status of all samples in bulk, go to **Settings -> Labeling** and click the "**Advanced**" link at the bottom of the page.
# Manage QA processes
How Segments.ai helps to streamline QA, how to set up a linting process and which additional features can be leveraged
## Use short feedback loops
The platform ensures short feedback loops through the following design choices:
1. Each sample is annotated by 1 labeler. A sample is either an individual image or point cloud, or a sequence of images, point clouds or multiple sensors, depending on the dataset settings - see [Main concepts](/background/main-concepts#sample).
2. Labelers are not able to manually select samples to label first. Through the "Start Labeling" workflow, samples are presented for them.
3. After a reviewer has rejected a sample, the original labeler has to correct the rejected sample before being able to continue labeling any other samples in the queue. The rejected sample will appear in the beginning of the labeler's queue. For more details about the label queue, see [Label queue mechanics](/background/label-queue-mechanics)
4. After a labeler has corrected a rejected sample, the original reviewer has to review the corrected sample. The corrected sample will appear in the beginning of the reviewer's queue.
## Set up a linting process
Linting is the process of performing static analysis to flag erroneous patterns. For example, one would want to programmatically ...
1. Identify cuboids with unexpected dimensions or positions
2. Spot too small segmentation masks or uncover unlabeled pixels
3. Observe movement errors in sequences
4. Flag incorrect categories
Using the API/SDK & webhooks system, it is straightforward to set up such linting process and verify labels against expected properties:
* The webhooks system allows to receive event notifications whenever a sample has been labeled, such that the linting process can be triggered (see also [Set up webhooks](/how-to-integrate/webhooks))
* The API/SDK offers a programmatic way to (see also [Python SDK quickstart](/tutorials/python-sdk-quickstart))
* List & pull labels
* Verify properties for these labels
* Change the status of labels to e.g. "Rejected"
* Report issues
For more details, please [contact us](https://segments.ai/contact).
## Discover additional features to improve QA
#### Work with issues
Leave comments, post screenshots or ask questions with the issues functionality. See [Work with issues](/guides/work-with-issues)
#### Review using ratings
Enable the ratings functionality in the dataset and leave star-based ratings
Leave star-based ratings
#### Add an additional QA round
Add a "Verified" label status, intended to support an additional QA round of all labels with status "Reviewed"
Enable ratings and add a "Verified" label status in the dataset settings
#### Enable validation check to warn users about unlabeled points in the 3D segmentation interface
This feature displays an alert when attempting to save if there are any unlabeled points.
This option is only available for datasets with the "Pointcloud" data type and the "Segmentation" task.
The warning is optional and can be enabled through dataset settings
# Open a sample in read-only mode
## Overview
A sample can be opened in read-only mode which restricts the actions of the user in the interface.
## How it works
The read-only mode disables the following actions in the interface:
* All annotation hotkeys (draw, edit, copy/paste...)
* Any action implying a modification
* Creating or modifying objects and keyframes
* All destructive actions (delete, merge, split tracks)
* Saving a sample
The user can still:
* View and browse the annotations
* Use navigation, camera controls, display toggle and selection hotkeys
* View and create issues
* Modify its visualization settings in the right sidebar
## How to use the read-only mode
There are two ways to trigger the read-only mode:
* Assign the Viewer role from the dataset Collaborators settings page (More information about roles settings [here](https://docs.segments.ai/guides/add-collaborators-to-a-dataset)). Opening a sample with this user role will enforce the read-only mode.
* Manually open a sample in read-only mode from the `Samples` tab, by clicking on the eye icon on one of listed samples at mouse hover.
Note that while using the read-only mode manually, the web page URL is appended with the query parameter `readOnly=true` . It can be useful to share a link using the read-only mode. However, if you refresh the page, the read-only mode is not used by default (except when in the Viewer role, as described above).
Viewers cannot edit annotations, use labeling shortcuts, or modify any dataset settings.
# Caching assets
A dedicated cache is used to store sample point cloud files and images to disk. This allows recently visited samples to load more quickly by loading cached assets from disk rather than downloading them new.
For samples in sequence datasets, assets can be proactively [prefetched](/how-to-annotate/label-sequences-of-data/prefetch).
Assets accessed in the past 7 days are retained and the maximum size is capped at 10 GB.
If needed, the cache can be refreshed or cleared from the settings sidebar.
# Import data
There are two ways to import the data you want to label:
1. Upload the data to Segments.ai's asset storage service via the web interface or Python SDK.
2. **Recommended**: Keep the data in your own cloud bucket and submit the URLs to Segments.ai via the Python SDK.
## 1. Upload data to Segments.ai's asset storage service
{% hint style="warning" %}
The maximum file size for our asset storage service is 100MB and gets stored in an AWS S3 bucket in eu-west-2 (London). The uploaded data is accessible via [public but unguessable URLs](#public-but-unguessable-urls).
For production use cases, we always recommend to [keep the data in your own cloud bucket](#id-2.-keep-the-data-in-your-cloud-bucket).
{% endhint %}
### Via the web interface
Within a dataset, click the "Add samples" button or drag and drop files to the page. The uploaded assets (e.g. image or point cloud files) are stored in the Segments AWS S3 bucket. The asset URLs are public but unguessable, making them only accessible to dataset collaborators.
### Via the Python SDK
See the [Python SDK reference](https://sdkdocs.segments.ai/en/latest/client.html#upload-an-asset-to-segments-s3-bucket).
## 2. Keep the data in your cloud bucket
If you want to keep the data in your own cloud bucket or on your own file server, you can use the [Python SDK to submit the URLs](https://sdkdocs.segments.ai/en/latest/client.html#create-a-sample) to Segments.ai. In this case, no data is copied to our own storage system, only a reference (URL) to the data is stored in our database. This can be done in three ways:
### **Public but unguessable URLs**
Keep the data in a cloud bucket whose content can be publicly accessed but not listed. You store the assets in this bucket with unguessable file names (containing a random uuid) such that they can only be accessed by third parties who you've shared the URLs with.
### **Customer-secured URLs**
Keep the data in a private cloud bucket or server, and generate proxied or pre-signed URLs on your end to retain full control of the access permissions. These URLs can have custom restrictions: expiry time, maximum number of accesses, IP whitelisting, rate limits, etc.
### **Cross-account access**
Keep the data in a private cloud bucket or server, and grant us cross-account access. In this case, we generate temporary pre-signed URLs whenever the images need to be displayed in the frontend. For setting this up, see [Cloud integrations](/how-to-integrate/import-data/cloud-integration).
# Cloud integrations
## AWS
### Granting cross-account access
If your data is stored in a private AWS S3 bucket, you can submit your image URLs in *virtual-hosted-style format*: `https://.s3..amazonaws.com/`.
We will then use the S3 API to access data from your S3 bucket, using AWS account ID `931508227573` (canonical ID `a2c85e730d80dcb51a2c0e1a8f852cf6dc8d6e04d9e00f49239c324de3c1e3e1`). We generate temporary [presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html) to serve the images from your S3 bucket in our frontend. These URLs expire after 24 hours.
This setup requires that you give the Segments.ai AWS account read-only access to the data in your bucket. You can do this by granting us [cross-account access](https://aws.amazon.com/premiumsupport/knowledge-center/cross-account-access-s3/) through setting an appropriate [bucket policy](https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html):
1. In your AWS account, go to the S3 Management Console.
2. Go to your bucket.
3. Go to the **Permissions** tab.
4. In the **Bucket Policy** section, click the **Edit** button.
5. Paste the following bucket policy and save your changes. Don't forget to replace `YOUR_BUCKET_NAME` with the name of your bucket.
```
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "segments-s3-access",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::931508227573:user/segmentsai-prod-user"
},
"Action": [
"s3:GetObject"
],
"Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"
}
]
}
```
{% hint style="info" %}
If you run into any issues when setting up this integration, please [contact us](https://segments.ai/contact).
{% endhint %}
### CORS configuration
You also need to configure [CORS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cors.html) for your S3 bucket:
1. In your AWS account, go to the S3 Management Console.
2. Go to your bucket.
3. Go to the **Permissions** tab.
4. In the **Cross-origin resource sharing (CORS)** section, click the **Edit** button.
5. Paste the following configuration and save your changes.
```
[
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
"GET"
],
"AllowedOrigins": [
"https://*.segments.ai"
],
"ExposeHeaders": []
}
]
```
## Google Cloud
### Granting cross-account access
1. Please [contact us](https://segments.ai/contact) to request a Service Account Email ID for this GCP integration.
2. **Create a new role** for the principal as follows:
1. In your GCS account, go to **Roles** and click **Create Role**
2. Give the role a title, ID and description
3. Click **Add permissions** and select the `storage.objects.get` permission
4. Click the **Create** button
3. In your GCS account, go to your **GCS bucket permissions**.
4. Click **Grant access** under the **View by principals** section and paste the Service Account Email ID that we shared with you in the **New principals** field.
5. **Select the custom role** you just created as the role, and press Save.
6. Configure **CORS** on your bucket as follows:
1. Click the **Activate Cloud Shell** button (a terminal window icon) in the top-right corner
2. In Cloud Shell, create a JSON file containing the CORS configuration by entering the command `echo '[{"origin":["https://*.`[`segments.ai`](http://segments.ai/)`"],"method":["GET"],"responseHeader":["*"]}]' > cors-config.json`
3. Apply the CORS configuration to the bucket with the command `gsutil cors set cors-config.json gs://`
7. Verify the CORS configuration with the command `gsutil cors get gs://`
8. Please [share the name of your bucket with us](https://segments.ai/contact), so we can enable the integration on our side
Once this is set up, you can start using `gs://` URLs in your samples, pointing to files in your private bucket.
## Azure
### Granting cross-account access
1. Sign in to the [Azure Portal](https://portal.azure.com/).
2. In the search bar at the top, type **Microsoft Entra ID.**
3. Copy the **Tenant ID** on the **Overview** page and [share it with us](https://segments.ai/contact).
4. Install the **Segments.ai Blob Link Service app** into your tenant by running `az ad sp create --id d47021c7-05d1-44c3-a594-93c2c30c68fc` ([Azure docs](https://learn.microsoft.com/en-us/entra/identity/enterprise-apps/create-service-principal-cross-tenant?pivots=azure-cli)).
5. Grant the app access to your storage account:
1. In the Azure Portal, go to the **Storage Account** you want to share.
2. In the left menu, select **Access control (IAM)**.
3. Click **+ Add → Add role assignment**.
4. Choose **Storage Blob Data Reader** as the role (= read access only).
5. Select **Assign access to → User, group, or service principal**.
6. Find the **Enterprise application** you just created for the Segments.ai - Blob Link Service app.
7. Save the assignment.
### CORS configuration
You also need to configure [CORS](https://learn.microsoft.com/en-us/cli/azure/storage/cors?view=azure-cli-latest) by running this Azure CLI command:
```
az storage cors add
--services b
--methods GET
--origins https://app.segments.ai
--max-age 3600
--account-name
```
You can also [configure CORS via the Azure Portal](https://learn.microsoft.com/en-us/azure/container-apps/cors?tabs=arm\&pivots=azure-portal).
# Export data
To download your labeled data, create a release on the Releases tab of your dataset. A release is a snapshot of your dataset at a specific point in time.
By clicking the download link of a release, you obtain a release file in JSON format. This release file contains all information about the dataset, tasks, samples, and labels in the release.
This section contains information about
1. [The structure of the release file](/how-to-integrate/export/structure-of-the-release-file)
2. [How to export the release file to different formats via the SDK](/how-to-integrate/export/exporting-image-annotations-to-different-formats)
Please refer to [this blog post](https://segments.ai/blog/speed-up-image-segmentation-with-model-assisted-labeling) for an example of training a segmentation model on exported data with some useful code snippets.
# Structure of the release file
The general structure of the release file is as follows:
{% code title="" %}
```json
{
"name": "first release",
"description": "This is a first release of Segments.ai playground dataset",
"created_at": "2020-07-09 10:20:19.888887+00:00",
"dataset": {
"name": "flowers",
"task_type": "segmentation-bitmap",
"task_attributes": {...} // the categories etc.
"labelsets": [
/** list of labelsets **/
],
"samples": {
/** list of samples **/
}
}
}
```
{% endcode %}
### Sample
Each sample entry contains information about the sample (name, image URL, ...) and a list of labels.
```json
{
"name": "donuts.jpg",
"attributes": {
"image": {
"url": "https://segmentsai-prod.s3.eu-west-2.amazonaws.com/assets/segments/3b8b3da2-f09a-494b-999e-37250dfbf5b6.jpg"
}
},
"labels": {
/** list of labels, indexed by labelset **/
}
}
```
### Label
The `attributes` field of a label contains all the info about the labeled objects. Its contents depend on the type of the label (image segmentation, cuboid,...). See [Label formats](/reference/label-types) for an overview of the attributes per label type. Each label also contains basic information such as the time it was created, the user who created it, its status (e.g. LABELED).
{% code title="" %}
```bash
{
"label_status": "LABELED",
"attributes": {
"format_version": "0.1",
"annotations": [
{
"id": 1,
"category_id": 1
},
{
"id": 2,
"category_id": 1
},
{
"id": 3,
"category_id": 1
},
{
"id": 4,
"category_id": 1
}
],
"segmentation_bitmap": {
"url": "https://segmentsai-prod.s3.eu-west-2.amazonaws.com/assets/segments/504e7633-ef51-49c3-8b0e-d4eb9100532d.png"
}
}
}
```
{% endcode %}
### Label set
Each `labelset` entry contains the labelset's name and description:
```json
{
"name": "ground-truth",
"description": ""
}
```
# Exporting image annotations to different formats
## Exporting the release file for image datasets to different formats
You can export the release file for image datasets to different formats with the Python SDK. Use the `export_dataset`util function for this, setting the `export_format` parameter to one of the following:
| Value | Description |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `coco-instance` | [COCO instance](https://cocodataset.org/#format-data) segmentation format |
| `coco-panoptic` | [COCO panoptic](https://cocodataset.org/#format-data) segmentation format |
| `yolo` | [Yolo Darknet](https://github.com/AlexeyAB/darknet) object detection format |
| `instance` | Grayscale PNGs (16-bit) where the values correspond to instance ids |
| `semantic` | Grayscale PNGs (8-bit) where the values correspond to category ids |
| `instance-color` | Colored PNGs where the colors correspond to different instances |
| `semantic-color` |
Colored PNGs where the colors correspond to different categories, with colors as configured in the label editor settings when available
|
| `polygon` | For exporting segmentation bitmap labels to polygons |
Example:
```python
# pip install segments-ai
from segments import SegmentsClient, SegmentsDataset
from segments.utils import export_dataset
# Initialize a SegmentsDataset from the release file
client = SegmentsClient('YOUR_API_KEY')
release = client.get_release('jane/flowers', 'v1.0') # Alternatively: release = 'flowers-v1.0.json'
dataset = SegmentsDataset(release, labelset='ground-truth', filter_by=['labeled', 'reviewed'])
# Export to COCO panoptic format
export_dataset(dataset, export_format='coco-panoptic')
```
Alternatively, you can use the initialized `SegmentsDataset` to loop through the samples and labels, and visualize or process them in any way you please:
```python
import matplotlib.pyplot as plt
from segments.utils import get_semantic_bitmap
for sample in dataset:
# Print the sample name and list of labeled objects
print(sample['name'])
print(sample['annotations'])
# Show the image
plt.imshow(sample['image'])
plt.show()
# Show the instance segmentation label
plt.imshow(sample['segmentation_bitmap'])
plt.show()
# Show the semantic segmentation label
semantic_bitmap = get_semantic_bitmap(sample['segmentation_bitmap'], sample['annotations'])
plt.imshow(semantic_bitmap)
plt.show()
```
# Create an API key
To create an API key, go to your [account page](https://segments.ai/account) and click "Add new" in the API key section.
# Upload model predictions
## Uploading as pre-labels
Once you've trained an initial machine learning model on your labeled data, you can upload the model predictions as *pre-labels* to further speed up your labeling workflow.
To add a label to a sample programmatically, use the [`client.add_label()`](https://sdkdocs.segments.ai/en/latest/client.html#create-a-label) function in the Python SDK. Note that the format of the `attributes` field depends on the [label type](/reference/label-types).
```json
sample_uuid = "602a3eec-a61c-4a77-9fcc-3037ce5e9123"
labelset = "ground-truth"
attributes = {
"format_version": "0.1",
"annotations": [
{
"id": 1,
"category_id": 1,
"type": "bbox",
"points": [
[12.34, 56.78],
[90.12, 34.56]
]
}
]
}
client.add_label(sample_uuid, labelset, attributes)
```
The sample will now have a label status of *prelabeled*, and will appear in the label queue along with any unlabeled samples. Instead of having to label the sample from scratch, the labelers can now focus on verifying and correcting the pre-label though.
# Set up webhooks
Listen for events on your account so your integration can automatically trigger reactions.
Webhooks are automated messages sent to your server when something happens. You can use webhooks to subscribe to certain events on your account and automatically trigger reactions.
To enable webhooks for the datasets under your account, go to [your account page](https://segments.ai/account) and click the "Enable" button in the webhooks section. Then click the "Manage webhooks" button to add endpoints, subscribe to events, view and replay webhooks.
Every webhook is signed with a unique key for increased security. Please refer to [these docs](https://docs.svix.com/receiving/verifying-payloads/how) to verify and validate your incoming webhooks.
You can currently subscribe to following events:
| Event type | Description |
| ----------------- | ------------------------------------------------------------------------ |
| `dataset.created` | Occurs whenever a dataset is created. |
| `dataset.updated` | Occurs whenever a dataset property has changed. |
| `dataset.deleted` | Occurs whenever a dataset is deleted. |
| `sample.created` | Occurs whenever a sample is created. |
| `sample.updated` | Occurs whenever a sample property has changed. |
| `sample.deleted` | Occurs whenever a sample is deleted. |
| `label.updated` | Occurs whenever a label property has changed. |
| `label.deleted` | Occurs whenever a label is deleted. |
| `issue.created` | Occurs whenever an issue is created. |
| `issue.updated` | Occurs whenever an issue has received a reply or a property has changed. |
| `release.created` | Occurs whenever a new release is successfully created. |
The webhook payload looks as follows:
```javascript
{
"event_type": "sample.created",
"created_at": "2021-08-09T14:40:19.987691+00:00",
"api_version": "2021-08-01",
"dataset": "jane/flowers",
"object": { ... }, // The dataset, sample, label or issue that was created or updated.
"text": "Sample rose.png created in dataset jane/flowers." // Text summary, useful for Slack integration.
}
```
## Setting up a Slack integration with webhooks
This is a guided tutorial on how to set up a Slack integration using webhooks with [Segments.ai](https://segments.ai). For more information, please refer to [Slack's instructions](https://api.slack.com/messaging/webhooks).
### Create a Slack app, enable incoming webhooks and create an endpoint URL in Slack
1. Create a Slack app, enable incoming webhooks and create an endpoint URL in Slack.
2. You'll be redirected to its settings page. From here, select the Incoming Webhooks feature and toggle on the Activate Incoming Webhooks.
3. Some extra options will now appear. In the bottom of the page, click the Add New Webhook to Workspace button and choose a channel.
4. You'll be sent back to your app settings, and you should now see a new Webhook URL entry with an endpoint URL formatted like`https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX`
You've now created an endpoint URL for the incoming webhook, specific to a single user and a single channel.
### Configure your webhooks in Segments.ai using the Slack endpoint URL
1. Go to your [Segments.ai account page](https://segments.ai/account/) and click the "Enable" button in the webhooks section.
2. Click the green "Manage webhooks" button and you'll be taken to the webhook configuration page.
3. Add an endpoint and provide the following items in the configuration window:
* The endpoint URL that you just created in the Slack settings
* An optional version number and description
* Which events to send notifications about to the Slack channel
The webhook integration is now set up.
{% hint style="info" %}
You will start to receive notifications, but only for those events within datasets of which you're an owner.
{% endhint %}
# Main concepts
## Dataset
Segments.ai is built around the concept of **datasets**. A dataset contains a collection of samples.
When you create a dataset, you have to choose the sample and label type. For example, *images* with *segmentation labels*, or *3D point clouds* with *cuboid labels*.
## Sample
A **sample** is a data point you want to label, like an image, 3D point cloud, or a [sequence](/background/sequences). The different sample types are defined in [Sample formats](/reference/sample-types).
## Label
When you label a sample and press the save button, you've created a **label** for that sample. Labels also come in different types (segmentation labels, vector labels, cuboid labels, ...), with the available options determined by the sample type. The different label types are specified in [Label formats](/reference/label-types).
## Label set
A **label** is linked to a sample in relation to a **label set**. When you label a sample and press the save button, the label is saved in the default *ground-truth* label set.
You can optionally create additional label sets to [upload your model predictions](/how-to-integrate/upload-model-predictions).
## Release
A **release** is a snapshot of a dataset at a specific point in time. Create a new release if you want to [export your data](/how-to-integrate/export).
# Sequences
A **sequence** is a sample comprised of multiple frames. Each frame represents a data point you want to label, such as an image or a 3D point cloud. Using sequences can help increase labeling speed and consistency when working with sequential data.
A sequence has a number of unique features:
* A track ID is assigned to each labeled object in a sequence. This track ID can be used to track an object over multiple frames. Learn how to use track IDs [here](/how-to-annotate/label-sequences-of-data/use-track-ids-in-sequences).
* A sequence is assigned to a single labeler. This can increase the consistency of the labels between the frames in the sequence.
* The image vector labeling interface and 3D point cloud cuboid interface allow you to use keyframe interpolation. This reduces the number of frames you need to label manually, which can decrease the labeling time significantly. Learn how to use keyframe interpolation [here](/how-to-annotate/label-sequences-of-data/use-keyframe-interpolation).
### Keyframes and remove-keyframes
In the context of interpolation, a **keyframe** is a marker that indicates that at that frame, the state of the object is defined by the user. In other frames (that are not keyframes), the state of the object label is calculated automatically through interpolation.
A **remove-keyframe** is a visual indication of where an object was removed from the sample. A remove-keyframe means that the object is not present in that frame and all frames before the next normal keyframe.
## Timeline interface
The timeline is always visible at the bottom of the editor and provides a visual representation of all tracks and their states across the sequence.
### What the timeline shows
The timeline displays tracks as horizontal rows with:
* **Track information** on the left (track ID and category name)
* **Frame columns** showing the track's state across time
* **Visual indicators** for keyframes and object presence
### Visual indicators
| Element | Appearance | Meaning |
| ------------------ | ------------------------- | ------------------------------------------- |
| Colored bar | Continuous horizontal bar | Track is visible in these frames |
| White diamond (◆) | Icon on colored bar | Keyframe (user-defined object state) |
| White cross (×) | Icon on blank space | Remove-keyframe (object explicitly removed) |
| Highlighted column | Vertical highlight | Active frame (current position) |
### Timeline views
The timeline adapts to show different information:
**When no track is selected:**
* Switch between viewing all tracks or scene attributes using the View dropdown
**When a track is selected:**
* Shows only that track with its associated track attributes
This dynamic display helps you focus on relevant information while labeling.
### Attributes in the timeline
The timeline can display two types of attributes:
**Scene attributes** - Properties that apply to the entire scene or individual frames:
* Example: Weather conditions, lighting, time of day
* Visible when no track is selected (Scene attributes view)
**Track attributes** - Properties specific to individual tracks:
* Example: Vehicle speed, object confidence score
* Visible when a track is selected
Attributes can be frame-level (changing per frame) or sequence-level (constant across frames).
{% hint style="info" %}
For detailed instructions on using the timeline interface, see Track timeline. For keyframe operations, see Use keyframe interpolation.
{% endhint %}
# Label queue mechanics
The label and review queue make it easy for teams to efficiently work on a dataset together.
{% hint style="info" %}
Note that a single [sample ](/background/main-concepts#sample)can either be an *individual image* or an *image sequence consisting of multiple frames*, depending on the chosen dataset type. Same for point cloud data.
{% endhint %}
When you upload a new sample, its initial status is *unlabeled*. Samples are always either *unlabeled*, *prelabeled*, *labeled (or labeling in progress)*, *reviewed (or reviewing in progress)*, *rejected*, or *skipped*.
Dataset **administrators and managers** can open any sample directly via the *Samples* tab. They can update the label and freely change the label status.
Dataset **labelers and reviewers** don't have access to the *Samples* tab. They can only click the blue *Start labeling* or *Start reviewing* buttons. This brings them in a workflow where they are automatically assigned samples from the label or review queue in a certain order, as explained below.
'Start Labeling' and 'Start Reviewing' workflows
### Label queue
When a labeler presses the *Start labeling* button, a single sample is fetched from the label queue.
In the labeling workflow, there are three buttons:
* **Submit**: set the sample status to *labeled (*moving it to the review queue) and go to the next sample in the queue.
* **Skip**: set the sample status to *skipped* and go to the next sample in the queue.
* **Save** *(only visible if enabled in the dataset settings)*: set the sample status to *labeling in progress* but don't go to the next sample in the queue yet. Can be helpful when labeling larger samples.
If a labeler presses the "Start labeling" button, they will get samples from the label queue in this order:
1. Samples which they started labeling but didn't finish yet. Only if the *save* button is enabled.
2. Samples they labeled but which were rejected in the reviewing step, and now need to be corrected.
3. Unlabeled, prelabeled or rejected samples which are [specifically assigned](/guides/customize-label-queue#assign-a-specific-labeler-or-reviewer) to this user, through the `assigned_labeler` field.
4. Unlabeled or prelabeled samples which are not assigned to a specific user.
5. If no such samples exist, the label queue is empty and no more samples need to be labeled.
Within each step, samples with higher priority are returned first. Read more about how you can [customize the queue priority](/guides/customize-label-queue).
### Review queue
When you press the *Start reviewing* button, a single sample is fetched from the review queue.
In the reviewing workflow, there are four buttons:
* **Accept**: set the sample status to *reviewed*.
* **Reject**: set the sample status to *rejected*, moving it back onto the label queue.
* **Skip** *(only visible if enabled in the dataset settings)*: set the sample status to *skipped*.
* **Save** *(only visible if enabled in the dataset settings)*: set the sample status to *reviewing in progress* but don't go to the next sample in the review queue yet. Can be helpful when reviewing larger samples.
If a reviewer presses the *Start reviewing* button, they will get samples from the review queue in this order:
1. Samples which they started reviewing but didn't finish yet. Only if the save button is enabled.
2. Samples they rejected before and have now been corrected by the original labeler, so they need to be reviewed again.
3. Labeled samples which are [specifically assigned](/guides/customize-label-queue#assign-a-specific-labeler-or-reviewer) to this user, through the `assigned_reviewer` field.
4. Labeled samples which are not assigned to a specific user, and which haven’t been labeled by this user (to avoid reviewing their own labeled samples if a team participates in both labeling and reviewing).
5. Labeled samples which are not assigned to a specific user, independent of who labeled them before (to prevent deadlock if a single user wants to both label and review a dataset).
6. If no such samples exist, the review queue is empty and no more samples need to be reviewed.
Within each step, samples with higher priority are returned first. For samples with the same priority, the oldest one is returned first. Read more about how you can [customize the queue priority](/guides/customize-label-queue#set-sample-priority).
# Labeling metrics
To get accurate labeling metrics, we recommend using the "Start labeling" and "Start reviewing" buttons for labeling. This will automatically assign each team member a sample from the label/review queue according to the rules described [here](/background/label-queue-mechanics). It also ensures that samples that got rejected by the reviewer go back to the original labeler and then back to that same reviewer after correction, creating a tight feedback loop.
Using the Start labeling/reviewing workflow also ensures that timings are properly categorized as either *label time* or *review time*. If you make any changes to a sample by directly opening it from the Samples tab and pressing the Save button, this gets categorized as *edit time*.
On the Insights tab of a dataset we show a number of labeling metrics, both on a per-user level (# labeled, total label time, average label time, # reviewed, total review time, average review time, # edited, total edit time, average edit time) and on a per-sample level (label time, review time, edit time, # rejected, # frames, # annotations).
On the Insights tab of an *organization*, you can see the per-user metrics aggregated across all datasets in that organization.
You can download these aggregated metrics as a csv/json/excel file directly from the webapp or programmatically through the API or Python SDK (reach out to us for more information).
## Advanced: workunits
Behind the scenes, the metrics are calculated from *workunits*.
A workunit represents a piece of labeling work done by a dataset collaborator on a certain sample. It is initiated when a sample is loaded in the interface, and is saved when a label is created or saved. This is generally in any of the following three events:
1. A labeler submits a sample via the 'start labeling' workflow
2. A reviewer accepts or rejects a sample via the 'start reviewing' workflow
3. A manager edits and saves a sample after opening it in the samples tab
A workunit contains the following fields:
* `uuid`: unique id of the workunit.
* `created_at`: creation time of the workunit. I.e., the time when the user saved or submitted the sample.
* `created_by`: the user who created the workunit.
* `work_type`: one of:
* **label** (sample opened in Start labeling workflow)
* **review** (sample opened in Start Reviewing workflow)
* **normal** (sample opened directly through the samples tab)
* `time`: the amount of time in milliseconds since the sample loaded, or since the previous time the save button was pressed, whichever is shorter. Note that this does not include inactive time, see below. A more appropriate name for this field would be `active_time`, but it's `time` because of legacy reasons. The metrics on the Insights tab are calculated using this value.
* `inactive_time`: the amount of inactive time during this workunit. Inactive time is defined as the sum of all time intervals between two mouse clicks or keyboard presses that are larger than the inactivity threshold (5 minutes).
* `next_label_status`: the new label status of the label, i.e. after the save/accept/reject/skip button is pressed.
If you want to run your own custom metrics calculations, you can [fetch the list of workunits of a dataset via the Python SDK](https://sdkdocs.segments.ai/en/latest/client.html#list-workunits).
## View the workunits of a sample
You can view the workunits of a specific sample in the webapp by going to the Samples tab, hovering over a sample, clicking the little info icon, and switching to the *History* tab.
# 3D Tiles
3D point clouds are collections of points in 3D space. Depending on the type of sensor used, the number of points in one scan can vary greatly. As 3D sensors have evolved, the resolution of 3D scans has generally increased. Furthermore, robots and autonomous vehicles typically have multiple 3D sensors. The outputs of these sensors can then be combined into one large point cloud scan.
High-resolution scans with hundreds of thousands of points, or even millions of points, make it easy to detect objects, but come at a cost of increased file size and compute requirements. This can become a problem when trying to label the point clouds in a browser. Due to the large file size of the point clouds, loading the point cloud files can take a long time, especially when the file is being loaded by a workforce in a region far from your data center. Displaying hundreds of thousands of points in the browser can also be taxing for the GPU, and if there are too many points, the labeling interface can become unresponsive.
To solve these issues, we've introduced an option to split your point clouds into 3D tiles. When viewing the point cloud, the application only loads the tiles which should be visible, and only loads the tiles that are close in high resolution. Thus, the point cloud can load faster, and the GPU can comfortably render all points. This approach is similar to the way Google Earth works; when you are zoomed out, the application only loads a low-resolution image of a whole area. When you zoom in, smaller high-resolution tiles are loaded automatically.
## Why use 3D tiles?
* Upload and label point clouds with **no limits on the point cloud size**
* Enable a [merged point cloud view](/how-to-annotate/label-3d-point-clouds/merged-point-cloud-view-for-static-objects) when labeling sequences of point clouds
* Enable the [batch mode](/how-to-annotate/label-3d-point-clouds/batch-mode-for-dynamic-objects) for labeling dynamic objects in point cloud sequences
## How to enable 3D tiles
This feature is currently available behind a feature flag. [Contact us](https://segments.ai/contact) if you'd like to enable 3D tiles for your point clouds.
{% hint style="info" %}
3D tiles are currently only available for 3D vector labeling tasks (cuboid, polygon/polyline, and keypoint annotation). \
\
[Contact us](https://segments.ai/contact) if you're interested in 3D tiles for point cloud segmentation.
{% endhint %}
# Security
Segments.ai is ISO27001 and GDPR compliant. Please [contact us](https://segments.ai/contact) if you have any questions.
# Task types
# Sample formats
A [sample](/background/main-concepts#sample) is a data point you want to label. Samples come in different types, like an image, a 3D point cloud, or a video sequence. When uploading ([`client.add_sample()`](https://sdkdocs.segments.ai/en/latest/client.html#create-a-sample)) or downloading ([`client.get_sample()`](https://sdkdocs.segments.ai/en/latest/client.html#get-a-sample)) a sample using the [Python SDK](https://sdkdocs.segments.ai/en/latest/client.html#), the format of the `attributes` field depends on the type of sample. The different formats are described here.
{% hint style="info" %}
The section [Import data](/how-to-integrate/import-data) shows how you can obtain URLs for your assets.
{% endhint %}
## Image
Supported image formats: jpeg, png, bmp.
```json
{
"image": {
"url": "https://example.com/image.jpg"
}
}
```
{% hint style="warning" %}
If the image file is on your local computer, you should first upload it to our asset storage service (using [`upload_asset()`](https://sdkdocs.segments.ai/en/latest/client.html#upload-an-asset-to-segments-s3-bucket)) or to another cloud storage service.
{% endhint %}
## Image sequence
Supported image formats: jpeg, png, bmp.
```json
{
"frames": [
{
"image": {
"url": "https://example.com/frame_00001.jpg"
},
"name": "frame_00001" // optional
},
{
"image": {
"url": "https://example.com/frame_00002.jpg"
},
"name": "frame_00002"
},
{
"image": {
"url": "https://example.com/frame_00003.jpg"
},
"name": "frame_00003"
}
]
}
```
## 3D point cloud
{% hint style="info" %}
On Segments.ai, the up direction is defined along the z-axis, i.e. the vector (0, 0, 1) points up. If you upload point clouds with a different up direction, you might have trouble navigating the point cloud.
{% endhint %}
```json
{
"pcd": {
"url": "https://example.com/pointcloud.pcd",
"type": "pcd"
},
"images": [
{ ... },
{ ... },
{ ... }
], // optional
"name": "frame_00001", // optional
"timestamp": "00001", // optional
"ego_pose": {
"position": {
"x": -2.7161461413869947,
"y": 116.25822288149078,
"z": 1.8348751887989483
},
"heading": {
"qx": -0.02111296123795955,
"qy": -0.006495469416730261,
"qz": -0.008024565904865688,
"qw": 0.9997181192298087
}
},
"default_z": -1, // optional, 0 by default
"bounds": { // optional
"min_z": -1,
"max_z": 3
}
}
```
Timestamp of the sample. Should be in nanoseconds for accurate velocity/acceleration calculations. Will also be used for interpolation unless disabled in dataset settings.
Pose of the sensor that captured the point cloud data.
default_z
float
Default z-value of the ground plane. 0 by default. Only valid in the point cloud cuboid editor. New cuboids will be drawn on top of the ground plane, i.e. the default z-position of a new cuboid is 0.5 (since the default height of a new cuboid is 1).
bounds
dict of <string, float>
Point cloud bounds: a dict with values that are used to initialize the limiting cuboid. The z-values are also used for height coloring when provided.
Supported values: min_x, max_x, min_y, max_y, min_z and max_z.
### Point cloud data
See [3D point cloud formats](/reference/sample-types/supported-file-formats#3d-point-cloud) for the supported file formats.
```json
{
"url": "https://example.com/pointcloud.bin",
"type": "kitti"
}
```
{% hint style="warning" %}
If the point cloud file is on your local computer, you should first upload it to our asset storage service (using [`upload_asset()`](https://sdkdocs.segments.ai/en/latest/client.html#upload-an-asset-to-segments-s3-bucket)) or to another cloud storage service.
{% endhint %}
### Camera image
A calibrated or uncalibrated reference image corresponding to a point cloud. The reference images can be opened in a new tab from within the labeling interface. You can determine the layout of the images by setting the `row` and `col` attributes on each image. If you also supply the calibration parameters (and distortion parameters if necessary), the main point cloud view can be set to the image to obtain a fused view.
```json
{
"name": "Camera example 1", // optional
"url": "https://example.com/image.jpg",
"row": 0,
"col": 0,
"intrinsics": { // optional
"intrinsic_matrix": [
[1266.417203046554, 0, 816.2670197447984],
[0, 1266.417203046554, 491.50706579294757],
[0, 0, 1]
]
},
"extrinsics": { // optional
"translation": {
"x": -0.012463384576629082,
"y": 0.76486688894964,
"z": -0.3109103442096661
},
"rotation": {
"qx": 0.713640516187247,
"qy": -0.001134052598226082,
"qz": 0.0036449450274057696,
"qw": 0.7005017073187271
}
},
"distortion": { // optional
"model": "fisheye",
"coefficients": {
"k1": -0.0539124,
"k2": -0.0101993,
"k3": -0.00202017,
"k4": 0.00120938
}
},
"camera_convention": "OpenCV", // optional
"rotation": 1.5708 // optional
}
```
Name
Type
Description
name
string
Name of the camera image.
url
string
Required. URL of the camera image.
row
int
Required. Row of this image in the images viewer.
col
int
Required. Column of this image in the images viewer.
Convention of the camera coordinates. We use the OpenGL/Blender coordinate convention for cameras. +X is right, +Y is up, and +Z is pointing back and away from the camera. -Z is the look-at direction. Other codebases may use the OpenCV convention, where the Y and Z axes are flipped but the +X axis remains the same. See diagram 1.
rotation
float
The rotation that needs to be applied when displaying the image. Valid options are 0, \frac{\pi}{4} , \frac{\pi}{2}, and \frac{3 \pi}{4}. Useful for when a camera is mounted upside-down.
{% hint style="warning" %}
If the image file is on your local computer, you should first upload it to our asset storage service (using [`upload_asset()`](https://sdkdocs.segments.ai/en/latest/client.html#upload-an-asset-to-segments-s3-bucket)) or to another cloud storage service.
{% endhint %}
#### Camera intrinsics
```json
{
"intrinsic_matrix": [
[1266.417203046554, 0, 816.2670197447984],
[0, 1266.417203046554, 491.50706579294757],
[0, 0, 1]
]
}
```
Name
Type
Description
intrinsic_matrix
2D array of floats representing 3x3 matrix Kin row-major order
Required. Intrinsic matrix K used in the pinhole camera model. K = \begin{bmatrix} f_x & 0 & o_x\\ 0 & f_y & o_y \\ 0 & 0 & 1 \end{bmatrix}
f_x and f_y are the focal lengths in pixels. We assume square pixels, so f_x = f_y. o_x and o_y are the offsets (in pixels) of the principal point from the top-left corner of the image frame.
Required. Translation of the camera in lidar coordinates, i.e., relative to the ego pose.
rotation
object: { "qx": float, "qy": float, "qz": float,
"qw": float }
Required. Rotation of the camera in lidar coordinates, i.e., relative to the ego pose (or equivalently: a transformation from camera frame to ego frame). Defined as a rotation quaternion. By default, we use the OpenGL/Blender coordinate convention for cameras. +X is right, +Y is up, and +Z is pointing back and away from the camera. -Z is the look-at direction. Other codebases may use the OpenCV convention, where the Y and Z axes are flipped but the +X axis remains the same. See diagram 1. You can specify the camera convention in #camera-image.
Diagram 1: camera convention for calibrated camera images on Segments.ai.
Required. Coefficients of the distortion model: k1, k2, k3, k4 for fisheye (see the OpenCV fisheye model) and k1, k2, k3, p1, p2 for Brown-Conrady (see the OpenCV distortion model, note that k_4 and k_5 are not used).
### Ego pose
The pose of the sensor used to capture the 3D point cloud data. This can be helpful if you want to obtain cuboids in world coordinates, or when your sensor is moving. In the latter situation, supplying an ego pose with each frame will ensure that static objects do not move when switching between frames.
```json
{
"position": {
"x": -2.7161461413869947,
"y": 116.25822288149078,
"z": 1.8348751887989483
},
"heading": {
"qx": -0.02111296123795955,
"qy": -0.006495469416730261,
"qz": -0.008024565904865688,
"qw": 0.9997181192298087
}
},
```
Name
Type
Description
position
object: { "x": float, "y": float, "z": float }
Required. XYZ position of the sensor in world coordinates.
{% hint style="warning" %}
Segments.ai uses 32-bit floats for the point positions. Keep in mind that 32-bit floats have limited precision. In fact, only 24 bits can be used to represent the number itself (the significand, excluding the sign bit), or about 7.22 decimal digits. If you want to keep two decimal places, this only leaves 5.22 decimal digits, so the numbers shouldn't be larger than 10^5.22 = 165958.
To avoid rounding problems, it is best practice to subtract the ego position of the first frame from all other ego positions. This way, the first ego position is set to (0, 0, 0) and the subsequent ego positions are relative to (0, 0, 0) . In your export script, you can add the ego position of the first frame back to the object positions.
{% endhint %}
## 3D point cloud sequence
```json
{
"frames": [
{ ... },
{ ... },
{ ... }
]
}
```
# Supported file formats
## Image
Following image file formats are supported: jpeg, png, bmp.
## 3D point cloud
### PCD (Point Cloud Data)
[Version 0.7](https://pointclouds.org/documentation/tutorials/pcd_file_format.html) of the PCD format is supported. PCD files can either be ASCII-encoded or binary files. The PCD files should contain at least x, y, and z coordinate fields. Optionally, you can supply an *intensity* or an RGB field. These fields are used for setting the color of the points. Intensity coloring can be enabled or disabled in the viewer.
Any other fields will be ignored.
Field name
Size (#bytes)
Type
Required
x
4
float
true
y
4
float
true
z
4
float
true
intensity
4
float
false
rgb
4
float
false
{% hint style="warning" %}
Please make sure that you supply the values as 32-bit (=4 byte) floats.
Keep in mind that 32-bit floats have limited precision. In fact, only 24 bits can be used to represent the number itself (the significand, excluding the sign bit), or about 7.22 decimal digits. If you want to keep two decimal places, this only leaves 5.22 decimal digits, so the numbers shouldn't be larger than 10^5.22 = 165958.
To avoid rounding problems, it is best practice to subtract the ego position of the first frame from all other ego positions. This way, the first ego position is set to (0, 0, 0) and the subsequent ego positions are relative to (0, 0, 0) . In your export script, you can add the ego position of the first frame back to the object positions.
{% endhint %}
### Binary xyzi(r) (KITTI/nuScenes)
Segments.ai supports the binary point cloud formats used by the [KITTI](http://www.cvlibs.net/datasets/kitti/index.php) and [nuScenes](https://www.nuscenes.org/) datasets. These formats do not contain a header and have a fixed number of fields. When uploading a sample with point clouds in one of these formats, use `binary-xyzi` (alias `kitti`) or `binary-xyzir` (alias `nuscenes`) for the type field.
Field name
Size (#bytes)
Type
Required
x
4
float
true
y
4
float
true
z
4
float
true
intensity
4
float
true
ring index
4
float
false
### PLY (Stanford Triangle Format)
The PLY file format can be used for point clouds by encoding the points as vertices. The PLY header should thus contain a vertex *element* containing x, y, and z *properties* and optionally also color or intensity properties. Both binary and ASCII PLY files are supported.
{% hint style="danger" %}
PLY Gaussian Splats are not supported
{% endhint %}
Property name
Type
Required
x
float32
true
y
float32
true
z
float32
true
red
uchar [0, 255]
false
green
uchar [0, 255]
false
blue
uchar [0, 255]
false
intensity
float32
false
### LAS
{% hint style="warning" %}
* LAS support is limited to 3D cuboid and 3D vector task types. It is not supported for 3D point cloud segmentation.
* Unlike other file formats, the uploaded files can only be viewed after the [tiling process](/background/3d-tiles) is completed.
* The LAS file format is currently only recommended for huge point clouds (e.g. merged maps) that cannot be tiled otherwise.
{% endhint %}
[Version 1.4](https://www.asprs.org/wp-content/uploads/2019/03/LAS_1_4_r14.pdf) of the LAS file format is supported. Only uncompressed LAS files are currently supported.
Point clouds can optionally include RGB fields, or an intensity field. If both are defined, the intensity will be discarded.
* For point clouds with RGB colors, make sure to set the point format field to 2.
* For point clouds with intensity values, make sure to set the point format field to 0.
Keep in mind to set the LAS scale/resolution small enough (e.g. 1e-6) to avoid discretization errors.
Property name
Type
Required
X
float32
true
Y
float32
true
Z
float32
true
Red
uint8 [0-255]
false
Green
uint8 [0-255]
false
Blue
uint8 [0-255]
false
Intensity
uint8 [0-255]
false
## Gaussian Splat
### .splat
Gaussian splats can only be uploaded as .splat files at the moment. PLY splat files are not supported. If you want to convert your splat file from PLY to .splat, you can either use [SuperSplat](https://playcanvas.com/supersplat/editor), an online splat editor, or the open-source [point-cloud-tool](https://github.com/SpectacularAI/point-cloud-tools) library.
# Label formats
When you label a sample and press the save button, you've created a [label](/background/main-concepts#label) for that sample. Labels come in different types, with the available options determined by the type of the corresponding sample.
When downloading or uploading labels using the [Python SDK](https://sdkdocs.segments.ai/en/latest/client.html), the format of the `attributes` field depends on the type of label. The different formats are described here.
A label can additionally also contain [#object-attributes](#object-attributes "mention") and [#image-attributes](#image-attributes "mention").
## Image
### Segmentation labels
Format of the `attributes` field in [`client.get_label()`](https://sdkdocs.segments.ai/en/latest/client.html#get-a-label):
```javascript
{
"format_version": "0.1",
"annotations": [
{
"track_id": 1, // the track id. A single object has the same track_id across frames. 0.
"id": 1, // this is a legacy field and is always equal to track_id.
"category_id": 1 // this is a category id
},
{
"track_id": 2,
"id": 2,
"category_id": 1
},
{
"track_id": 3,
"id": 3,
"category_id": 4
}
],
"segmentation_bitmap": {
"url": "https://segmentsai-staging.s3.eu-west-2.amazonaws.com/assets/davy/ddf55e99-1a6f-42d2-83e9-8657de3259a1.png"
}
}
```
{% hint style="warning" %}
The`segmentation_bitmap_url`refers to a 32-bit RGBA png image which contains the segmentation masks. The alpha channel is set to 255, and the remaining 24-bit values in the RGB channels correspond to the object ids in the `annotations` list. Unlabeled regions should have a value of 0. Because of the large dynamic range, **these png images may appear black in an image viewer.**
{% endhint %}
{% hint style="info" %}
**When downloading a label**, you can use the utility function `utils.load_label_bitmap_from_url(url)` in the Python SDK to load the label bitmap as a numpy array containing object ids.
**When uploading a label**, the easiest way to transform a segmentation bitmap into this format and upload it is by using the util function`bitmap2file`:
```python
from segments.utils import bitmap2file
# segmentation_bitmap is a numpy array of type np.uint32, with values corresponding to instance_ids
file = bitmap2file(segmentation_bitmap)
asset = client.upload_asset(file, "label.png")
segmentation_bitmap_url = asset.url
```
For a full example of uploading model-generated labels to Segments.ai, please refer to [this blogpost](https://segments.ai/blog/speed-up-image-segmentation-with-model-assisted-labeling).
{% endhint %}
### Vector labels (bounding box, polygon, polyline, keypoint)
Format of the `attributes` field in [`client.get_label()`](https://sdkdocs.segments.ai/en/latest/client.html#get-a-label):
{
"format_version": "0.1",
"annotations": [
{
"track_id": 1, // the track id. A single object has the same track_id across frames.
"id": 1, // this is a legacy field and is always equal to track_id.
"category_id": 1, // the category id
"type": "bbox", // refers to the annotation type (bounding box)
"points": [
[12.34, 56.78], // x0, y0 (upper left corner of bbox)
[90.12, 34.56] // x1, y1 (lower right corner of bbox)
]
},
{
"track_id": 2,
"id": 2,
"category_id": 2,
"type": "polygon", // refers to the annotation type (polygon)
"points": [
[12.34, 56.78], // x0, y0 (starting point of the polygon)
[90.12, 34.56], // x1, y1
[78.91, 23.45], // x2, y2
[67.89, 98.76], // x3, y3
[54.32, 10.01] // x4, y4
]
},
{
"track_id": 3,
"id": 3,
"category_id": 3,
"type": "polyline", // refers to the annotation type (polyline)
"points": [
[12.34, 56.78], // x0, y0 (starting point of the polyline)
[90.12, 34.56], // x1, y1
[78.91, 23.45], // x2, y2
[67.89, 98.76], // x3, y3
[54.32, 10.01] // x4, y4
]
},
{
"id": 4,
"track_id": 4,
"category_id": 4,
"type": "point", // refers to the annotation type (keypoint)
"points": [
[12.34, 56.78] // x, y (coordinates of keypoint)
]
},
],
"links": [
{
"from_id": 1,
"to_id": 2,
"attributes": {} // the link attributes
}
]
}
## Image sequence
### Segmentation labels
Format of the `attributes` field in [`client.get_label()`](https://sdkdocs.segments.ai/en/latest/client.html#get-a-label):
```json
{
"format_version": "0.1",
"frames": [
{ ... },
{ ... },
{ ... }
]
}
```
| Name | Type | |
| ---------------- | ------------------------------------------------------ | ------------------------------------------------------------ |
| `format_version` | `string` | Format version. |
| `frames` | `array` of [segmentation labels](#segmentation-labels) | List of segmentation labels (one per frame in the sequence). |
### Vector labels (bounding box, polygon, polyline, keypoint)
Format of the `attributes` field in [`client.get_label()`](https://sdkdocs.segments.ai/en/latest/client.html#get-a-label):
```jsonp
{
"format_version": "0.2",
"frames": [
{ ... },
{ ... },
{ ... }
],
"links": [
{
"from_id": 1,
"to_id": 2,
"attributes": {} // the link attributes
}
]
}
```
Where each frames object has the following format:
```jsonp
{
"format_version": "0.1",
"timestamp": "00001", // In nanoseconds. This field is only included if the sample has a timestamp
"annotations": [
{
"track_id": 1, // the track id. A single object has the same track_id across frames.
"id": 1, // this is a legacy field and is always equal to track_id.
"category_id": 1, // the category id
"is_keyframe": true, // whether this frame is a keyframe
"type": "bbox", // refers to the annotation type (bounding box)
"points": [
[12.34, 56.78], // x0, y0 (upper left corner of bbox)
[90.12, 34.56] // x1, y1 (lower right corner of bbox)
]
},
{
"track_id": 2,
"id": 2,
"category_id": 2,
"is_keyframe": true,
"type": "polygon", // refers to the annotation type (polygon)
"points": [
[12.34, 56.78], // x0, y0 (starting point of the polygon)
[90.12, 34.56], // x1, y1
[78.91, 23.45], // x2, y2
[67.89, 98.76], // x3, y3
[54.32, 10.01] // x4, y4
]
},
{
"track_id": 3,
"id": 3,
"category_id": 3,
"is_keyframe": true,
"type": "polyline", // refers to the annotation type (polyline)
"points": [
[12.34, 56.78], // x0, y0 (starting point of the polyline)
[90.12, 34.56], // x1, y1
[78.91, 23.45], // x2, y2
[67.89, 98.76], // x3, y3
[54.32, 10.01] // x4, y4
]
},
{
"track_id": 4,
"id": 4,
"category_id": 4,
"is_keyframe": true,
"type": "point", // refers to the annotation type (keypoint)
"points": [
[12.34, 56.78] // x, y (coordinates of keypoint)
]
},
],
}
```
## 3D point cloud
### Segmentation label
The annotations array contains the different objects ("annotations") in the label with their category (the `category_id` should correspond to an id defined in the [categories](/reference/categories-and-attributes#categories)).
The `point_annotations` array contains the object/annotation id for each point in the point cloud. The order of the ids in this array is the same as the order of the points in the point cloud.
```json
{
"format_version": "0.1",
"annotations": [
{
"track_id": 1, // the track id. A single object has the same track_id across frames.
"id": 1, // this is a legacy field and is always equal to track_id.
"category_id": 1 // the category id
},
{
"track_id": 2,
"id": 2,
"category_id": 1
},
{
"track_id": 3,
"id": 3,
"category_id": 4
}
],
"point_annotations": [0, 0, 0, 3, 2, 2, 2, 1, 3...], // refers to track ids
}
```
### Cuboid label
```jsonp
{
"format_version": "0.2",
"annotations": [ // list of cuboid annotations, see below
{
"track_id": 1, // the track id. A single object has the same track_id across frames.
"id": 1, // this is a legacy field and is always equal to track_id
"type": "cuboid",
...
},
{
...
}
],
"links": [
{
"from_id": 1,
"to_id": 2,
"attributes": {} // the link attributes
}
]
}
```
| Name | Type | Description |
| ---------------- | --------------------------------------------------- | ------------------------------- |
| `format_version` | `string` | Format version. |
| `annotations` | `array` of [cuboid annotations](#cuboid-annotation) | List of the cuboid annotations. |
| `links` | `array` of [link annotations](#links) | List of the links |
### Cuboid annotation
A cuboid annotation represents a single cuboid in a point cloud (frame).
```json
{
"track_id": 1, // the track id. A single object has the same track_id across frames.
"id": 1, // this is a legacy field and is always equal to track_id
"category_id": 1,
"type": "cuboid",
"position": {
"x": 0.0,
"y": 0.2,
"z": 0.5
},
"dimensions": {
"x": 1.2,
"y": 1,
"z": 1
},
"yaw": 0.63,
"rotation": {
"qx": 0,
"qy": 0.0491566,
"qz": 0.3096865,
"qw": 0.9495672
}, // only when 3D cuboid rotation is enabled in dataset settings
"is_keyframe": true, // only in sequences
"index": 0, // only in sequences
}
```
| Name | Type | Description |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id` | `integer` | Object id. |
| `category_id` | `integer` | Category id. |
| `type` | `string` | Object type, which is always "cuboid" for cuboid annotations. |
| `position` |
object: { "x": float, "y": float, "z": float }
| XYZ position of the center of the cuboid in world coordinates. |
| `dimensions` |
object: { "x": float, "y": float, "z": float }
| Dimensions of the cuboid. "x" corresponds to the length, "y" to the width, and "z" to the height. See diagram 1. |
| `yaw` | `float` | Cuboid rotation along the z-axis in radians between \[-π, π]. 0 yaw corresponds to a cuboid aligned with the x-axis pointing to increasing x-values. The yaw value increases with a counter-clockwise rotation up to π, and decreases with a clockwise rotation up to -π. See diagram 2. |
| `rotation` |
object: { "qx": float, "qy": float, "qz": float,
"qw": float }
| Cuboid rotation defined as a [rotation quaternion](https://danceswithcode.net/engineeringnotes/quaternions/quaternions.html). Only available when 3D rotation is enabled in the dataset settings (under "Labeling"). |
| `track_id` | `integer` | Track ID of the object. This ID is used to track an object over multiple frames. Only relevant for sequences. |
| `is_keyframe` | `boolean` | Whether this cuboid annotation is a keyframe or an interpolated frame. Only relevant for sequences. |
| `index` | `integer` | The frame index. Only relevant for sequences. |
Diagram 2: yaw rotation of a cuboid. The red arrow shows the cuboid heading. yaw = π/2 corresponds to a heading in the direction of increasing y values, while yaw = -π/2 corresponds to a heading in the direction of decreasing y values.
### Vector label (polygon, polyline, keypoint)
```json
"format_version": "0.1",
"annotations": [
{
"track_id": 1, // the track id. A single object has the same track_id across frames.
"id": 1, // this is a legacy field and is always equal to track_id
"category_id": 2,
"type": "polygon", // refers to the annotation type (polygon)
"points": [
[12.34, 56.78, 0], // x0, y0, z0 (starting point of the polygon)
[90.12, 34.56, 0], // x1, y1, z1
[78.91, 23.45, 0], // x2, y2, z2
[67.89, 98.76, 0], // x3, y3, z3
[54.32, 10.01, 0] // x4, y4, z4
],
"is_keyframe": true, // only in sequences
"index": 0 // only in sequences
},
{
"track_id": 2,
"id": 2,
"category_id": 3,
"type": "polyline", // refers to the annotation type (polyline)
"points": [
[12.34, 56.78, 0], // x0, y0, z0 (starting point of the polyline)
[90.12, 34.56, 0], // x1, y1, z1
[78.91, 23.45, 0], // x2, y2, z2
[67.89, 98.76, 0], // x3, y3, z3
[54.32, 10.01, 0] // x4, y4, z4
],
"is_keyframe": false, // only in sequences
"index": 1 // only in sequences
},
{
"track_id": 3,
"id": 3,
"category_id": 4,
"type": "point", // refers to the annotation type (keypoint)
"points": [
[12.34, 56.78, 0] // x, y, z (coordinates of keypoint)
],
"is_keyframe": false, // only in sequences
"index": 2 // only in sequences
}
]
}
```
## 3D point cloud sequence
### Segmentation label
```jsonp
{
"format_version": "0.2",
"frames": [
{ ... },
{ ... },
{ ... }
]
}
```
Where each frames object has the following format:
```jsonp
{
"format_version": "0.2",
"annotations": [
{
"track_id": 1, // the track id. A single object has the same track_id across frames.
"id": 1, // this is a legacy field and is always equal to track_id
"category_id": 1, // the category id
},
{
"track_id": 2,
"id": 2,
"category_id": 1,
},
{
"track_id": 3,
"id": 3,
"category_id": 4,
},
],
"point_annotations": [0, 0, 0, 3, 2, 2, 2, 1, 3...], // refers to object ids
}
```
### Cuboid label
```jsonp
{
"format_version": "0.2",
"frames": [
{ ... },
{ ... },
{ ... }
]
}
```
| Name | Type | Description |
| ---------------- | ----------------------------------------- | ------------------------------------------------------ |
| `format_version` | `string` | Format version. |
| `frames` | `array` of [cuboid labels](#cuboid-label) | List of cuboid labels (one per frame in the sequence). |
### Vector label (polygon, polyline, keypoint)
Format of the `attributes` field in [`client.get_label()`](https://sdkdocs.segments.ai/en/latest/client.html#get-a-label):
```json
{
"format_version": "0.2",
"frames": [
{ ... },
{ ... },
{ ... }
],
"links": [
{
"from_id": 1,
"to_id": 2,
"attributes": {} // the link attributes
}
]
}
```
| Name | Type | |
| ---------------- | ------------------------------------------------------------------- | ------------------------------------------------------ |
| `format_version` | `string` | Format version. |
| `frames` | `array` of [vector labels](#vector-label-polygon-polyline-keypoint) | List of vector labels (one per frame in the sequence). |
| `links` | `array` of [link annotations](#links) | List of the links |
## Multi-sensor sequence
{% hint style="warning" %}
When uploading pre-labels for multi-sensor samples, make sure that the order of the sensors is the same in the label as in the sample. It is **not** sufficient for the sensors to have the same name, the order must also match.
{% endhint %}
```json
{
"sensors": [
{
"name": "Lidar",
"task_type": "pointcloud-cuboid-sequence",
"attributes": { ... }
},
{
"name": "Camera 1",
"task_type": "image-vector-sequence",
"attributes": { ... }
},
...
],
"links": [
{
"from_id": 1,
"to_id": 2,
"attributes": {} // the link attributes
}
]
}
```
| Name | Type | Description |
| --------- | ------------------------------------- | ------------------------------------------ |
| `sensors` | `array` of [sensors](#sensor) | List of the sensors with label attributes. |
| `links` | `array` of [link annotations](#links) | List of the links |
#### Sensor
| Name | Type | Description |
| ------------ | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | `string` | Name of the sensor. |
| `task_type` | `string` | The [task type](/reference/sample-and-label-types) of the sensor. Currently, `pointcloud-cuboid-sequence` and `image-vector-sequence` are supported. |
| `attributes` | `object` | The label attributes for the sensor. Currently, [3D point cloud sequence](#3d-point-cloud-sequence) and [image sequence](#image-sequence) are supported. |
## Object attributes
Objects in the annotations list can optionally also contain an attributes field to store object-level attributes. Make sure to properly [configure the label editor](/guides/configure-label-editor) if you're using object-level attributes.
```javascript
{
"format_version": "0.1",
"annotations": [
{
"id": 1,
"category_id": 1,
"attributes": { // object-level attributes
"is_crowd": "1",
"color": "red"
}
},
{
"id": 2,
"category_id": 1,
"attributes": {
"is_crowd": "0",
"color": "blue"
}
},
{
"id": 3,
"category_id": 4,
"attributes": {
"is_crowd": "1",
"color": "yellow"
}
}
],
...
}
```
## Image attributes
You can also define image-level attributes. These can be useful in image classification tasks. Make sure to properly [configure the label editor](/guides/configure-label-editor) if you're using image-level attributes.
```javascript
{
"format_version": "0.1",
"annotations": [...],
"image_attributes": { // sample-level attributes
"scene_type": "crossroads",
"weather": "sunny"
}
}
```
### Links
The links are stored at the top level of the annotations.
```
[
{
"from_id": 1,
"to_id": 2,
"attributes": {
"side": "left",
"strength": 10
}
]
```
| Name | Type | Description |
| ------------ | -------- | ------------------------------- |
| `from_id` | `number` | Link's object/track origin |
| `to_id` | `number` | Link's object/track destination |
| `attributes` | `object` | The link attributes. |
# Categories and attributes
When [editing the category and task attribute configuration directly](https://docs.segments.ai/guides/configure-label-editor), you need to adhere to the following format:
## Configuration format
```json
{
"format_version": "0.1",
"categories": [
{ ... },
{ ... },
{ ... }
],
"image_attributes": [ // optional image-level attributes
{ ... },
{ ... },
{ ... }
],
"circle_radius": 50, // optional ego circle in 3D vector interfaces
"warning_rules": [ // optional set of rules that trigger warnings
{ ... },
{ ... },
{ ... }
]
}
```
| Name | Type | Description |
| ------------------ | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `categories` | `array` of [categories](#category) | **Required.** List of all possible categories for a label in this dataset. |
| `image_attributes` | `array` of [attributes](#attribute) | List of image-level attributes. |
| `circle_radius` | `int` | The radius of the circle around the ego position in the 3D cuboid and vector interface. Can be used to indicate a region in which objects should be annotated. |
| `warning_rules` | `array` of [warning rules](#warning-rule) | List of rules that should generate warnings running a check on a sample |
{% hint style="warning" %}
The `categories` array should contain at least one category.
{% endhint %}
### Category
```json
{
"name": "car",
"id": 1,
"color": [33, 138, 33], // optional
"has_instances": true, // optional
"lock_dimensions": true, // optional, false by default
"lock_rotation": true, // optional, false by default
"lock_position": true, // optional, false by default
"attributes": [ // optional object-level attributes
{ ... },
{ ... },
{ ... }
],
"link_attributes": [ // optional link attributes
{ ... },
{ ... },
{ ... }
],
"dimensions": { // optional, only valid in the point cloud cuboid editor
"x": 0.6564944386482239,
"y": 1.3789583444595337,
"z": 1.6037739515304565
},
"foo": "bar" // optional custom key-value pairs. These will be ignored.
}
```
| Name | Type | Description |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | `string` | **Required.** Name of the category. |
| `id` | `int` | **Required.** Index of the category. |
| `color` | `array` of 3 `float` values in \[0, 255] | RGB color of the category. |
| `has_instances` | `boolean` | Whether the category contains instances (person, car) or not (sky, road) |
| `attributes` | `array` of [attributes](#object-attribute-format) | List of object-level attributes. |
| `link_attributes` | `array` of [attributes](#object-attribute-format) | List of link attributes when the object is on the "from" side. |
| `dimensions` |
object: { "x": float, "y": float, "z": float }
| Default XYZ dimensions of a new cuboid. Only valid in the point cloud cuboid editor (see [3D point cloud cuboid interface](/how-to-annotate/label-3d-point-clouds/3d-point-cloud-cuboid-interface#create-a-cuboid-with-default-dimensions)). |
| ... | ... | Other key-value pairs can be supplied, but will be ignored. |
### Attribute
```json5
{
"name": "color",
"input_type": "select",
"values": [
"green",
"yellow",
"red"
],
"default_value": "red" // optional
"is_mandatory": true // optional
"is_track_level: true // for sequence interfaces, optional
}
```
Name
Type
Description
name
string
Required. Name of the attribute.
input_type
string: select | text | number | checkbox
Required. Type of the attribute.
values
array of strings
Required wheninput_typeisselect. List of possible values.
min
string
Valid when input_type is number. Minimum value the attribute can be.
max
string
Valid when input_type is number. Maximum value the attribute can be.
step
string
Valid when input_type is number. Step when incrementing/decrementing the value of the attribute.
default_value
string | boolean depending on input_type
Default value of the attribute.
is_mandatory
boolean
Valid when input_type is select, text or number. Whether the attribute is mandatory. Mandatory attributes raise a warning when not filled in.
is_track_level
boolean
Valid in sequence datasets. Whether an attribute should remain constant across all frames for an object with a certain track ID. If false, the attribute can change on each frame.
synced_across_sensors
boolean
Valid in multi-sensor datasets. Whether an attribute should remain constant across all sensors for an object with a certain track ID. If false, the attribute can change on each sensor.
sensors
string: 2D |3D| all
Valid in multi-sensor datasets. Whether an attribute applies to 2D sensors, 3D sensors, or all sensors.
#### Additional examples
```json
{
"name": "description",
"input_type": "text",
"default_value": "A nice car.", // optional
"is_mandatory": true // optional
},
```
```json
{
"name": "number_of_wheels",
"input_type": "number",
"min": "1", // optional
"max": "20", // optional
"step": "1", // optional
"default_value": 4, // optional
"is_mandatory": true // optional
},
```
```json
{
"name": "is_electric",
"input_type": "checkbox",
"default_value": false // optional
}
```
### Warning rule
Different warning rules can be configured. Each rule type can be added multiple times for different categories / category groups according to the rule specifications. Currently, the following list of options is allowed
* `intersecting-cuboids`
* `cuboid-dimension-limits`
Each rule has its own set of properties to fully configure it.
#### Intersecting cuboids
An intersecting cuboids rule should either have an `excluded_set_of_categories` prop or an `excluded_categories` prop, not both.
Name
Type
Description
name
string: intersecting-cuboids
required: Set this property to intersecting-cuboids when adding a rule to warn for intersecting cuboids.
excluded_set_of_categories
array of numbers
Add categories to this list that are allowed to intersect with each other. Note that intersections between elements of the exact same category will still raise warnings.
excluded_categories
array of numbers
Add categories to this list that should not trigger warnings when they intersect with any category (including their own category).
```json
// Activate the rule without exclusions
{
"name": "intersecting-cuboids",
"excluded_categories": [],
}
// Activate the rule, but exclude intersections between categories 1 and 3, 1 and 5,
// and between 3 and 5 from raising warnings
{
"name": "intersecting-cuboids",
"excluded_set_of_categories": [1, 3, 5],
}
// Activate the rule, but don't raise warnings when category 3 intersects
{
"name": "intersecting-cuboids",
"excluded_categories": [3],
}
```
#### Cuboid dimension limits
An intersecting cuboids rule should either have an `excluded_set_of_categories` prop or an `excluded_categories` prop, not both.
Name
Type
Description
name
string: cuboid-dimension-limits
required: Set this property to cuboid-dimension-limits when adding a rule to warn for invalid dimensions on cuboids.
categories
array of numbers
Add categories to this list for which this rule should apply. Pass an empty array to apply this rule to all categories.
Set the maximum dimensions allowed for cuboids. It is not required to set all x, y and z values. Only set the values that you want to be warned for.
```json
// Activate the rule and warn for any cuboid with an x value larger than 10
{
"name": "cuboid-dimension-limits",
"max" {
"x": 10
}
}
// Activate the rule and warn for any cuboid in category 1 or 2
// with an x or y value smaller than or a z value larger than 3
{
"name": "cuboid-dimension-limits",
"categories": [1, 2],
"min": {
"x": 0.5,
"y": 0.5
},
"max" {
"x": 3
}
}
```
{% hint style="warning" %}
Note that the inline comments in the examples should be left out, as comments of the form`//…` or `/*…*/` are not allowed in JSON.
{% endhint %}
# API
The API is available at \*\*;
To authenticate, add an API key in the header of each request:
```bash
curl -H "Authorization: APIKey YOUR_API_KEY"
```
An API key can be created on your [user account page](https://segments.ai/account).
## Datasets
### List datasets
```bash
GET /users/:owner/datasets
```
To get all datasets of the currently logged in users, you can use this shortcut:
```bash
GET /user/datasets
```
{% hint style="info" %}
Note that this will only return datasets which are public, and datasets which are private but where the logged in user is a collaborator.
{% endhint %}
#### Response
{% code title="Status: 200 OK" %}
```bash
[
{
"name": "cats",
"description": "A dataset of cat images.",
"data_type": "IMAGE",
"category": "other",
"public": false,
"owner": {
"username": "bert",
"email": "bert@segments.ai",
"created_at": "2020-05-11T14:00:53.763278Z"
},
"created_at": "2020-04-10T20:09:31Z",
"collaborators_count": 0,
"samples_count": 94
}
]
```
{% endcode %}
### Get a dataset
```bash
GET /datasets/:owner/:dataset
```
#### Response
{% code title="Status: 200 OK" %}
```bash
{
"name": "cats",
"description": "A dataset of cat images.",
"category": "other",
"public": false,
"owner": {
"username": "bert",
"created_at": "2020-05-11T14:00:53.763278Z"
},
"created_at": "2020-07-20T14:59:36.242218Z",
"collaborators_count": 0,
"samples_count": 94,
"task_type": "segmentation-bitmap",
"task_attributes": {
"format_version": "0.1",
"categories": [
{
"name": "cat",
"id": 0
},
{
"name": "dog",
"id": 1
}
]
},
"labelsets": [
{
"name": "ground-truth",
"description": "Ground truth labels.",
},
{
"name": "predictions",
"description": "My model predictions.",
},
]
}
```
{% endcode %}
### Create a dataset
```bash
POST /user/datasets
```
#### Input
Name
Type
Description
Name
Type
Description
name
string
Required. The name of the dataset.
category
string
Required. Category of the data, e.g. "other"
task_type
string
Required. The task type of the dataset. Can be one of:
segmentation-bitmap: Semantic, panoptic and instance segmentation.