Guide #05: custom neural network workflow

iPython Project Created 4 days ago Free
How to run plugins (DTL/train/inference) from code to organize and automate custom workflow: training data preparation -> training -> inference
Free Signup

Supervisely Tutorial #5

Neural networks: training workflow with Supervisely online API

In this tutorial we will show how use Supervisely online API to perform a full cycle of preparing neural network training data, train a model and then run inference on a test dataset.

Setup steps

Before we can start issuing inference requests, we need to connect to the Supervisely web instance, make sure the model we need is available and set up a worker machine to load the model on.

Necessary imports

Simply import the Supervisrly Python SDK module:

In [1]:
import supervisely_lib as sly

Initialize API access with your credentials

Before starting to interact with a Supervisely web instance using our API, you need to supply your use credentials: your unique access token that you can find under your profile details:

In [2]:
import os

# Jupyter notebooks hosted on Supervisely can get their user's
# credentials from the environment varibales.
# If you are running the notebook outside of Supervisely, plug
# the server address and your API token here.
# You can find your API token in the account settings:
# -> click your name in the top-right corner
# -> select "account settings"
# -> select "API token" tab on top.
address = os.environ['SERVER_ADDRESS']
token = os.environ['API_TOKEN']

print("Server address: ", address)
print("Your API token: ", token)

# Initialize the API access object.
api = sly.Api(address, token)
Out [2]:
Server address:
Your API token:  2lS1r2jGvNJrPErjUESrcK77NBCWI7lotHvQqMgMMfFAtJfuglxU3OQBiWSNlyb6dUAcdTTGm8YqeimRwXMGZUGerD5j2pAp6JjW6TF9s3aBjia60qTQyWkJgSpa7Gwt

Define the active workspace

In Supervisely, every neural network model (and also every data project) is stored in a context of a certain workspace. See our tutorial #2 for a detailed guide on how to work with workspaces using our online API.

Here we will create a new workspace to avoid interfering with any existing work.

In [3]:
# In Supervisely, a user can belong to multiple teams.
# Everyone has a default team with just their user in it.
# We will work in the context of that default team.
team =[0]

# Set up the name of a new workspace to be created.
workspace_name = "api_training_tutorial"

# Just in case there is already a workspace with this name,
# we can ask the web instance for a new unique name to use.
if api.workspace.exists(, workspace_name):
    workspace = api.workspace.get_info_by_name(, workspace_name)
    workspace = api.workspace.create(, workspace_name)

# Print out the results.
# Here we will see which name our workspace ended up with.
print("Team: id={}, name={}".format(,
print("Workspace: id={}, name={}".format(,
Out [3]:
Team: id=7, name=antonc
Workspace: id=106, name=api_training_tutorial

Add a neural network to the workspace to be a starting point for training

Training neural networks completely from scratch is done very rarely in practice. Instead, one uses an existing model, trained on a large dataset like ImageNet, as a starting point, and fine-tunes the weights for the specific task at hand. This way most of the initial network layers, extracting low-level features, can be reused almost without change, since low-level features typically transfer well between different tasks.

In this tutorial we will be basing our model on UNet trained on ImageNet dataset. This model is publically available out of the box in Supervisely, so we only need to clone it into our dataset:

In [4]:
# Set the destination model name within our workspace
model_name = "unet_vgg"

# Grab a unique name in case the one we chose initially is busy.
if api.model.exists(, model_name):
    model_name = api.model.get_free_name(, model_name)

# Request the model to be copied from our public repository.
# This kicks off an asynchronous task.
task_id = api.model.clone_from_explore('Supervisely/Model Zoo/UNet (VGG weights)',, model_name)

# Wait for the copying to complete.
api.task.wait(task_id, api.task.Status.FINISHED)

# Query the metadata for the copied model.
model = api.model.get_info_by_name(, model_name)
print("Model: id = {}, name = {!r}".format(,
Out [4]:
Model: id = 349, name = 'unet_vgg'

Select the agent to use

Neural network inference is a computationally intensive process, so it is infeasible to have the inference run on the same machine that serves the Supervisely web instance. Instead, you need to connect a worker machine (with a GPU) to the web instance to run the computations. The worker is connected using the Supervisely Agent - an open-source daemon that runs on the worker, connects to the web instance and listens for tasks to execute. See for details on how to run the agent.

From now on the tutorial assumes that you have launched the agent on your worker machine and it shows up on your "Cluster" page in the Supervisely web instance. We first query the instance for the agent ID by name.

In [5]:
# Replace this with your agent name. You can find the list of
# all your agents in the "Cluster" menu in the Supervisely instance.
agent_name = "Flabby Duck"

agent = api.agent.get_info_by_name(, agent_name)
if agent is None:
    raise RuntimeError("Agent {!r} not found".format(agent_name))
if agent.status is api.agent.Status.WAITING:
    raise RuntimeError("Agent {!r} is not running".format(agent_name))

Prepare a project with the training data

Here we will take a small project with labeled data, apply augmentations to it to increase the data variability and produce the resuling project with the training data.

Copy an existing labeled project

We will clone one of the publically available in Supervisely projects.

In [6]:
project_annotated_name = "lemons_annotated_copy"

# Grab a free project name if ours is taken.
if api.project.exists(, project_annotated_name):
    project_annotated_name = api.project.get_free_name(, project_annotated_name)

# Kick off the a project clone task and wait for completion.
task_id = api.project.clone_from_explore('Supervisely/Demo/lemons_annotated',, project_annotated_name)
api.task.wait(task_id, api.task.Status.FINISHED)

src_project = api.project.get_info_by_name(, project_annotated_name)
print("Project: id = {}, name = {!r}".format(,
Out [6]:
Project: id = 1102, name = 'lemons_annotated_copy'

Data augmentation using a Data Transformation Language plugin

To increase variaility of the training data and make the learned models more robust, a common approach is to perform augmentations. Augmentations are image transformations that preserve the essential appearance of the objects, like random crops or rotations. The transformed data is then used as training data for the model.

In Supervisely, one way to conveniently augment your data is to run a Data Transformation Language plugin with a set of predefined transformtations. The output of the plugin forms a new project, which we will then use for training the model. The DTL workflow will also tag images as train and val to distinguish between training and validation folds.

In [7]:
import json

project_train_name = "lemons_train"

# read graph template and define input/output projects
with open('./dtl_segmentation_graph.json', 'r') as file:
    dtl_graph_str =

# Plug in the source and destination project names in
# the DTL transformations definition.
dtl_graph_str = dtl_graph_str.replace('%SRC_PROJECT_NAME%', project_annotated_name)
dtl_graph_str = dtl_graph_str.replace('%DST_PROJECT_NAME%', project_train_name)

# Parse the JSON data with filled in project names.
dtl_graph = json.loads(dtl_graph_str)

# Run the DTL transformation unless we already have the output project
# (say in case you are executing this notebook multiple times).
task_id = None
if not api.project.exists(, project_train_name):
    # Kick off asynchronous DTL task.
    task_id = api.task.run_dtl(, dtl_graph,
    print('DTL task (id={}) is started'.format(task_id))
    # Wait for the task to complete.
    if task_id is not None:
        api.task.wait(task_id, api.task.Status.FINISHED)
# Inspect the results.
project_train = api.project.get_info_by_name(, project_train_name)
print("Training dataset {!r} contains {} images".format(,
Out [7]:
DTL task (id=1943) is started
Training dataset 'lemons_train' contains 72 images

Run neural network training

We are ready to fine-tune the original UNet model to find lemons and kiwis from our toy dataset:

In [8]:
# Select a name for the trained model.
trained_model_name = "nn_lemon_kiwi"

# Fill in the training config.
training_config = {
  "lr": 0.001,       # Learning rate
  "epochs": 10,      # Number of epochs (full iteration over training data) to use.
  "val_every": 0.5,  # Validate every 0.5 epoch (i.e. twice per epoch).
  "input_size": {    # Resize input images to this size before feeding the neural net.
    "width": 256,
    "height": 256
  "gpu_devices": [ 0 ],  # Use the first available GPU for training.
  "dataset_tags": {  # Use the images tagged as 'train' for training and 'val' for validation.
    "val": "val",
    "train": "train"
  "special_classes": {
    "background": "bg"
  # Transfer learning means we are not reusing the set of classes that the initial
  # model, and instead introducing a new set of classes, so the head (last layer)
  # of the model should be reinitialized.
  "weights_init_type": "transfer_learning"

task_id = api.task.run_train(,,, trained_model_name, training_config)
print('Train task (id={}) is started'.format(task_id))

api.task.wait(task_id, api.task.Status.FINISHED)
print('Train task (id={}) is finished'.format(task_id))

trained_model = api.model.get_info_by_name(, trained_model_name)
if trained_model is None:
    raise RuntimeError("Model {!r} not found".format(trained_model_name))

print("trained model: id = {}, name = {!r}".format(,
Out [8]:
Train task (id=1944) is started
Train task (id=1944) is finished
trained model: id = 350, name = 'nn_lemon_kiwi'

Run inference on the freshly trained model

Now we can run inference with the learned model. Here we only show a minimal overview. See Supervisely tutorial #4 for more details on online inference API.

Prepare an input dataset for inference

We will clone a publically available unlabeled project to run inference on.

In [12]:
project_test_name = "lemons_test_copy"

# Grab a free project name if ours is taken.
if api.project.exists(, project_test_name):
    project_test_name = api.project.get_free_name(, project_test_name)

# Kick off the a project clone task and wait for completion.
task_id = api.project.clone_from_explore('Supervisely/Demo/lemons_test',, project_test_name)
api.task.wait(task_id, api.task.Status.FINISHED)

project_test = api.project.get_info_by_name(, project_test_name)
print("Project: id = {}, name = {!r}".format(,
Out [12]:
Project: id = 1106, name = 'lemons_test_copy'

Run inference on the input project

In [13]:
# Name of destination project with the test results.
project_inf_name = "lemons_test_inf"

task_id = api.task.run_inference(,,, project_inf_name)
print('Inference task (id={}) is started'.format(task_id))

api.task.wait(task_id, api.task.Status.FINISHED)
print('Inference task (id={}) is finished'.format(task_id))
Out [13]:
Inference task (id=1948) is started
Inference task (id=1948) is finished


More Info

First released
4 days ago
Last updated
3 hours ago