sticky_pi_api package

Subpackages

• sticky_pi_api.database package
  • Submodules
  • sticky_pi_api.database.images_table module
  • sticky_pi_api.database.uid_annotations_table module
  • sticky_pi_api.database.users_tables module
  • sticky_pi_api.database.utils module
  • Module contents

Submodules

sticky_pi_api.client module

Main module of the client. Implements classes to interact with the API.

class sticky_pi_api.client.BaseClient(local_dir, n_threads=8, *args, **kwargs)

Bases: sticky_pi_api.specifications.BaseAPISpec

Abstract class that defines the methods of the client (common between remote and client).

Parameters

• local_dir (str) – The path to a client directory that acts as a client storage

• n_threads (int) – The number of parallel threads to use to compute statistics on the image (md5 and such)

delete_cache()

get_images_with_uid_annotations_series(info, what_image='metadata', what_annotation='metadata')

Retrieves images alongside their annotations (if available) for images from a given device and within a given datetime range

Parameters

• info (List[Dict[str, str]]) – A list of dicts. each dicts has, at least, the keys: 'device', 'start_datetime' and 'end_datetime'

• what_image (str) – The nature of the image objects to retrieve. One of {'metadata', 'image', 'thumbnail', 'thumbnail_mini'}

• what_annotation (str) – The nature of the object to retrieve. One of {'metadata', 'json'}.

Return type

List[Dict[str, Union[float, int, str]]]

Returns

A list of dictionaries with one element for each queried value. Each dictionary contains the fields present in the underlying database tables (see UIDAnnotations and Images).

get_ml_bundle_dir(bundle_name, bundle_dir, what)

Return type

List[Dict[str, Union[float, str]]]

get_uid_annotations_series(info, what='metadata')

Retrieves annotations for images within a given datetime range

Parameters

• info (List[Dict[str, str]]) – A list of dicts. each dicts has, at least, the keys: 'device', 'start_datetime' and 'end_datetime'

• what (str) – The nature of the object to retrieve. One of {'metadata', 'json'}.

Return type

List[Dict[str, Union[float, int, str]]]

Returns

A list of dictionaries with one element for each queried value. Each dictionary contains the fields present in the underlying database table (see UIDAnnotations). In the case of what='metadata', the field json=''. Otherwise, it contains a json string with the actual annotation data.

property local_dir

put_images(files)

Incrementally upload a list of client files

Parameters

files (List[str]) – the paths to the client files

Return type

List[Dict[str, Union[float, int, str]]]

Returns

the data of the uploaded files, as represented in by API

put_ml_bundle_dir(bundle_name, bundle_dir, what='all')

Return type

List[Dict[str, Union[float, str]]]

class sticky_pi_api.client.LocalClient(local_dir, n_threads=8, *args, **kwargs)

Bases: sticky_pi_api.client.BaseClient, sticky_pi_api.specifications.LocalAPI

Abstract class that defines the methods of the client (common between remote and client).

Parameters

• local_dir (str) – The path to a client directory that acts as a client storage

• n_threads (int) – The number of parallel threads to use to compute statistics on the image (md5 and such)

sticky_pi_api.configuration module

class sticky_pi_api.configuration.BaseAPIConf(env_file=None, **kwargs)

Bases: dict

class sticky_pi_api.configuration.LocalAPIConf(env_file=None, **kwargs)

Bases: sticky_pi_api.configuration.BaseAPIConf

class sticky_pi_api.configuration.RequiredConfVar

Bases: object

sticky_pi_api.image_parser module

class sticky_pi_api.image_parser.ImageParser(file)

Bases: dict

A class derived from dict that contains image metadata in its fields. It parses data from an input JPEG image file taken by a Sticky Pi and retrieves its metadata from filename an exif fields. In addition, it computes md5 sum and generate thumbnails for the input image. :param file: path to file or file like object

property file_blob

property filename

property thumbnail

property thumbnail_mini

sticky_pi_api.specifications module

class sticky_pi_api.specifications.BaseAPI(api_conf, *args, **kwargs)

Bases: sticky_pi_api.specifications.BaseAPISpec

get_image_series(info, what='metadata')

Retrieves image sequences (i.e. series). A series contains all images from a given device within a datetime range.

Parameters

• info (List[Dict[str, Union[float, int, str]]]) – A list of dicts. each dicts has, at least, the keys: 'device', 'start_datetime' and 'end_datetime'. device is interpreted to the MySQL like operator. For instance,one can match all devices with device="%".

• what (str) – The nature of the objects to retrieve. One of {'metadata', 'image', 'thumbnail', 'thumbnail_mini'}

Returns

A list of dictionaries with one element for each queried value. Each dictionary contains the fields present in the underlying database plus a 'url' fields to retrieve the actual object requested (i.e. the what) argument. In the case of what='metadata', url='' (i.e. no url is generated).

get_images(info, what='metadata')

Retrieves information about a given set of images, defined by their parent device and the datetime of the picture. If an image is not available, no data is returned for this image.

Parameters

• info (List[Dict[str, Union[float, int, str]]]) – A list of dicts. each dicts has, at least, keys: 'device' and 'datetime'

• what (str) – The nature of the objects to retrieve. One of {'metadata', 'image', 'thumbnail', 'thumbnail_mini'}

Returns

A list of dictionaries with one element for each queried value. Each dictionary contains the fields present in the underlying database plus a 'url' fields to retrieve the actual object requested (i.e. the what) argument. In the case of what='metadata', url='' (i.e. no url is generated).

get_uid_annotations(info, what='metadata')

Reteives annotations for a given set of images.

Parameters

• info (List[Dict[str, Union[float, int, str]]]) – A list of dict with keys: 'device' and 'datetime'

• what (str) – The nature of the object to retrieve. One of {'metadata', 'json'}.

Returns

A list of dictionaries with one element for each queried value. Each dictionary contains the fields present in the underlying database table (see UIDAnnotations). In the case of what='metadata', the field json=''. Otherwise, it contains a json string with the actual annotation data.

get_users(info=None)

Get a list of API users. Either all users (Default), or filter users by field if info is specified. In the latter case, the union of all matched users is returned.

Parameters

info (Optional[Dict[str, str]]) – A dictionary acting as a filter, using an SQL like-type match. For instance {'username': '%'} return all users.

Return type

List[Dict[str, Any]]

Returns

A list of users as represented in the underlying database, as one dictionary [per user, with the keys being database column names. Note that the crypo/sensitive fields are not returned (e.g. password_hash)

put_uid_annotations(info)

Parameters

info (List[Dict[str, Union[List, Dict[str, Any]]]]) –

A list of dictionaries corresponding to annotations (one list element per image). The annotations are formatted as a dictionaries with two keys: 'annotations' and 'metadata'.

• 'metadata' must have the fields:

  • 'algo_name': the name of the algorithm used to find the object (e.g. 'sticky-pi-universal-insect-detector')

  • 'algo_version': The version of the algorithm as timestamp-md5 (e.g. '1598113346-ad2cd78dfaca12821046dfb8994724d5')

  • 'device': The device that took the annotated image (e.g. '5c173ff2')

  • 'datetime': The datetime at which the image was taken (e.g. '2020-06-20_21-33-24')

  • 'md5': The md5 of the image that was analysed (e.g. '9e6e908d9c29d332b511f8d5121857f8')

• 'annotations' is a list where each element represent an object. It has the fields:

  • 'contour': a 3d array encoding the position of the vertices (as convention in OpenCV)

  • 'name': the name/type of the object (e.g. 'insect')

  • 'fill_colour' and 'stroke_colour': the colours of the contour (if it is to be drawn – e.g. '#0000ff')

  • 'value': an optional integer further describing the contour (e.g. 1)

Returns

The metadata of the uploaded annotations (i.e. a list od dicts. each field of the dict naming a column in the database). This corresponds to the annotation data as represented in UIDAnnotations

put_users(info)

Add a list of users defined by a dict of proprieties.

Parameters

info (List[Dict[str, Any]]) – A list of dictionary each dictionary has the fields {'username', 'password'}, and optionally: {'email', 'is_admin',``’model’`` },

Return type

List[Dict[str, Any]]

Returns

A list of dictionaries describing the users that were created

class sticky_pi_api.specifications.BaseAPISpec

Bases: object

get_image_series(info, what='metadata')

Retrieves image sequences (i.e. series). A series contains all images from a given device within a datetime range.

Parameters

• info – A list of dicts. each dicts has, at least, the keys: 'device', 'start_datetime' and 'end_datetime'. device is interpreted to the MySQL like operator. For instance,one can match all devices with device="%".

• what (str) – The nature of the objects to retrieve. One of {'metadata', 'image', 'thumbnail', 'thumbnail_mini'}

Return type

List[Dict[str, Union[float, int, str]]]

Returns

A list of dictionaries with one element for each queried value. Each dictionary contains the fields present in the underlying database plus a 'url' fields to retrieve the actual object requested (i.e. the what) argument. In the case of what='metadata', url='' (i.e. no url is generated).

get_images(info, what='metadata')

Retrieves information about a given set of images, defined by their parent device and the datetime of the picture. If an image is not available, no data is returned for this image.

Parameters

• info (List[Dict[str, str]]) – A list of dicts. each dicts has, at least, keys: 'device' and 'datetime'

• what (str) – The nature of the objects to retrieve. One of {'metadata', 'image', 'thumbnail', 'thumbnail_mini'}

Return type

List[Dict[str, Union[float, int, str]]]

Returns

A list of dictionaries with one element for each queried value. Each dictionary contains the fields present in the underlying database plus a 'url' fields to retrieve the actual object requested (i.e. the what) argument. In the case of what='metadata', url='' (i.e. no url is generated).

get_uid_annotations(info, what='metadata')

Reteives annotations for a given set of images.

Parameters

• info (List[Dict[str, str]]) – A list of dict with keys: 'device' and 'datetime'

• what (str) – The nature of the object to retrieve. One of {'metadata', 'json'}.

Return type

List[Dict[str, Union[float, int, str]]]

Returns

A list of dictionaries with one element for each queried value. Each dictionary contains the fields present in the underlying database table (see UIDAnnotations). In the case of what='metadata', the field json=''. Otherwise, it contains a json string with the actual annotation data.

get_users(info=None)

Get a list of API users. Either all users (Default), or filter users by field if info is specified. In the latter case, the union of all matched users is returned.

Parameters

info (Optional[Dict[str, str]]) – A dictionary acting as a filter, using an SQL like-type match. For instance {'username': '%'} return all users.

Return type

List[Dict[str, Any]]

Returns

A list of users as represented in the underlying database, as one dictionary [per user, with the keys being database column names. Note that the crypo/sensitive fields are not returned (e.g. password_hash)

put_uid_annotations(info)

Parameters

info (List[Dict[str, Union[List, Dict[str, Any]]]]) –

A list of dictionaries corresponding to annotations (one list element per image). The annotations are formatted as a dictionaries with two keys: 'annotations' and 'metadata'.

• 'metadata' must have the fields:

  • 'algo_name': the name of the algorithm used to find the object (e.g. 'sticky-pi-universal-insect-detector')

  • 'algo_version': The version of the algorithm as timestamp-md5 (e.g. '1598113346-ad2cd78dfaca12821046dfb8994724d5')

  • 'device': The device that took the annotated image (e.g. '5c173ff2')

  • 'datetime': The datetime at which the image was taken (e.g. '2020-06-20_21-33-24')

  • 'md5': The md5 of the image that was analysed (e.g. '9e6e908d9c29d332b511f8d5121857f8')

• 'annotations' is a list where each element represent an object. It has the fields:

  • 'contour': a 3d array encoding the position of the vertices (as convention in OpenCV)

  • 'name': the name/type of the object (e.g. 'insect')

  • 'fill_colour' and 'stroke_colour': the colours of the contour (if it is to be drawn – e.g. '#0000ff')

  • 'value': an optional integer further describing the contour (e.g. 1)

Return type

List[Dict[str, Union[float, int, str]]]

Returns

The metadata of the uploaded annotations (i.e. a list od dicts. each field of the dict naming a column in the database). This corresponds to the annotation data as represented in UIDAnnotations

put_users(info)

Add a list of users defined by a dict of proprieties.

Parameters

info (List[Dict[str, Any]]) – A list of dictionary each dictionary has the fields {'username', 'password'}, and optionally: {'email', 'is_admin',``’model’`` },

Return type

List[Dict[str, Any]]

Returns

A list of dictionaries describing the users that were created

class sticky_pi_api.specifications.LocalAPI(api_conf, *args, **kwargs)

Bases: sticky_pi_api.specifications.BaseAPI

sticky_pi_api.storage module

class sticky_pi_api.storage.BaseStorage(api_conf, *args, **kwargs)

Bases: object

get_ml_bundle_file_list(bundle_name, what='all')

List and describes the files present in a ML bundle.

Parameters

• bundle_name (str) – the name of the machine learning bundle to fetch the files from

• what (str) – One of {'all', 'data',``’model’`` }, to return all files, only the training data(training), or only the model (inference), respectively.

Return type

List[Dict[str, Union[float, str]]]

Returns

A list of dict containing the fields key and url of the files to be downloaded, which can be used to download the files

get_ml_bundle_upload_links(bundle_name, info)

Request a list of upload links to put files in a given ML bundle

Parameters

• bundle_name (str) –

• info (List[Dict[str, Union[float, str]]]) – A list of dict containing the fields key, md5 mtime describing the upload candidates.

Return type

List[Dict[str, Union[float, str]]]

Returns

A list like info with the extra key url pointing to a destination where the file can be copied/posted. The list contains only files that did not exist on remote – hence can be empty.

get_url_for_image(image, what='metadata')

Retrieves the URL to the file corresponding to an image in the database.

Parameters

• image (Images) – an image object

• what (str) – One of {'metadata', 'image', 'thumbnail', 'thumbnail_mini'}

Return type

str

Returns

a url/path as a string. For what='metadata', an empty string is returned. for consistency

store_image_files(image)

Saves the files corresponding to a an image. Those are generally the original JPEG plus thumbnail and thumbnail-mini

Parameters

image (Images) – an image object

Return type

None

class sticky_pi_api.storage.DiskStorage(api_conf, *args, **kwargs)

Bases: sticky_pi_api.storage.BaseStorage

get_ml_bundle_file_list(bundle_name, what='all')

List and describes the files present in a ML bundle.

Parameters

• bundle_name (str) – the name of the machine learning bundle to fetch the files from

• what (str) – One of {'all', 'data',``’model’`` }, to return all files, only the training data(training), or only the model (inference), respectively.

Return type

List[Dict[str, Union[float, str]]]

Returns

A list of dict containing the fields key and url of the files to be downloaded, which can be used to download the files

get_ml_bundle_upload_links(bundle_name, info)

Request a list of upload links to put files in a given ML bundle

Parameters

• bundle_name (str) –

• info (List[Dict[str, Union[float, str]]]) – A list of dict containing the fields key, md5 mtime describing the upload candidates.

Return type

List[Dict[str, Union[float, str]]]

Returns

A list like info with the extra key url pointing to a destination where the file can be copied/posted. The list contains only files that did not exist on remote – hence can be empty.

get_url_for_image(image, what='metadata')

Retrieves the URL to the file corresponding to an image in the database.

Parameters

• image (Images) – an image object

• what (str) – One of {'metadata', 'image', 'thumbnail', 'thumbnail_mini'}

Return type

str

Returns

a url/path as a string. For what='metadata', an empty string is returned. for consistency

store_image_files(image)

Saves the files corresponding to a an image. Those are generally the original JPEG plus thumbnail and thumbnail-mini

Parameters

image (Images) – an image object

Return type

None

sticky_pi_api.types module

sticky_pi_api.utils module

sticky_pi_api.utils.datetime_to_string(dt)

sticky_pi_api.utils.local_bundle_files_info(bundle_dir, what='all', allowed_ml_bundle_suffixes=('.yaml', '.yml', 'model_final.pth', '.svg', '.jpeg', '.jpg'), ml_bundle_ml_data_subdir=('data', 'config'), ml_bundle_ml_model_subdir=('output', 'config'), multipart_chunk_size=8388608)

sticky_pi_api.utils.md5(file, chunk_size=32768)

sticky_pi_api.utils.multipart_etag(file, chunk_size)

sticky_pi_api.utils.string_to_datetime(string)

Module contents

this is the __init__ of sticky_pi_api