Storages¶

Storages are the file systems that are indexed by and known to Cantemo and its underlying backends. Storages usually map to file storage that can be written to or read from, and are used for storing the media assets, sending files for editing or importing from by users.

Each storage can have one or more storage methods attribute, and this is the way that the system can access the storage and the files contained within it.

List View¶

_images/portal_storages.png — Storages overview page.¶

The storage list view can be reached from the admin menu, storages.

It will list all the storages known to Cantemo and its backends, plus a graphical status of how much space is left and what storage methods are configured for it.

Storage attributes¶

Storages have several configurable attributes.

Storage methods

Discussed in more detail below

Name

A reference name

Storage Group

See storage grouping below for more detail on groups, but this will decide what storage group the system will be in.

Capacity

This is the capacity that will be monitored and used. Please enter a positive integer. Its possible to auto-detect the size of the storage, but please not that this takes some minutes to calculate the total size correctly.

Low-Water Mark

Safe amount for this storage to utilize. See High-Water Mark.

High-Water Mark

The server will start deleting unneeded files (that are stored elsewhere) on the storage when the capacity reaches this mark, then it will try to copy and move files until it reaches the low-water mark.

Storage type

The storage type determines how files on the storage are read and written. The following is an explanation of the different options.

Local - This is the default storage type and files are both read and written to this storage.
Shared - This storage is shared with another application so files are only read from here. No files will be written to or removed from this storage.
Remote - Files on this storage will not be accessed directly and will be copied to a local storage before use.
External - This is a storage placeholder and this storage will not be used by the system. This type is not used by Cantemo.
Export - This storage can be used as an export location. Files which are written here cannot be copied back to other storages and files cannot be imported from this storage type.

Scan Interval

Configure how often the server scans this storage for new and modified files. The default is 60 seconds.

Refresh Interval

Configure how often the server does a complete scan of the entire storage, updating the metadata for each file. The default is 240 seconds.

Maximum Bandwidth

Configures how fast the server tries to read/write data to the storage. Can be used to throttle usage of a particular storage and is useful if the available network bandwidth exceeds the speed of the storage solution.

Exclude filters

The exclude filters allows the administrator to exclude certain file name patterns from appearing in the files list of a storage as well as excluding them from Auto Ingest. This can for example be used to exclude Apple resource forks. The default for new storages is a filter which matches all files and directories with a leading period character (these files are normally hidden in unix filesystems)

Importable

Files that are not imported on this storage can be view in the “Import From Storage” page usually located under the Ingest menu. This is also needed to show that this storage can be used to import from and when selecting auto ingest.

Auto Ingest

This flag denotes that any file that will be placed on this storage will after a short period of time be ingested into the system.

Storage grouping¶

Storages can be grouped together to be more representative of workflows or to help when organizing automated rules. To create a group, click on “Add Storage Group” and fill in the name and description.

Storages can then added to that group.

Adding storages¶

To add monitored storages:

Create the storage group that you wish to use for your storage and then click “Add New Storage” within that storage group.

Fill in the details of the storage. By default, a storage method of type file is added to a new storage, but this can be replaced before saving.
Click “Add Storage” to finish.

The storage will be added to the system and in a few minutes will be indexed for files that could possibly be ingested.

Adding an auto import storage¶

To add an auto ingest storage, check the Auto-ingest checkbox. Note that the Importable checkbox will automatically be checked as well as it is required for this type of storages. You will be presented with two additional drop downs.

Ingest Profile determines the transcode and access rule profile which should be applied to the ingested content. This reflects a profile defined for a certain group, and is in line with the ingest profile chosen upon an Upload or Import from Storage.

Upon an auto ingest, metadata can automatically be found if it shares the same filename and has the extension “.xml”, for example video.avi and video.xml. Metadata Transformation defines which (if any) transformation should be used for mapping the XML file to the expected metadata format.

Updating¶

To update click on the heading of the storage and you will be presented with a menu for that storage. Alter the storage as needed and click “Update Storage”.

Deleting¶

To delete the storage go to the storage listing page from the admin menu, and click on the cross top right of the storage. You will be asked to confirm once, after which all references to the storage will be removed.

Storage Methods¶

Storage methods are ways of accessing the storage.

HTTP

The storage is accessible by means of the HTTP protocol. This can be setup with a username and password, an address including port, and the type. Note that the webserver must have the WebDAV extensions enabled for this method to be used.

File

Used for locally mounted file systems to the backend.

FTP

Used for accessing files over the FTP protocol. This method is preferable over HTTP for moving files around.

OMMS

MatrixStore (Object Matrix) file system. Please note that if the storage is configured with write permission, you must also configure the Vault with the property “Data Updatable” set to true in MatrixStore Admin.

Make sure the correct Vidispine API:s are installed to talk to OMMS. Especially important if you upgrade OMMS.

S3

This allows you to configure Cantemo to access an S3 bucket as a storage. For Amazon S3 it is recommended that you use Amazon IAM to create an IAM user and limit that user’s access to only the relevant S3 buckets. For an S3 storage configured in this way, you should use the Access Key ID as the username and the Secret Access Key as the password. In the address field, fill in bucketname you want to connect to. If you want to use a subfolder you can use the form bucketname/foldername

Please note that regions which require Version 4 signatures are not supported in this version of Cantemo. See http://apidoc.vidispine.com/latest/storage/storage.html#s3 for more infomation. Currently, this affects Beijing and Frankfurt.

SFTP

Allows you to configure a storage location on a remote server, accessed via the SFTP protocol.

DS3

Allows you to configure a Spectralogic BlackPearl Storage

Azure

Allows you to configure a Microsoft Azure storage

**Google Cloud Storage*

Allows you to configure a Google Cloud Storage

Each method takes a number of parameters which need to be set for the storage to work correctly and those should be based upon the needs of the method used.

Username

User for the method. Not needed for file, and optional on some other methods.

Password

Password for the method. Not needed for file, and optional on some other methods.

Address

This should be the address of the resource. Do not put the protocol in front, as this is added automatically, but you can add a port and a trailing slash. Examples are:

10.0.0.1:80/
/var/www/
/srv/mediamount1/
192.168.0.100/

ClusterID

This is used for MatrixStore storage only.

Vault ID

This is used for MatrixStore storage only.

Browse

Server can browse the storage.

Read

Server can read the storage.

Write

Server can write to the storage.

Type

The type of the storage. AUTO should only be used with HTTP methods when you need to serve the file through HTTP - typically for preview video.

Examples

A basic file method takes a file path such as ‘/srv/mount1/’ and can have a READ, WRITE and BROWSE method enabled on it.

If you are creating an HTTP method type can be set to auto if you wish to serve up the media files from its root. For instance create a method:

http://127.0.0.1:8080/ AUTO

Do not use an AUTO type on anything other than an HTTP method that points at the root the installation with a READ method. Make sure that it has no browse, read or write.

Create another storage method such as file, which does have browse read and write if needed. This storage is then used by Cantemo for creating previews etc. This should be enough for setting up the storage for previewing through Cantemo.

Viewing files on storages.¶

To view all files on the storage click on “View files” and you will be taken to the files view for the storage.

You can paginate, sort and search through the files that are listed and it will show:

Item: An item reference number
Filename: The filename as indexed on the storage
File method icon: To link to the file
Size: File size of the file. This is rounded up.
State: Whether the file is imported or not imported into the system.
Timestamp: A timestamp of the file as reported by the storage
Checkbox: For selecting one or more files.

From here various operations can be completed on the files themselves.

Deleting: Remove the file from storage all together.

Moving the files: Choose the files you wish to move, click “Move files”, and then select the destination storage. Click “Move”. Your file will be moved into the new storage.
Import files: Clicking “Import files” will take you to a metadata entry form for manually importing the files.

Rescanning files on a storage¶

Cantemo automatically scans each storage periodically, as determined by the Scan Interval setting. If you want to trigger a rescan manually, you can do so using the Rescan button on the files list page.

Automatic file movements¶

The storage capacity and low- and high-water marks can be used together to create a storage hierarchy.

Capacity

The storage capacity can either be automatically detected, in which case the capacity of the underlying filesystem is used, or it can be manually specified. Specifying the capacity manually is useful in situations where you want to limit the use of a specific filesystem and only use a portion of it as a Cantemo storage.

Low- and high water marks

These water marks are used when a storage is nearing full capacity and are specified as a percentage of the storage capacity. Once a storage fills up above the high water mark, files starts to be moved off of the storage, to another storage with free capacity. Only files which have been ingested into the system and belong to an item will be eligible to be moved. The system will continue to move files off of the storage until the usage is below the low water mark

Both the low- and high water marks are by default set to 100%, which means that no automatic evacuation of the storages occur.

By setting both the low- and high water marks to 0%, you will force files to be moved off of the storage as soon as the files have been ingested. This can be used to create an drop folder-like behavior where users can place files in a directory after which they are either automatically or manually imported.

Note

If you have files not managed by Cantemo on a storage the utilization may behave a bit strange since the watermarks are calculated based on free space, but the watermark must be reached by filling up Cantemo managed files.