Storages are the file systems that are indexed by and known to Portal and its underlying backends. Storages usually map to file storage that can be written to or read from, and are used for storing the media assets, sending files for editing or importing from by users.
Each storage can have one or more storage methods attribute, and this is the way that the system can access the storage and the files contained within it.
The storage list view can be reached from the admin menu, storages.
It will list all the storages known to Portal and its backends, plus a graphical status of how much space is left and what storage methods are configured for it.
Storages have several configurable attributes.
- Storage methods
Discussed in more detail below
A reference name
- Storage Group
See storage grouping below for more detail on groups, but this will decide what storage group the system will be in.
This is the capacity that will be monitored and used. Please enter a positive integer. Its possible to auto-detect the size of the storage, but please not that this takes some minutes to calculate the total size correctly.
- Low-Water Mark
Safe amount for this storage to utilize. See High-Water Mark.
- High-Water Mark
The server will start deleting unneeded files (that are stored elsewhere) on the storage when the capacity reaches this mark, then it will try to copy and move files until it reaches the low-water mark.
- Storage type
The storage type determines how files on the storage are read and written. The following is an explanation of the different options.
Local - This is the default storage type and files are both read and written to this storage.
Shared - This storage is shared with another application so files are only read from here. No files will be written to or removed from this storage.
Remote - Files on this storage will not be accessed directly and will be copied to a local storage before use.
External - This is a storage placeholder and this storage will not be used by the system. This type is not used by Portal.
Export - This storage can be used as an export location. Files which are written here cannot be copied back to other storages and files cannot be imported from this storage type.
- Scan Interval
Configure how often the server scans this storage for new and modified files. The default is 60 seconds.
- Refresh Interval
Configure how often the server does a complete scan of the entire storage, updating the metadata for each file. The default is 240 minutes.
- Maximum Bandwidth
Configures how fast the server tries to read/write data to the storage. Can be used to throttle usage of a particular storage and is useful if the available network bandwidth exceeds the speed of the storage solution.
- Exclude filters
The exclude filters allows the administrator to exclude certain file name patterns from appearing in the files list of a storage as well as excluding them from Auto Ingest. This can for example be used to exclude Apple resource forks. The default for new storages is a filter which matches all files and directories with a leading period character (these files are normally hidden in unix filesystems)
Files that are not imported on this storage can be view in the “Import From Storage” page usually located under the Ingest menu. This is also needed to show that this storage can be used to import from and when selecting auto ingest.
- Auto Ingest
This flag denotes that any file that will be placed on this storage will after a short period of time be ingested into the system.
Storages can be grouped together to be more representative of workflows or to help when organizing automated rules. To create a group, click on “Add Storage Group” and fill in the name and description.
Storages can then added to that group.
To add monitored storages:
Create the storage group that you wish to use for your storage and then click “Add New Storage” within that storage group.
Fill in the details of the storage. By default, a storage method of type file is added to a new storage, but this can be replaced before saving.
Click “Add Storage” to finish.
The storage will be added to the system and in a few minutes will be indexed for files that could possibly be ingested.
Adding an auto import storage¶
To add an auto ingest storage, check the Auto-ingest checkbox. Note that the Importable checkbox will automatically be checked as well as it is required for this type of storages. You will be presented with two additional drop downs.
Ingest Profile determines the transcode and access rule profile which should be applied to the ingested content. This reflects a profile defined for a certain group, and is in line with the ingest profile chosen upon an Upload or Import from Storage.
Upon an auto ingest, metadata can automatically be found if it shares the same filename and has the extension “.xml”, for example video.avi and video.xml. Metadata Transformation defines which (if any) transformation should be used for mapping the XML file to the expected metadata format.
To update click on the heading of the storage and you will be presented with a menu for that storage. Alter the storage as needed and click “Update Storage”.
To delete the storage go to the storage listing page from the admin menu, and click on the cross top right of the storage. You will be asked to confirm once, after which all references to the storage will be removed.
Storage methods are ways of accessing the storage.
The storage is accessible by means of the HTTP protocol. This can be setup with a username and password, an address including port, and the type. Note that the webserver must have the WebDAV extensions enabled for this method to be used.
Used for locally mounted file systems to the backend.
Used for accessing files over the FTP protocol. This method is preferable over HTTP for moving files around.
MatrixStore (Object Matrix) file system. Please note that if the storage is configured with write permission, you must also configure the Vault with the property “Data Updatable” set to true in MatrixStore Admin.
Make sure the correct Vidispine API:s are installed to talk to OMMS. Especially important if you upgrade OMMS.
This allows you to configure Portal to access an S3 bucket as a storage. For Amazon S3 it is recommended that you use Amazon IAM to create an IAM user and limit that user’s access to only the relevant S3 buckets. For an S3 storage configured in this way, you should use the Access Key ID as the username and the Secret Access Key as the password. In the address field, fill in bucketname you want to connect to. If you want to use a subfolder you can use the form bucketname/foldername
Please note that regions which require Version 4 signatures are not supported in this version of Portal. See http://apidoc.vidispine.com/latest/storage/storage.html#s3 for more infomation. Currently, this affects Beijing and Frankfurt.
Allows you to configure a storage location on a remote server, accessed via the SFTP protocol.
Allows you to configure a Spectralogic BlackPearl Storage
Allows you to configure a Microsoft Azure storage
- **Google Cloud Storage*
Allows you to configure a Google Cloud Storage
Each method takes a number of parameters which need to be set for the storage to work correctly and those should be based upon the needs of the method used.
User for the method. Not needed for file, and optional on some other methods.
Password for the method. Not needed for file, and optional on some other methods.
This should be the address of the resource. Do not put the protocol in front, as this is added automatically, but you can add a port and a trailing slash. Examples are:
10.0.0.1:80/ /var/www/ /srv/mediamount1/ 192.168.0.100/
This is used for MatrixStore storage only.
- Vault ID
This is used for MatrixStore storage only.
Server can browse the storage.
Server can read the storage.
Server can write to the storage.
The type of the storage. AUTO should only be used with HTTP methods when you need to serve the file through HTTP - typically for preview video.
A basic file method takes a file path such as ‘/srv/mount1/’ and can have a READ, WRITE and BROWSE method enabled on it.
If you are creating an HTTP method type can be set to auto if you wish to serve up the media files from its root. For instance create a method:
Do not use an AUTO type on anything other than an HTTP method that points at the root the installation with a READ method. Make sure that it has no browse, read or write.
Create another storage method such as file, which does have browse read and write if needed. This storage is then used by Portal for creating previews etc. This should be enough for setting up the storage for previewing through Portal.
Viewing files on storages.¶
To view all files on the storage click on “View files” and you will be taken to the files view for the storage.
You can paginate, sort and search through the files that are listed and it will show:
An item reference number
The filename as indexed on the storage
- File method icon
To link to the file
File size of the file. This is rounded up.
Whether the file is imported or not imported into the system.
A timestamp of the file as reported by the storage
For selecting one or more files.
From here various operations can be completed on the files themselves.
Remove the file from storage all together.
- Moving the files
Choose the files you wish to move, click “Move files”, and then select the destination storage. Click “Move”. Your file will be moved into the new storage.
- Import files
Clicking “Import files” will take you to a metadata entry form for manually importing the files.
Rescanning files on a storage¶
Portal automatically scans each storage periodically, as determined by the Scan Interval setting. If you want to trigger a rescan manually, you can do so using the Rescan button on the files list page.
Automatic file movements¶
The storage capacity and low- and high-water marks can be used together to create a storage hierarchy.
The storage capacity can either be automatically detected, in which case the capacity of the underlying filesystem is used, or it can be manually specified. Specifying the capacity manually is useful in situations where you want to limit the use of a specific filesystem and only use a portion of it as a Portal storage.
Low- and high water marks
These water marks are used when a storage is nearing full capacity and are specified as a percentage of the storage capacity. Once a storage fills up above the high water mark, files starts to be moved off of the storage, to another storage with free capacity. Only files which have been ingested into the system and belong to an item will be eligible to be moved. The system will continue to move files off of the storage until the usage is below the low water mark
Both the low- and high water marks are by default set to 100%, which means that no automatic evacuation of the storages occur.
By setting both the low- and high water marks to 0%, you will force files to be moved off of the storage as soon as the files have been ingested. This can be used to create an drop folder-like behavior where users can place files in a directory after which they are either automatically or manually imported.
If you have files not managed by Portal on a storage the utilization may behave a bit strange since the watermarks are calculated based on free space, but the watermark must be reached by filling up Portal managed files.