Health Check

To help diagnose problems, Cantemo comes with a built in health check which verifies the different components of the system. This check can be reached via the System Overview option in the Admin menu. On the System Overview page, an aggregated status is shown to give an idea of the system status at a quick glance. By clicking on the status you can then open up the full Health Check page which allows you to drill down into the status of each sub-component by clicking on the individual components.

The system has four status levels for each component:

  • CRITICAL - A failure which will cause the system to not function

  • FAILED - A failure which may cause some loss of functionality

  • WARNING - May cause a loss of redundancy

  • OK - The component is functioning as expected

_images/portal_healthcheck.png

REST API

The health check is also available via a REST API for easy integration into external monitoring systems. There are two endpoints, the main health check is available under /API/v2/healthcheck/ and returns a json or XML document with the same information as the health check page. This endpoint requires authentication since it can reveal internal information about the system.

In addition to the main health check, a simple version is also available under /API/v2/healthcheck/simple/ which can be used without authentication. This check only tests a limited number of subcomponents which can be tested quickly. This endpoint is suitable to use in for example HAproxy or some other frontend which frequently polls an endpoint to test system availability.

Command line

If you have problem running Cantemo web you can run the health check from the command line:

/opt/cantemo/portal/manage.py health_check

Configuring health checks

There may be reasons why you do not want to monitor certain areas of the system. On the Systems settings page you can configure the System Healthchecks.

_images/portal_healthcheck_settings.png

Distributed deployments

By default the health check assumes that all Cantemo services run on the same host: services are checked with systemctl and the Nginx tests use http://127.0.0.1. When the services run in separate containers or hosts, for example on AWS ECS or Kubernetes, these checks report failures although the system is healthy. Rather than disabling the health check, set the following in the [healthcheck] section of portal.conf (see Available settings):

HEALTHCHECK_SERVICES_MODE = distributed
HEALTHCHECK_NGINX_URL = https://nginx.example.internal

HEALTHCHECK_SERVICES_MODE defaults to systemd and does not need to be set on a single-host installation. In distributed mode the services are checked over the network instead: the Celery workers through the Celery broker, and the other services over their own protocols.

Every enabled service on the System Settings page is reported by name. Besides OK and CRITICAL, a service can be reported as:

  • NOT APPLICABLE - the service is not used by this system. This is not a failure.

  • UNABLE TO VERIFY - the service cannot be checked with the current configuration. It is reported as a warning.

The following is required for the checks to work:

  • HEALTHCHECK_NGINX_URL and the Celery broker must be reachable from the Cantemo services that run the health check.

  • The Nginx connection test sends the Vidispine credentials, so HEALTHCHECK_NGINX_URL must use HTTPS with a valid certificate unless it points to localhost. The Nginx service check sends no credentials and also accepts plain HTTP to another host, which is useful when nginx is reached through an internal load balancer.

  • HEALTHCHECK_NGINX_URL must not contain credentials, a query or a fragment.

  • Celerybeat is checked through the periodic tasks it has run, so it must use django_celery_beat.schedulers:DatabaseScheduler.