JATOS can run on multiple nodes in a cluster to achieve high availability and scalability.
Things to know before running JATOS in a multi-node setup
- JATOS, in a multi-node setup, needs a MySQL or MariaDB database (and cannot be used with the embedded H2 database).
- All JATOS nodes need to share some folders: study assets, study uploads, study logs, and JATOS' tmp folder.
- All JATOS nodes need the same secret, otherwise the session cookie used for authentication won't work.
- Updating is arguably easier by just changing the tag of JATOS docker image to a higher version (but JATOS' auto-updater can't be used).
All these points (and more) are addressed in this page.
Multi-node installation with Docker Compose
A setup of JATOS with multiple nodes through Docker Compose might not make much sense, because all JATOS instances still run on the same machine. But it highlights some general concepts and caveats pretty well, so we describe it here.
How to get started with JATOS and Docker Compose is explained in another page. You might want to follow the instructions there to get a JATOS installation with a MySQL database and Nginx running.
Now, if you want to run JATOS in multiple containers in parallel you need to configure the compose.yaml additionally (if you haven't already):
-Djatos.multiNode=truein the command section of the jatos service.
- Set the
JATOS_SECRETenvironment variable to a string with at least than 15 characters (otherwise the session cookie that JATOS uses for authentication won't work).
It's important to share some of JATOS folders between all JATOS nodes. In our Docker composed setup this is already achieved with the shared volumes jatos-data, jatos-logs, and jatos-db. Nothing to do here.
Finally, to scale up and run multiple JATOS instances use the
--scale parameter, e.g. to run two JATOS instances:
docker compose -f compose.yaml up --scale jatos=2
JATOS with Kubernetes
Kubernetes is a system for container orchestration and automatic deployments. It offers vast possibilities to do so in many different ways that might also depend on your cloud provider. Here we used it with DigitalOcean - but with some adjustments it should work on any Kubernetes cluster.
For the JATOS cluster we use Kustomize to define Kubernetes objects through kustomization YAML files.
We assembled all necessary files in a git repository.
git clone https://github.com/JATOS/JATOS_with_kubernetes.git
Then, after you set up everything, you can start the cluster with:
kubectl apply -k <my_JATOS_kustomize_directory>
Load-balancing and scaling
In our jatos.yaml, for auto-balancing in our JATOS cluster, we use the one integrated in DigitalOcean. This is specified in the Service object, with the annotation
- port: 80
And our cluster does automatic horizontal scaling with an
HorizontalPodAutoscaler. Here we set up a minimum of 2 and maximum of 10 JATOS pods and as scaling metric a average CPU utilization of 100%.
- type: Resource
As said before, JATOS, if running on multiple nodes, needs to share some folders. Translated to Kubernetes this means the PersistentVolumeClaim needs the
Although many cloud provider have their own storage system to achieve this, we use a common NFS storage. E.g. there is an easy-to-use helm chart for this purpose: nfs-server-provisioner. And since we want to run on DigitalOcean we need the parameter
persistence.storageClass set to
helm install nfs-server stable/nfs-server-provisioner --set persistence.enabled=true,persistence.storageClass=do-block-storage,persistence.size=11Gi
Then in our jatos.yaml the NFS storage is used in a
And the volume is mounted in every JATOS pod:
- name: jatos-data-storage
Configure JATOS' deployment
In jatos.yaml, to run JATOS in on multiple nodes in a cluster you have to set the parameter
-Djatos.multiNode=true. Also the parameter
-Djatos.logs.appender=ASYNCSTDOUT redirects the logging to stdout, which is what you probably want with Kubernetes.
-J-Xmx defines the maximum memory the Java Virtual Machine (JVM) that runs JATOS is allowed to use. If you don't set this, the JVM might take too much memory for itself and strangle the operating system. Here we set it to 1500 MB but it really depends on the kind of underlying machine you are using to run your nodes.
You might want to change the Docker image version to a different one.
# Maybe use a newer image version
- image: jatos/jatos:3.8.4
# Necessary to run JATOS on multiple nodes
# Logging to stdout
# Set the JVM maximum memory usage. It has to fit your machine.
We assume here that you have your MySQL database set up and ready already. Have a look at JATOS with MySQL to get started.
In jatos.yaml you have to change the environmental variable
JATOS_DB_URL. The IP and port need to be the ones from your MySQL IP and port.
Liveness probe and startup probe
Applications running on the JVM can need some initial warm-up time before they are fully functional. Therefore we have, additionally to the
livenessProbe in jatos.yaml, a
startupProbe that accounts for this. You might have to tweak
periodSeconds on your system.
securityContext and affinity
The jatos.yaml also has a securityContext and a affinity section. You probably don't have to change anything there. We just want to explain them here shortly.
The securityContext sets the UID and GID of the user defined in JATOS' Docker image.
In the affinity section we define a
podAntiAffinity to ensure that each Kubernetes pod runs only one JATOS.
- topologyKey: kubernetes.io/hostname
Updating JATOS with Kubernetes
The easiest way to update a JATOS Kubernetes cluster is to just change the JATOS' Docker image tag to a higher version. JATOS' auto-updater cannot be used here.
But there are some constraints:
- Kubernetes' rolling updates are not possible with JATOS. You have to turn off all JATOS pods, do the update (change the Docker image tag) and turn them back on.
- JATOS is only allowed to update to higher version numbers - downgrading will likely break your installation.
- And please do backups before updating.