Installing a DataShyft Runtime

The DataShyft Customer Runtime is distributed as a Docker image.  This allows easy installation and quick configuration of the runtime on systems.  To install and execute the Runtime, you will need a Docker server (or other OCI-compliant Engine) to host the container.  Once the host is set up, you can follow the instructions below to setup and configure the DataShyft Customer Runtime.

Installation in Docker

The DataShyft Customer Runtime can be deployed on a Docker host.  The following instructions explain the steps necessary to complete this installation.

  1. Create a Volume
    1. Give the Volume a name, i.e. “datashyft-config”
    2. This volume will be mapped to the path /datashyft-config inside the container.
  2. Set up the network
    1. The Container will require a Public IP address. Optionally, a DNS name can be attached to this IP address if desired - this DNS name, if used, will need to be available publicly as well.
  3. Pull the Container Image
    1. The Container Image is at datashyft/system-runtime:latest on Docker Hub
  4. Run the Container
    1. Where possible, we recommend using a flag that indicates that the container should automatically restart unless it has been explicilty stopped (e.g. --restart unless-stopped in Docker)
    2. Container Engines may offer port-mapping abilities. Map the internal ports 32100 and 8752 to whichever external ports you would like to use
      1. Ensure that any firewalls or other networking infrastructure properly allow traffic through to the chosen external ports, as appropriate
  5. Access the logs
    1. The container will output the Runtime Admin Authorization Token to Standard Out - you will need this Token during Registration.
      1. In Docker, this can be accessed by executing docker logs <Container name>
  6. Proceed to the Runtime Registration section to finish configuring the runtime.

Installation on Azure as a Container Instance

The DataShyft Customer Runtime can be deployed on Microsoft Azure as an Azure Container Instance (ACI).  The following instructions assume that you already have an Azure account, have created a Resource Group where the DataShyft Runtime will be deployed, and have created an Azure Storage Account that will be used to store the DataShyft Runtime’s persistent storage volume.

Deploying DataShyft Customer Runtime

  1. To deploy DataShyft Customer Runtime to Azure, you will need to use the Azure CLI (due to limitations in Azure Portal). To prepare a machine to do this, you will need to install the Azure CLI - the following instructions assume you will be using a BASH shell to do so, however PowerShell and others can also be used. You can find more information about them here: https://learn.microsoft.com/en-us/azure/container-instances/container-instances-quickstart
  2. Once you have the Azure CLI installed, you will need to log in with the Azure CLI via the az logincommand
  3. The following script automates the creation of the persistent File Share on the Azure Storage Account and the creation of the DataShyft Runtime container instance.  Copy the script to your local system and edit the variables at the top of the file as needed.
#!/bin/bash

#DataShyft Customer Runtime Container Information
# -- Edit these values to specify where the runtime should be 
#    created, the name to assign to it, and the amount of 
#    resources to allocate to it.

RESOURCE_GROUP="<resourceGroup> "
CONTAINER_NAME="<containerName>"
CONTAINER_CPU="2"
CONTAINER_MEMORY="8"


#Azure Storage Account Information
# -- Edit these to reflect the location and name of the 
#    Azure Storage Account where the runtimes persistent volume 
#    will be stored.  The fie Share name will be the same as 
#    the Container Name.

STORAGE_RESOURCE_GROUP="<resourceGroup>"
STORAGE_RESOURCE_NAME="<fileShareName>"


#Docker Account information
# -- Edit these to specify the username and password of your 
#    Docker Hub account so that the Azure Container can retrieve 
#    the DataShyft Customer Runtime image.

DOCKER_SERVER="docker.io"
DOCKER_USERNAME="<accountName>"
DOCKER_ACCESS_TOKEN="<personalAccessToken>"


# Extracts the Storage key for the Azure Storage Account 
# for the Azure Container instance to use

STORAGE_KEY=$(az storage account keys list --resource-group $STORAGE_RESOURCE_GROUP --account-name $STORAGE_RESOURCE_NAME --query "[0].value" --output tsv)
STORAGE_KEY=${STORAGE_KEY//$'\r'}


# Check if the File Share exists

FILE_SHARE_NAME=${CONTAINER_NAME}
EXISTING_SHARE=$(az storage share list --account-key ${STORAGE_KEY} --account-name ${STORAGE_RESOURCE_NAME} --prefix ${FILE_SHARE_NAME} --output tsv | grep "${FILE_SHARE_NAME}" | wc -l)


# If the File Share doesn't exist yet, create it.

if [ ${EXISTING_SHARE} -eq 0 ] ; then 
echo "Creating File Sharer for the Container"

az storage share create \
--account-key ${STORAGE_KEY} \
--account-name ${STORAGE_RESOURCE_NAME} \
--name ${FILE_SHARE_NAME} \
--quota 100
fi


# Create the Azure Container Instance for the new Runtime

az container create \
--resource-group ${RESOURCE_GROUP} \
--name ${CONTAINER_NAME} \
--image docker.io/datashyft/system-runtime:latest \
--cpu ${CONTAINER_CPU} \
--memory ${CONTAINER_MEMORY} \
--os-type Linux \
--registry-login-server ${DOCKER_SERVER} \
--registry-username ${DOCKER_USERNAME} \
--registry-password ${DOCKER_ACCESS_TOKEN} \
--azure-file-volume-account-name ${STORAGE_RESOURCE_NAME} \
--azure-file-volume-account-key ${STORAGE_KEY} \
--azure-file-volume-share-name ${FILE_SHARE_NAME} \
--azure-file-volume-mount-path /datashyft-config \
--ip-address Public \
--dns-name-label ${CONTAINER_NAME} \
--ports 8752 32100

CONTAINER_FQDN=$(az container show \
--resource-group ${RESOURCE_GROUP} \
--name ${CONTAINER_NAME} \
--query ipAddress.fqdn \
--output tsv)

TOKEN=$(az container logs \
--resource-group ${RESOURCE_GROUP} \
--name ${CONTAINER_NAME} | \
grep "Runtime Admin Authorization Token")

echo "DataShyft Runtime Address: ${CONTAINER_FQDN}"
echo "${TOKEN}"
  1. Once you have filled in the correct information in the script and saved it, execute the script to create the Azure Container Instance running DataShyft Customer Runtime. This may take a moment.
  2. Make note of the Container’s address and Runtime Admin Authorization Token for use during Runtime Registration.  They will be output at the end of the script execution.
  3. The Azure Container Instance is now created and the DataShyft Customer Runtime is running. You may now proceed to the Runtime Registration section to finish configuring the runtime.

Installation on AWS In Elastic Container Service (ECS)

  1. Login to your AWS account.
  2. Create a new VPC to host the Runtime Infrastructure
  3. Create a new Security Group in the VPC for the DataShyft Runtime
    1. Allow inbound traffic on port 8752 from the Internet
    2. Allow inbound traffic on port 32100 from the Internet
  4. Create a new Elastic Container Service (ECS) Cluster using the Fargate Infrastructure
  5. Create a new ECS Task Definition
    1. Task Definition Family: DataShyftRuntime
    2. Infrastructure Requirements: AWS Fargate
    3. Set the Task Size to a minimum of 2 vCPU and 8GB memory
    4. Configure the Runtime container as Container 1
      1. Name: datashyft-runtime
      2. Image URI: public.ecr.aws/n2m2h1j8/datashyft/system-runtime:latest
      3. Configure the Port Mappings
        1. 8752 - TCP – webservice-port – HTTP
        2. 32100 – TCP – bridge-port – None
    5. Configure the Storage by adding a volume and mount point
      1. Add Volume
        1. Specify the Volume Name (e.g., datashyft-config).  The volume name must be unique for each runtime that is created.
        2. Set the task to be configured at task definition creation
        3. Set the Volume Type to Bind mount
      2. Add Mount Point
        1. Continer: datashyft-runtime
        2. Source Volume: datashyft-config
        3. Container path: /datashyft-config 
  6. Deploy the new task into the cluster: 
    1. Select Task definitions in left sidebar
    2. Select DataShyftRuntime in list, choose the newest revision in the list, click Deploy, then Create Service
    3. Configure the Environment for the new Service
      1. Set the Cluster to the newly create cluster
      2. Set the Compute Configuration to Launch Type
      3. Set the Launch Type to FARGATE using the LATEST version
    4. Setup the Deployment Configuration for the service
      1. Set the Application type to Service
      2. Provide a name for this service (e.g. DataShyftRuntime-1)
      3. Set the Service Type to Replica
      4. Set the number of Desired Tasks to 1
    5. Networking
      1. Select the VPC you created
      2. Select just the public Subnets of the VPC
      3. Select the Security group you created
      4. Turn on the option to assign a Public IP to the Service
  7. Collect the required information about the new container
    1. Click on the new Service in the Cluster’s Services tab
    2. Click on the newly launched Task and copy the Public IP for use in during Runtime Registration
    3. Click on the Logs tab and copy the Runtime Admin Authorization Token for use during Runtime Registration
  8. Proceed to the Runtime Registration section to finish configuring the runtime.

Runtime Registration

DataShyft Customer Runtimes need to be registered with your DataShyft tenant before they can be used to host blueprint deployments.  This section covers how to register runtimes, list the system a runtime is registered with, and remove a runtime registration.

Register the New Runtime with your Tenant

  1. Access the Runtime Management Interface of the newly started Customer Runtime.  Use the IP address or DNS name and mapped port of the container to visit https://<containerAddress>:<mappedPort>/.
    1. The Runtime uses a self-signed HTTPS certificate to encrypt the connection.  The web browser will likely give you a warning about the certificate.  Tell the browser to proceed to the site and trust the certificate.  You can replace the SSL certificate after registering the runtime (see the Error! Reference source not found. section later in this document).
  2. Fill out the Runtime Registration Form with the required information:
    1. Tenant URL – the URL of the DataShyft Tenant this runtime should be registered with.
    2. Email Address – the email address of an administrator account on the DataShyft Tenant.
    3. Password – the password for the administrator account.
    4. System Name – the name to assign to this runtime on the DataShyft Tenant.  A new system will be created.
    5. Public Address – The public IP Address or DNS Name assigned to the DataShyft Customer Runtime instance.
    6. Authorization Token – The authorization token of the runtime that was retrieved from the logs during the Installation process.
  3. Submit the Runtime Registration Form.  The runtime will authenticate with the DataShyft Tenant, create a new system with the specified name, and register itself as that system.  Once complete, the runtime will redirect to the Registration Information page.  After a few moments, the runtime status on this page will update to Online. Once it does, the runtime is fully connected to your DataShyft tenant and is ready to accept deployments.  You can close the Runtime Management window.
Did this answer your question? Thanks for the feedback There was a problem submitting your feedback. Please try again later.

Still need help? Contact Us Contact Us