Kubernetes Validator Deployment

This section was copied from existing reviewed documentation. Source: docs/src/validator_operator/validator_helm.rst Reviewers: Skip this section. Remove markers after final approval.

This section describes how to deploy a standalone validator node in Kubernetes using Helm charts. The Helm charts deploy a validator node along with associated wallet and CNS UIs, and connect it to a global synchronizer.

Requirements

Validator Network Diagram

Preparing a Cluster for Installation

Create the application namespace within Kubernetes.

    kubectl create ns validator

The validator deployment assumes one validator per namespace. If you wish to run multiple validators in the same cluster, please create a separate namespace for each.

HTTP Proxy configuration

If you need to use an HTTP forward proxy for egress in your environment, you need to set https.proxyHost and https.proxyPort in additionalJvmOptions in the validator and participant helm charts to use the HTTP proxy for outgoing connections:

  additionalJvmOptions: |
    -Dhttps.proxyHost=your.proxy.host
    -Dhttps.proxyPort=your_proxy_port

Replace your.proxy.host and your_proxy_port with the actual host and port of your HTTP proxy. Proxy authentication is currently not supported.

Bypassing the proxy for specific hosts

Setting http.nonProxyHosts affects:

The HTTP client used by the CN apps (Validator, Scan, SV, Wallet).
JDK-level HTTP clients in the same JVM (via the default ProxySelector). This includes the Auth0 JWK library used by the CN apps and by the Canton participant for JWKS / OIDC discovery, as well as file downloads that use java.net.HttpURLConnection.
gRPC egress from other components, because gRPC’s Netty transport delegates proxy decisions to the default JDK ProxySelector.

You can set http.nonProxyHosts to bypass the proxy for specific target hosts. Matching hosts will be contacted directly rather than through the configured proxy. This is useful for services that are reachable on the local network, such as an in-cluster Scan instance or internal monitoring endpoints. The value is a |-separated list of patterns that follows the standard Java nonProxyHosts grammar:

Patterns match the request host name case-insensitively.
* is a wildcard. Conventionally it is used at the start (*.internal) or end (10.*) of a pattern.
Matching is performed on the raw host string from the request URI. No DNS resolution is performed, so localhost and 127.0.0.1 are treated as different names unless you list both.
An empty value (e.g. -Dhttp.nonProxyHosts=) means “no bypass patterns”.

Example additionalJvmOptions for the validator helm chart that proxies external traffic but bypasses the proxy for localhost / 127.0.0.1, any host in the .internal domain, and any IPv4 address whose literal string representation starts with 10.:

additionalJvmOptions: |
  -Dhttps.proxyHost=your.proxy.host
  -Dhttps.proxyPort=your_proxy_port
  -Dhttp.nonProxyHosts=localhost|127.0.0.1|*.internal|10.*

Configuring PostgreSQL authentication

The PostgreSQL instance that the helm charts create, and all apps that depend on it, require the user’s password to be set through Kubernetes secrets. Currently, all apps use the Postgres user cnadmin. The password can be setup with the following command, assuming you set the environment variable POSTGRES_PASSWORD to a secure value:

call out the option of using a managed postgres instance

    kubectl create secret generic postgres-secrets \
        --from-literal=postgresPassword=${POSTGRES_PASSWORD} \
        -n validator

Preparing for Validator Onboarding

Ensure that your validator onboarding secret ONBOARDING_SECRET is set in the namespace you created earlier.

    kubectl create secret generic splice-app-validator-onboarding-validator \
        "--from-literal=secret=${ONBOARDING_SECRET}" \
        -n validator

Configuring Authentication

For security, the various components that comprise your Validator node need to be able to authenticate themselves to each other, as well as be able to authenticate external UI and API users. We use JWT access tokens for authentication and expect these tokens to be issued by an (external) OpenID Connect (OIDC) provider. You must:

Set up an OIDC provider in such a way that both backends and web UI users are able to obtain JWTs in a supported form.
Configure your backends to use that OIDC provider.

The validator supports non-authenticated deployments as well, but this is strongly discouraged for production deployments. If you wish to run without authentication, please refer to the notes in helm-validator-no-auth.

OIDC Provider Requirements

This section provides pointers for setting up an OIDC provider for use with your Validator node. Feel free to skip directly to helm-validator-auth0 if you plan to use Auth0 for your Validator node’s authentication needs. These docs focus on Auth0, and are being continuously tested and maintained. Other OIDC providers can be used, and are in active use by various community members, who have contributed some notes and examples in Okta and Keycloak community authored examples. Your OIDC provider must be reachable¹ at a well known (HTTPS) URL. In the following, we will refer to this URL as OIDC_AUTHORITY_URL. Both your Validator node and any users that wish to authenticate to a web UI connected to your Validator node must be able to reach the OIDC_AUTHORITY_URL. We require your OIDC provider to provide a discovery document at OIDC_AUTHORITY_URL/.well-known/openid-configuration. We furthermore require that your OIDC provider exposes a JWK Set document. In this documentation, we assume that this document is available at OIDC_AUTHORITY_URL/.well-known/jwks.json. For machine-to-machine (Validator node component to Validator node component) authentication, your OIDC provider must support the OAuth 2.0 Client Credentials Grant flow. This means that you must be able to configure (CLIENT_ID, CLIENT_SECRET) pairs for all Validator node components that need to authenticate themselves to other components. Currently, this is the validator app backend - which needs to authenticate to the Validator node’s Canton participant. The sub field of JWTs issued through this flow must match the user ID configured as ledger-api-user in helm-validator-auth-secrets-config. In this documentation, we assume that the sub field of these JWTs is formed as CLIENT_ID@clients. If this is not true for your OIDC provider, pay extra attention when configuring ledger-api-user values below. For user-facing authentication - allowing users to access the various web UIs hosted on your Validator node, your OIDC provider must support the OAuth 2.0 Authorization Code Grant flow and allow you to obtain client identifiers for the web UIs your Validator node will be hosting. Currently, these are the Wallet web UI and the CNS web UI. You might be required to whitelist a range of URLs on your OIDC provider, such as “Allowed Callback URLs”, “Allowed Logout URLs”, “Allowed Web Origins”, and “Allowed Origins (CORS)”. If you are using the ingress configuration of this runbook, the correct URLs to configure here are https://wallet.validator.YOUR_HOSTNAME (for the Wallet web UI) and https://cns.validator.YOUR_HOSTNAME (for the CNS web UI). YOUR_HOSTNAME is a placeholder that you need to replace with the actual domain name or IP address of the server hosting your services. An identifier that is unique to the user must be set via the sub field of the issued JWT. On some occasions, this identifier will be used as a user name for that user on your Validator node’s Canton participant. In helm-validator-install, you will be required to configure a user identifier as the validatorWalletUser -make sure that whatever you configure there matches the contents of the sub field of JWTs issued for that user. All JWTs issued for use with your Validator node:

must be signed using the RS256 signing algorithm

In the future, your OIDC provider might additionally be required to issue JWTs with a scope explicitly set to daml_ledger_api (when requested to do so as part of the OAuth 2.0 authorization code flow). Summing up, your OIDC provider setup must provide you with the following configuration values:

Name	Value
OIDC_AUTHORITY_URL	The URL of your OIDC provider for obtaining the `openid-configuration` and `jwks.json`.
VALIDATOR_CLIENT_ID	The client id of your OIDC provider for the validator app backend.
VALIDATOR_CLIENT_SECRET	The client secret of your OIDC provider for the validator app backend.
WALLET_UI_CLIENT_ID	The client id of your OIDC provider for the wallet UI.
CNS_UI_CLIENT_ID	The client id of your OIDC provider for the CNS UI.

We are going to use these values, exported to environment variables named as per the Name column, in helm-validator-auth-secrets-config and helm-validator-install. When first starting out, it is suggested to configure both JWT token audiences below to the same value: https://canton.network.global. Once you can confirm that your setup is working correctly using this (simple) default, we recommend that you configure dedicated audience values that match your deployment and URLs. This is important for security to avoid tokens for your validators on one network be usable for your validators on another network. You can configure audiences of your choice for the participant ledger API and the validator backend API. We will refer to these using the following configuration values:

Name	Value
OIDC_AUTHORITY_LEDGER_API_AUDIENCE	The audience for the participant ledger API. e.g. `https://ledger_api.example.com`
OIDC_AUTHORITY_VALIDATOR_AUDIENCE	The audience for the validator backend API. e.g. `https://validator.example.com/api`

Your IAM may also require a scope to be specified when the validator backend requests a token for the ledger API. We will refer to that using the following configuration value:

Name	Value
OIDC_AUTHORITY_LEDGER_API_SCOPE	The scope for the participant ledger API. Optional

In case you are facing trouble with setting up your (non-Auth0) OIDC provider, it can be beneficial to skim the instructions in helm-validator-auth0 as well, to check for functionality or configuration details that your OIDC provider setup might be missing.

Configuring an Auth0 Tenant

To configure Auth0 as your validator’s OIDC provider, perform the following:

Create an Auth0 tenant for your validator
Create an Auth0 API that controls access to the ledger API:
1. Navigate to Applications > APIs and click “Create API”. Set name to Daml Ledger API, set identifier to https://canton.network.global. Alternatively, if you would like to configure your own audience, you can set the identifier here. e.g. https://ledger_api.example.com.
2. Under the Permissions tab in the new API, add a permission with scope daml_ledger_api, and a description of your choice.
3. On the Settings tab, scroll down to “Access Settings” and enable “Allow Offline Access”, for automatic token refreshing.
(Optional) If you want to configure a different audience to your APIs, you can do so by creating new Auth0 APIs with an identifier set to the audience of your choice. For example,
1. Create another API by setting name to Validator App API, set identifier for the Validator backend app e.g. https://validator.example.com/api.
Create an Auth0 Application for the validator backend:
1. In Auth0, navigate to Applications -> Applications, and click the “Create Application” button.
2. Name it Validator app backend, choose “Machine to Machine Applications”, and click Create.
3. Choose the Daml Ledger API API you created in step 2 in the “Authorize Machine to Machine Application” dialog and click Authorize.
Create an Auth0 Application for the wallet web UI.
1. In Auth0, navigate to Applications -> Applications, and click the “Create Application” button.
2. Choose “Single Page Web Applications”, call it Wallet web UI, and click Create.
3. Determine the URL for your validator’s wallet UI. If you’re using the ingress configuration of this runbook, that would be https://wallet.validator.YOUR_HOSTNAME.
4. In the Auth0 application settings, add the URL of the validator wallet to the following:
  - “Allowed Callback URLs”
  - “Allowed Logout URLs”
  - “Allowed Web Origins”
  - “Allowed Origins (CORS)”
5. Save your application settings.
Create an Auth0 Application for the CNS web UI. Repeat all steps described in step 5, with following modifications:
- In step b, use CNS web UI as the name of your application.
- In steps c and d, use the URL for your validator’s CNS UI. If you’re using the ingress configuration of this runbook, that would be https://cns.validator.YOUR_HOSTNAME.

Please refer to Auth0’s own documentation on user management for pointers on how to set up end-user accounts for the two web UI applications you created. Note that you will need to create at least one such user account for completing the steps in helm-validator-install - for being able to log in as your Validator node’s administrator. You will be asked to obtain the user identifier for this user account. It can be found in the Auth0 interface under User Management -> Users -> your user’s name -> user_id (a field right under the user’s name at the top). We will use the environment variables listed in the table below to refer to aspects of your Auth0 configuration:

Name	Value
OIDC_AUTHORITY_URL	`https://AUTH0_TENANT_NAME.us.auth0.com`
OIDC_AUTHORITY_LEDGER_API_AUDIENCE	The optional audience of your choice for Ledger API. e.g. `https://ledger_api.example.com`
VALIDATOR_CLIENT_ID	The client id of the Auth0 app for the validator app backend
VALIDATOR_CLIENT_SECRET	The client secret of the Auth0 app for the validator app backend
WALLET_UI_CLIENT_ID	The client id of the Auth0 app for the wallet UI
CNS_UI_CLIENT_ID	The client id of the Auth0 app for the CNS UI

The AUTH0_TENANT_NAME is the name of your Auth0 tenant as shown at the top left of your Auth0 project. You can obtain the client ID and secret of each Auth0 app from the settings pages of that app.

Configuring Authentication on your Validator

We are now going to configure your Validator node software based on the OIDC provider configuration values you exported to environment variables at the end of either helm-validator-auth-requirements or helm-validator-auth0. (Note that some authentication-related configuration steps are also included in helm-validator-install) The validator app backend requires the following secret (omit the scope if it is not needed in your setup)

    kubectl create --namespace validator secret generic splice-app-validator-ledger-api-auth \
        "--from-literal=ledger-api-user=${VALIDATOR_CLIENT_ID}@clients" \
        "--from-literal=url=${OIDC_AUTHORITY_URL}/.well-known/openid-configuration" \
        "--from-literal=client-id=${VALIDATOR_CLIENT_ID}" \
        "--from-literal=client-secret=${VALIDATOR_CLIENT_SECRET}" \
        "--from-literal=audience=${OIDC_AUTHORITY_LEDGER_API_AUDIENCE}" \
        "--from-literal=scope=${OIDC_AUTHORITY_LEDGER_API_SCOPE}"

To setup the wallet and CNS UI, create the following two secrets.

    kubectl create --namespace validator secret generic splice-app-wallet-ui-auth \
        "--from-literal=url=${OIDC_AUTHORITY_URL}" \
        "--from-literal=client-id=${WALLET_UI_CLIENT_ID}"

    kubectl create --namespace validator secret generic splice-app-cns-ui-auth \
        "--from-literal=url=${OIDC_AUTHORITY_URL}" \
        "--from-literal=client-id=${CNS_UI_CLIENT_ID}"

Running without Authentication

Running without authentication is highly insecure. Anyone with access to the wallet UI, or to the validator in any other way, may log in to your wallet as a user of their choice, or otherwise transact on-ledger on your behalf. For any production use, you should configure proper authentication as described in the sections above.

In order to run the validator without authentication, add disableAuth: true to both splice-node/examples/sv-helm/validator-values.yaml and splice-node/examples/sv-helm/participant-values.yaml. Note that you must disable auth in both places, otherwise the validator will not be able to connect to the participant. When running without authentication, the username of the validator administrator is administrator.

Installing the Software

Configuring the Helm Charts

Installing the Helm Charts

Configuring the Cluster Ingress

The following routes should be configured in your cluster ingress controller.

Services	Port	Routes
`wallet-web-ui`		`https://wallet.validator.<YOUR_HOSTNAME>`
`validator-app` `ans-web-ui`	5003	`https://wallet.validator.<YOUR_HOSTNAME>/api/validator` `https://cns.validator.<YOUR_HOSTNAME>`
`validator-app`	5003	`https://cns.validator.<YOUR_HOSTNAME>/api/validator`
`participant`	7575	`https://<YOUR_HOSTNAME>/api/json-api` (optional, not required by the validator itself but if you want to access the ledger API yourself. You can change the route freely)

https://wallet.validator.<YOUR_HOSTNAME> should be routed to service wallet-web-ui in the validator namespace
https://wallet.validator.<YOUR_HOSTNAME>/api/validator should be routed to /api/validator at port 5003 of service validator-app in the validator namespace
https://cns.validator.<YOUR_HOSTNAME> should be routed to service ans-web-ui in the validator namespace
https://cns.validator.<YOUR_HOSTNAME>/api/validator should be routed to /api/validator at port 5003 of service validator-app in the validator namespace

To keep the attack surface on your validator deployment small, please disallow ingress connections to all other services in your validator deployment. It should be assumed that opening up any additional port or service represents a security risk that needs to be carefully evaluated on a case-by-case basis. In addition, it is recommended to restrict access to above services to a limited number of explicitly trusted IP address ranges.

Internet ingress configuration is often specific to the network configuration and scenario of the cluster being configured. To illustrate the basic requirements of a Validator node ingress, we have provided a Helm chart that configures ingress according to the routes above using Istio, as detailed in the sections below.

Requirements

In order to install the reference charts, the following must be satisfied in your cluster:

cert-manager must be available in the cluster (See cert-manager documentation)
istio should be installed in the cluster (See istio documentation)

Note that their deployments are often platform-dependent and good documentations on how to set them up can be found online. Example of Istio installation:

    helm repo add istio https://istio-release.storage.googleapis.com/charts
    helm repo update
    helm install istio-base istio/base -n istio-system --set defaults.global.istioNamespace=cluster-ingress --wait
    helm install istiod istio/istiod -n cluster-ingress --set global.istioNamespace="cluster-ingress" --set meshConfig.accessLogFile="/dev/stdout"  --wait

Installation Instructions

Logging into the wallet UI

After you deploy your ingress, open your browser at https://wallet.validator.YOUR_HOSTNAME and login using the credentials for the user that you configured as validatorWalletUser earlier. Once logged in one should see the transactions page.

explain the config sections below in a way that makes them also accessible to the Docker compose users

Configuring automatic traffic purchases

By default your node will be configured to automatically purchase traffic on a pay-as-you-go basis (see automatically purchase traffic). To disable or tune to your needs, edit the following section in the validator-values.yaml file:

# Configuring a validator's traffic top-up loop;
# see documentation for more detailed information.
topup:
  # set to false in order to disable automatic traffic top-ups
  enabled: true
  # target throughput in bytes / second of sequenced traffic; targetThroughput=0 <=> enabled=false
  targetThroughput: 20000
  # minTopupInterval - minimum time interval that must elapse before the next top-up
  minTopupInterval: "1m"

* show error message that people will see while the traffic purchase fails due to insufficient funds; it is currentlye here: error-insufficient-funds * link to the option to disable automatic top-ups, and call out the option of using third-party traffic providers

Configuring sweeps and auto-accepts of transfer offers

To do so, uncomment and fill in the following section in the validator-values.yaml file:

# If you want funds sweeped out of parties in this validator, uncomment and fill in the following:
#walletSweep:
#  "<senderPartyId>":
#    maxBalanceUSD: <maxBalanceUSD>
#    minBalanceUSD: <minBalanceUSD>
#    receiver: "<receiverPartyId>"
#    useTransferPreapproval: false # sweep by transferring directly through the transfer preapproval of the receiver,
#                                    if set to false sweeping creates transfer offers that need to be accepted on the receiver side.
#                                    Note that this refers to the preapprovals described in https://docs.dev.sync.global/background/preapprovals.html
#                                    and not to auto accepting transfers. Auto accept transfers does not setup preapproval contracts that allow
#                                    for a direct transfer but just automates the acceptance of the transfer offer so in that case
#                                    useTransferPreapproval should be set to false.

Similarly, you can configure the validator to automatically accept transfer offers from certain parties on the network. To do so, uncomment and fill in the following section in the validator-values.yaml file:

# To configure the validator to auto-accept transfer offers from specific parties, uncomment and fill in the following:
#autoAcceptTransfers:
#  "<receiverPartyId>":
#    fromParties:
#      - "<senderPartyId>"

Logging into the CNS UI

You can open your browser at https://cns.validator.YOUR_HOSTNAME and login using the credentials for the user that you configured as validatorWalletUser earlier. You will be able to register a name on the Canton Name Service.

Participant Pruning

By default, participants preserve all history. However, this leads to gradually growing databases and can slow down certain queries, in particular, queries for the active contract set on the ledger API. To mitigate that, it is possible to enable participant pruning which will remove all history beyond a specified retention point and only preserve the active contract set. Note that this only affects the participant stores. The CN apps (Validator, SV and Scan) are unaffected by enabling this, so e.g., the history in your wallet will never be pruned. Below you can see an example of the pruning config that you need to add to validator-values.yaml to retain only the history for the last 48h. Note that if your node is down for longer than the pruning window (48 hours in the example above), your node will most probably get corrupted, as the apps race catching up with the participant’s attempts to keep pruning. It is therefore advisable to set the pruning window to a value that you are comfortable with in terms of guaranteeing uptime of your node. Setting it to 30 days is in general a reasonable choice, as the sequencers currently are also pruned after 30 days, so you will not be able to catch up with the network after a longer downtime anyway (see Disaster Recovery for disaster recovery guidelines). Refer to the Canton documentation for more details on participant pruning:

# To configure participant pruning uncomment the following section.
# Refer to the documentation for more details.
# participantPruningSchedule:
#   cron: 0 /10 * * * ? # Run every 10min
#   maxDuration: 5m # Run for a max of 5min per iteration
#   retention: 48h # Retain history that is newer than 48h.

Configuring init containers

If you need to configure init containers on the participant or validator deployments, you can use the following helm values for splice-participant or splice-validator:

    # if you want to disable the default postgres init container:
    persistence:
      enablePgInitContainer: false

    # if you want additional init containers:
    extraInitContainers:
      - name: my-extra-container
        image: busybox
        command: [ "sh", "-c", "echo 'example extra container'" ]

Working around volume ownership issues

The containers in the splice-validator chart run as non-root users (specifically, user:group 1001:1001) for security reasons. The pod mounts volumes for use by the containers, and these volumes need to be owned by the user that the containers run as. The Helm chart uses an fsGroup security context to ensure that the mounted volumes are owned by the correct user. In certain environments, however, this does not work as expected and the mounted volumes are owned by root. If you encounter this issue, you can work around it by creating init containers that change the ownership of the mounted volumes to the correct user. For example, for the /domain-upgrade-dump volume (required for synchronizer upgrades), you can add the following to your validator-values.yaml file:

    extraInitContainers:
        - name: chown-domain-upgrade-dump
          image: busybox:1.37.0
          command: ["sh", "-c", "chown -R 1001:1001 /domain-upgrade-dump"]
          volumeMounts:
            - name: domain-upgrade-dump-volume
              mountPath: /domain-upgrade-dump

A similar workaround will be required for mounting a usable /participant-bootstrapping-dump (required when recovering from identities backup).

The URL must be reachable from the Canton participant and validator app running in your cluster, as well as from all web browsers that should be able to interact with the wallet and CNS UIs. ↩

Overview

Validator Deployment

Super Validator Deployment

Splice Fundamentals

Canton Console

Production Operations

Extension Synchronizers

Troubleshooting

Release Notes

Reference

Community Resources

Requirements

Validator Network Diagram

Preparing a Cluster for Installation

HTTP Proxy configuration

Bypassing the proxy for specific hosts

Configuring PostgreSQL authentication

Preparing for Validator Onboarding

Configuring Authentication

OIDC Provider Requirements

Configuring an Auth0 Tenant

Configuring Authentication on your Validator

Running without Authentication

Installing the Software

Configuring the Helm Charts

Installing the Helm Charts

Configuring the Cluster Ingress

Requirements

Installation Instructions

Logging into the wallet UI

Configuring automatic traffic purchases

Configuring sweeps and auto-accepts of transfer offers

Logging into the CNS UI

Participant Pruning

Configuring init containers

Working around volume ownership issues

Overview

Validator Deployment

Super Validator Deployment

Splice Fundamentals

Canton Console

Production Operations

Extension Synchronizers

Troubleshooting

Release Notes

Reference

Community Resources

Documentation Index

​Requirements

​Validator Network Diagram

​Preparing a Cluster for Installation

​HTTP Proxy configuration

​Bypassing the proxy for specific hosts

​Configuring PostgreSQL authentication

​Preparing for Validator Onboarding

​Configuring Authentication

​OIDC Provider Requirements

​Configuring an Auth0 Tenant

​Configuring Authentication on your Validator

​Running without Authentication

​Installing the Software

​Configuring the Helm Charts

​Installing the Helm Charts

​Configuring the Cluster Ingress

​Requirements

​Installation Instructions

​Logging into the wallet UI

​Configuring automatic traffic purchases

​Configuring sweeps and auto-accepts of transfer offers

​Logging into the CNS UI

​Participant Pruning

​Configuring init containers

​Working around volume ownership issues

Footnotes

Requirements

Validator Network Diagram

Preparing a Cluster for Installation

HTTP Proxy configuration

Bypassing the proxy for specific hosts

Configuring PostgreSQL authentication

Preparing for Validator Onboarding

Configuring Authentication

OIDC Provider Requirements

Configuring an Auth0 Tenant

Configuring Authentication on your Validator

Running without Authentication

Installing the Software

Configuring the Helm Charts

Installing the Helm Charts

Configuring the Cluster Ingress

Requirements

Installation Instructions

Logging into the wallet UI

Configuring automatic traffic purchases

Configuring sweeps and auto-accepts of transfer offers

Logging into the CNS UI

Participant Pruning

Configuring init containers

Working around volume ownership issues