Product: Semarchy Data Platform Self-Hosted
Version: 1.4.2 (compared with xDM 2025.1)
Platforms: Amazon Web Services, Microsoft Azure, Rancher RKE2, Red Hat OpenShift


Need

You run Semarchy xDM today and plan to run the self-hosted edition of the Semarchy Data Platform (SDP). You remain responsible for operating the platform, but the platform itself changes: a Java application on an application server becomes a set of services on Kubernetes, installed with Helm and backed by four external services. This article lists what changes for administrators, area by area, and what to plan before go-live. It is written for platform administrators, database administrators, DevOps engineers and architects.


SDP SaaS, the edition hosted by Semarchy, is not covered here.


Summarized Solution


1.  Design moves from the data model to the data product, which you deploy, version, promote and assign ownership to as one unit.

2.  Architecture: one application server and one repository database become a Kubernetes cluster plus PostgreSQL, Kafka, OpenSearch and an SMTP relay.

3.  Installation and upgrades are Helm operations driven by a values file kept under change control.

4.  Access is managed at two levels: the platform (users, groups, single sign-on) and Data Management (roles), linked by role mapping.

5.  Environments are Helm releases, and data products are deployed from the command line with the Design CLI.

6.  Routine Data Management administration moves from the Configuration module and Management perspective to the Platform CLI (sem dm admin).

7.  Logs are centralized in Log Explorer; monitoring, backup and disaster recovery stay your team's responsibility, across more components.

8.  Security work shifts to Kubernetes secrets, ingress certificates and one least-privilege identity per backing service.

9.  SDP Self-Hosted has its own versioning (major.minor.patch) and a license key that expires every year.


Detailed Solution


Terms used below: Semarchy xDM is the installation you operate today, made up of an application server, a repository database and the Configuration module. SDP Self-Hosted is the edition that your team deploys with Helm on a Kubernetes cluster; xDM capabilities are delivered by its Data Management module.

1. From data models to data products

SDP replaces the data model with the data product as the primary unit of design, ownership and deployment. A data product is versioned and deployed as one unit, and governed once for all of its consumers.

Semarchy xDM

SDP Self-Hosted

Primary unit

Technical components: models, pipelines, deployments

The data product

Scope

Data management, integration and consumption handled separately

One data product contains the data model, the data integrations and the data consumption services

Ownership and reuse

Limited

Defined ownership, lifecycle and governance for each data product

Semarchy defines three maturity levels, and you decide which capabilities each data product includes:

  1. Essential: certified master data, applications and workflows for data stewards and business users, and integration pipelines.
  2. Advanced: adds developer APIs, embedded governance, and lineage and explainability.
  3. AI-native: adds semantic models, AI-ready assets and agentic access through governed vectors, RAG and MCP.

Expected: you can name, for each current xDM model, the data product that replaces it and its owner. Golden records and data stewardship remain part of the data product.


2. Architecture and deployment

SDP Self-Hosted runs on Amazon Elastic Kubernetes Service, Azure Kubernetes Service, Red Hat OpenShift or Rancher RKE2, as listed in the hosting requirements. Helm charts and container images are pulled from the Semarchy container registry with the registry credentials available in the License portal. Images are built for the x86_64 architecture.

Component

Role in the platform

Documented minimum

Kubernetes

Runs all platform services in a dedicated namespace

Kubernetes 1.32 to 1.35

PostgreSQL

Stores Keycloak and Data Management data

PostgreSQL 16.4 or later, 2 GB RAM, 2 vCPUs

Kafka-compatible service

Platform events, notifications and messaging between services

Kafka 3.8 to 4.2, 2 GB RAM, 2 vCPUs

OpenSearch

Search and central log store

OpenSearch 2.17 or later, 1 GB RAM, 2 vCPUs

SMTP relay

User and notification emails

Reachable from the SDP cluster, with basic or OAuth 2.0 authentication

Ingress and certificates

External HTTPS access

Traefik and cert-manager in the reference setup

These minimums support a basic installation: size the cluster and the data location databases for your users, data volumes and workloads. Requirements change between platform versions, so check the requirements page for the version you install.

Expected: the four external services and the cluster are provisioned and sized before you run helm upgrade --install. What stays the same: you still design your data products and own the databases that hold your master data.


3. Installation, upgrades and patching

Installations and upgrades are Helm operations driven by a values file. Some platform versions require changes to that file before you upgrade. Semarchy xDM follows a quarterly release cadence with LTS and MS labels; SDP Self-Hosted is versioned by platform version, each with its own release notes and upgrade page.

The installation follows these steps:

  1. Review the system requirements and retrieve your registry credentials from the License portal.
  2. Provision Kubernetes, PostgreSQL, Kafka, OpenSearch, the SMTP relay, certificates and DNS.
  3. Configure Kubernetes: namespace, secrets, ingress controller and certificate manager.
  4. Prepare the Helm values file with database connections, resource limits and ingress settings.
  5. Run the prechecks, then install with Helm. The installation takes at least 10 minutes.
  6. Create the first administrator account.

The update procedure applies to configuration changes, secret changes and new platform versions. Run each update as follows:

  1. Ask connected users to save their work, because the update restarts all platform applications.
  2. Back up the Semarchy databases, and record your current Helm values and search engine configuration.
  3. For production, rehearse the upgrade on a test cluster first.
  4. Run helm upgrade with the target version and your values file.
  5. Check the login screen and confirm that all containers are running.
export NAMESPACE=<SDP_NAMESPACE>
export RELEASE=<HELM_RELEASE_NAME>     # for example: sdp

helm upgrade --install "$RELEASE" <CHART_REFERENCE> -n "$NAMESPACE" -f values.yaml --version <TARGET_VERSION>
kubectl get pods -n "$NAMESPACE"

Expected: helm status "$RELEASE" -n "$NAMESPACE" reports deployed and every pod is Running. To check the result in detail, follow How to verify an SDP Self-Hosted installation (see Related articles). Keep the Platform CLI, the Design CLI and the VS Code extension within the versions supported by your platform version.


4. Identity and access management

SDP Self-Hosted provides the same capabilities as Semarchy xDM for connecting external identity providers and for defining users and roles. Because SDP identity management covers every module of the platform, authorization is managed at two levels:

Level

What you manage

Platform

Users, groups and access to platform modules, and external identity providers for single sign-on

Data Management

Data Management roles and privileges, as in Semarchy xDM. Linked to the platform level through role mapping between platform groups and Data Management roles

Expected: before go-live, your identity provider connections and role mappings are re-created, and sign-in is tested for each user profile.


5. Environments and promotion

In SDP, an environment is a Helm release. Data products move between environments through command-line deployments instead of the Application Builder. Several environments can share the same prerequisite services, as described in deployment topology.

Task

Semarchy xDM

SDP Self-Hosted

Author data products

Application Builder

VS Code extension

Deploy a data product

Deploy a model edition from the Management perspective

Design CLI deploy command, which you can run in a CI/CD pipeline. It takes the instance URL, datasource, data location and credentials

Add an environment

New installation or new data location

New installation or new data location

Review deployments and loads

Management perspective

Platform CLI deployments, loads and integration batches commands

Manage datasources

Configuration module

Platform CLI datasource commands

Expected: data products are kept in source control, deployed from a pipeline that uses a service identity, and the Helm values of each environment are in version control, with secrets stored separately.


6. Day-to-day Data Management administration

The settings that you manage today in the xDM Configuration module and Management perspective are available as sem dm admin commands in the Platform CLI, so you can script routine administration.

Task

Semarchy xDM

SDP Self-Hosted (Platform CLI)

Platform status and repository

System and Repository Information screen; repository upgrade during the xDM upgrade

status, info, repository and repository upgrade (System Admin role required)

API keys

API Keys: roles, default user, expiry date, IP restrictions, deactivation

api-keys: list, create, get, set, set-all, delete

Data Management roles

Roles with platform privileges

roles: list, get, set, set-all, delete

User role assignments

Users screen, including users and passwords of the internal identity provider

users: list profiles, get, assign and unassign roles. User accounts are managed at platform level

Notification servers

Notification Servers for job and workflow emails

notification-servers: list, get, set, set-all, delete, test

Variable value providers

Variable Value Providers (datasource or LDAP)

variable-providers: list, get, set, set-all, delete, test

Plugins

Plugins: install or update a JAR bundle

plugin-bundles: list, get, import, export, delete (System Admin role required)

REST clients

REST Clients for enrichment calls

rest-clients: list, get, set, set-all, delete, test

Image libraries

Image Libraries

image-libraries: list, export, import

Custom translations

Custom Translations: export, edit and import language files

translations: list, set

Global application settings

Global Configuration: server base URL, export limits, autocomplete, header logo

properties: list and set the Data Management properties

Execution engine

Execution Engine editor: clusters, queues, suspended jobs

execution-engine: status, start, stop, clusters, queues

Integration batch poller

Integration Batch Poller editor: stop, start, polling schedule

integration-batch-poller: status, start, stop, configuration set

Job logs and purge

Job Logs editor: filter, drill down, purge

job-executions: filter by status, type, batch, owner or period, get, purge; job-definitions

Recommended practices:

  • Export each configuration with --json and re-apply it with set-all -i <file>, so that the notification servers, REST clients, variable providers, roles and API keys of each environment are kept in version control.
  • Run the test commands for notification servers, REST clients and variable providers after each change or upgrade.
  • Use --read-only for audits, and give pipelines an OAuth2 client (--client-id, --client-secret) instead of a personal account.

Expected: each environment's configuration is exported as JSON in version control, and the test commands succeed after every upgrade.


7. Monitoring, logging, backup and disaster recovery

You monitor and protect more components than before. SDP Self-Hosted includes a Log Explorer service that filters, searches and aggregates logs across the platform, and a log collector in the identity and access management component. Logs are stored in OpenSearch, and a setup job creates the OpenSearch index configuration when it does not exist. The Platform CLI probes check the readiness of a data location, the REST API and each application, and can be used for external health checks.

Concern

Semarchy xDM

SDP Self-Hosted

Infrastructure monitoring

Host, JVM, database

Cluster, nodes, pods, PostgreSQL, Kafka, OpenSearch

Platform logs

Server and trace logs

Log Explorer, backed by OpenSearch

Log retention

Your file rotation policy

Lifecycle policy installed by default in the search engine; size OpenSearch for your retention needs

Application health checks

Built by your team

Platform CLI probes

Backup scope

Repository and data location databases

PostgreSQL, data location databases, Helm values and secrets

Disaster recovery

Designed by your team

Designed by your team, across Kubernetes and the four external services

Semarchy does not set backup, restore or recovery-time objectives for self-hosted deployments: you define them for each component.

Backup and restore highlights

The persistent state of the platform is held in three places: the PostgreSQL databases for the repository and Keycloak, your data locations, and your data products and configuration in source control. A typical approach uses the native backup and point-in-time recovery of your managed PostgreSQL service for the repository and Keycloak, a similar service from your database provider for the data locations, and rebuilds the Kubernetes deployment from Git and your container images. The Kafka topic carries only transient user events, so it does not need a separate backup.

Also protect the following items:

  1. Kubernetes secrets created before installation: PostgreSQL, Kafka, OpenSearch, mail, registry pull and TLS or CA secrets. Keep them in a secrets manager, not only in the cluster.
  2. The Helm values file and the license key. When you rebuild, keep global.site_id and global.site_name unchanged, because they identify the Keycloak realm and the setup state.
  3. The license binding to the namespace. A production license is tied to the UID of your Kubernetes namespace. A namespace rebuilt during disaster recovery gets a new UID, so the license must be re-registered to that UID in the License portal before Keycloak starts. Make sure that a person with License portal access is available during recovery.
  4. Search engine: it holds technical logs and the license usage metrics. Losing it does not stop the platform, but you lose log and usage history. Take snapshots if you need either.
  5. A consistent restore point: restore the repository and Keycloak databases to the same point in time, and as close as possible to the restore point of the data locations. This prevents mismatches between users, roles and deployed data products.

Expected: a full restore, including identity data in PostgreSQL and the search store, has been tested in a non-production environment before go-live. A rebuild from your container registry and Git usually takes less than an hour. The two steps most likely to extend it are re-registering the license and recovering secrets that are missing from your secrets manager.

Important: helm rollback is not supported for SDP Self-Hosted. To return to an earlier version, restore from your backups.


8. Security, certificates and network administration

Security work shifts toward Kubernetes secrets, credentials between services, and certificates for every connection between the platform and its backing services. In the Kubernetes setup, sensitive connection details are stored in Kubernetes secrets: PostgreSQL roles and passwords, SMTP credentials and registry credentials. If OpenSearch uses a private or custom certificate authority, create a secret that holds the CA certificate before deployment. The external services guide has you create dedicated Kafka users with SASL authentication and a dedicated OpenSearch role and user, so that each service has its own least-privilege identity.

Topic

Semarchy xDM

SDP Self-Hosted

Secrets

Application server and datasource settings

Kubernetes secrets and Helm values

TLS

Application server or reverse proxy

Ingress with cert-manager in the reference setup, or your own certificates

Service accounts

Repository database account

Separate PostgreSQL, Kafka and OpenSearch identities

Private certificate authority

Java truststore

CA certificate secret referenced in the Helm values

Network flows

Users to server, server to databases

Users to ingress, cluster to the four services, and cluster to the Semarchy image registry (except in an air-gapped deployment)

Hardening

Operating system, JVM, application server

Cluster policies, node images, network policies and service-level security

The reference setup installs Traefik as the ingress controller and cert-manager for TLS certificates, and requires a wildcard DNS record that points to the ingress load balancer. Review the Helm chart defaults against your security policy. If you do not use cert-manager, plan certificate rotation with another method. SDP supports Kubernetes-compatible secret management solutions such as Azure Key Vault or AWS Secrets Manager.

Expected: every secret is stored outside the cluster as well, each backing service has its own identity, and certificate renewal is automated or scheduled.


9. Licensing, support and version lifecycle

SDP Self-Hosted is activated with a license key and uses its own version numbering (major.minor.patch) instead of the xDM year.quarter.patch scheme. Each platform version, from 1.0.0 to 1.4.2 at the date of this article, has its own release notes and upgrade page. Semarchy gives you access to the License portal, where you retrieve your registry credentials and your license key.

Topic

Semarchy xDM

SDP Self-Hosted

Versioning

Year.quarter.patch, labeled LTS or MS

Platform version major.minor.patch, with module versions inside each platform version

License

License key

License key

Upgrade obligation

Stay within the support window

Stay within the support window of your platform version

Pricing

Contact your Semarchy account team

Contact your Semarchy account team

The license key expires every year. On the expiration date:

  1. In the License portal, check that the expiration date has been extended.
  2. Copy the new license key into your values.yaml file.
  3. Restart the Keycloak pod so that the platform uses the new key.

Expected: the new license key is in values.yaml and the platform is running with it after the Keycloak pod restarts.


Checklist

Area

Expected

☐

Data products

Each xDM model mapped to a data product with an owner

☐

Architecture

Kubernetes, PostgreSQL, Kafka, OpenSearch and SMTP provisioned and sized

☐

Installation and upgrades

Values file under change control, upgrade rehearsed on a test cluster

☐

Identity and access

Identity providers and role mappings re-created, sign-in tested per user profile

☐

Environments

Data products deployed with the Design CLI from a pipeline using a service identity

☐

Administration

Configuration exported as JSON per environment, test commands succeed

☐

Monitoring

Log Explorer reachable, Platform CLI probes used for health checks

☐

Backup and recovery

Full restore tested, including identity data and the search store

☐

License and recovery runbook

License re-registration to the new namespace UID documented, License portal contact available

☐

Security

Secrets stored outside the cluster, certificate renewal automated or scheduled

☐

License renewal

Yearly key renewal scheduled before the expiration date


Related articles