Product: Semarchy Data Platform Self-Hosted
Version: 1.4.2 (compared with xDM 2025.1)
Platforms: Amazon Web Services, Microsoft Azure, Rancher RKE2, Red Hat OpenShift
Need
You run Semarchy xDM today and plan to run the self-hosted edition of the Semarchy Data Platform (SDP). You remain responsible for operating the platform, but the platform itself changes: a Java application on an application server becomes a set of services on Kubernetes, installed with Helm and backed by four external services. This article lists what changes for administrators, area by area, and what to plan before go-live. It is written for platform administrators, database administrators, DevOps engineers and architects.
SDP SaaS, the edition hosted by Semarchy, is not covered here.
Summarized Solution
1. Design moves from the data model to the data product, which you deploy, version, promote and assign ownership to as one unit.
2. Architecture: one application server and one repository database become a Kubernetes cluster plus PostgreSQL, Kafka, OpenSearch and an SMTP relay.
3. Installation and upgrades are Helm operations driven by a values file kept under change control.
4. Access is managed at two levels: the platform (users, groups, single sign-on) and Data Management (roles), linked by role mapping.
5. Environments are Helm releases, and data products are deployed from the command line with the Design CLI.
6. Routine Data Management administration moves from the Configuration module and Management perspective to the Platform CLI (sem dm admin).
7. Logs are centralized in Log Explorer; monitoring, backup and disaster recovery stay your team's responsibility, across more components.
8. Security work shifts to Kubernetes secrets, ingress certificates and one least-privilege identity per backing service.
9. SDP Self-Hosted has its own versioning (major.minor.patch) and a license key that expires every year.
Detailed Solution
Terms used below: Semarchy xDM is the installation you operate today, made up of an application server, a repository database and the Configuration module. SDP Self-Hosted is the edition that your team deploys with Helm on a Kubernetes cluster; xDM capabilities are delivered by its Data Management module.
1. From data models to data products
SDP replaces the data model with the data product as the primary unit of design, ownership and deployment. A data product is versioned and deployed as one unit, and governed once for all of its consumers.
Semarchy xDM | SDP Self-Hosted | |
|---|---|---|
Primary unit | Technical components: models, pipelines, deployments | The data product |
Scope | Data management, integration and consumption handled separately | One data product contains the data model, the data integrations and the data consumption services |
Ownership and reuse | Limited | Defined ownership, lifecycle and governance for each data product |
Semarchy defines three maturity levels, and you decide which capabilities each data product includes:
- Essential: certified master data, applications and workflows for data stewards and business users, and integration pipelines.
- Advanced: adds developer APIs, embedded governance, and lineage and explainability.
- AI-native: adds semantic models, AI-ready assets and agentic access through governed vectors, RAG and MCP.
Expected: you can name, for each current xDM model, the data product that replaces it and its owner. Golden records and data stewardship remain part of the data product.
2. Architecture and deployment
SDP Self-Hosted runs on Amazon Elastic Kubernetes Service, Azure Kubernetes Service, Red Hat OpenShift or Rancher RKE2, as listed in the hosting requirements. Helm charts and container images are pulled from the Semarchy container registry with the registry credentials available in the License portal. Images are built for the x86_64 architecture.
Component | Role in the platform | Documented minimum |
|---|---|---|
Kubernetes | Runs all platform services in a dedicated namespace | Kubernetes 1.32 to 1.35 |
PostgreSQL | Stores Keycloak and Data Management data | PostgreSQL 16.4 or later, 2 GB RAM, 2 vCPUs |
Kafka-compatible service | Platform events, notifications and messaging between services | Kafka 3.8 to 4.2, 2 GB RAM, 2 vCPUs |
OpenSearch | Search and central log store | OpenSearch 2.17 or later, 1 GB RAM, 2 vCPUs |
SMTP relay | User and notification emails | Reachable from the SDP cluster, with basic or OAuth 2.0 authentication |
Ingress and certificates | External HTTPS access | Traefik and cert-manager in the reference setup |
These minimums support a basic installation: size the cluster and the data location databases for your users, data volumes and workloads. Requirements change between platform versions, so check the requirements page for the version you install.
Expected: the four external services and the cluster are provisioned and sized before you run helm upgrade --install. What stays the same: you still design your data products and own the databases that hold your master data.
3. Installation, upgrades and patching
Installations and upgrades are Helm operations driven by a values file. Some platform versions require changes to that file before you upgrade. Semarchy xDM follows a quarterly release cadence with LTS and MS labels; SDP Self-Hosted is versioned by platform version, each with its own release notes and upgrade page.
The installation follows these steps:
- Review the system requirements and retrieve your registry credentials from the License portal.
- Provision Kubernetes, PostgreSQL, Kafka, OpenSearch, the SMTP relay, certificates and DNS.
- Configure Kubernetes: namespace, secrets, ingress controller and certificate manager.
- Prepare the Helm values file with database connections, resource limits and ingress settings.
- Run the prechecks, then install with Helm. The installation takes at least 10 minutes.
- Create the first administrator account.
The update procedure applies to configuration changes, secret changes and new platform versions. Run each update as follows:
- Ask connected users to save their work, because the update restarts all platform applications.
- Back up the Semarchy databases, and record your current Helm values and search engine configuration.
- For production, rehearse the upgrade on a test cluster first.
- Run
helm upgradewith the target version and your values file. - Check the login screen and confirm that all containers are running.
export NAMESPACE=<SDP_NAMESPACE> export RELEASE=<HELM_RELEASE_NAME> # for example: sdp helm upgrade --install "$RELEASE" <CHART_REFERENCE> -n "$NAMESPACE" -f values.yaml --version <TARGET_VERSION> kubectl get pods -n "$NAMESPACE"
Expected: helm status "$RELEASE" -n "$NAMESPACE" reports deployed and every pod is Running. To check the result in detail, follow How to verify an SDP Self-Hosted installation (see Related articles). Keep the Platform CLI, the Design CLI and the VS Code extension within the versions supported by your platform version.
4. Identity and access management
SDP Self-Hosted provides the same capabilities as Semarchy xDM for connecting external identity providers and for defining users and roles. Because SDP identity management covers every module of the platform, authorization is managed at two levels:
Level | What you manage |
|---|---|
Platform | Users, groups and access to platform modules, and external identity providers for single sign-on |
Data Management | Data Management roles and privileges, as in Semarchy xDM. Linked to the platform level through role mapping between platform groups and Data Management roles |
Expected: before go-live, your identity provider connections and role mappings are re-created, and sign-in is tested for each user profile.
5. Environments and promotion
In SDP, an environment is a Helm release. Data products move between environments through command-line deployments instead of the Application Builder. Several environments can share the same prerequisite services, as described in deployment topology.
Task | Semarchy xDM | SDP Self-Hosted |
|---|---|---|
Author data products | Application Builder | |
Deploy a data product | Deploy a model edition from the Management perspective | Design CLI deploy command, which you can run in a CI/CD pipeline. It takes the instance URL, datasource, data location and credentials |
Add an environment | New installation or new data location | New installation or new data location |
Review deployments and loads | Management perspective | Platform CLI deployments, loads and integration batches commands |
Manage datasources | Configuration module | Platform CLI datasource commands |
Expected: data products are kept in source control, deployed from a pipeline that uses a service identity, and the Helm values of each environment are in version control, with secrets stored separately.
6. Day-to-day Data Management administration
The settings that you manage today in the xDM Configuration module and Management perspective are available as sem dm admin commands in the Platform CLI, so you can script routine administration.
Task | Semarchy xDM | SDP Self-Hosted (Platform CLI) |
|---|---|---|
Platform status and repository | System and Repository Information screen; repository upgrade during the xDM upgrade |
|
API keys | API Keys: roles, default user, expiry date, IP restrictions, deactivation |
|
Data Management roles | Roles with platform privileges |
|
User role assignments | Users screen, including users and passwords of the internal identity provider |
|
Notification servers | Notification Servers for job and workflow emails |
|
Variable value providers | Variable Value Providers (datasource or LDAP) |
|
Plugins | Plugins: install or update a JAR bundle |
|
REST clients | REST Clients for enrichment calls |
|
Image libraries | Image Libraries |
|
Custom translations | Custom Translations: export, edit and import language files |
|
Global application settings | Global Configuration: server base URL, export limits, autocomplete, header logo |
|
Execution engine | Execution Engine editor: clusters, queues, suspended jobs |
|
Integration batch poller | Integration Batch Poller editor: stop, start, polling schedule |
|
Job logs and purge | Job Logs editor: filter, drill down, purge |
|
Recommended practices:
- Export each configuration with
--jsonand re-apply it withset-all -i <file>, so that the notification servers, REST clients, variable providers, roles and API keys of each environment are kept in version control. - Run the
testcommands for notification servers, REST clients and variable providers after each change or upgrade. - Use
--read-onlyfor audits, and give pipelines an OAuth2 client (--client-id,--client-secret) instead of a personal account.
Expected: each environment's configuration is exported as JSON in version control, and the test commands succeed after every upgrade.
7. Monitoring, logging, backup and disaster recovery
You monitor and protect more components than before. SDP Self-Hosted includes a Log Explorer service that filters, searches and aggregates logs across the platform, and a log collector in the identity and access management component. Logs are stored in OpenSearch, and a setup job creates the OpenSearch index configuration when it does not exist. The Platform CLI probes check the readiness of a data location, the REST API and each application, and can be used for external health checks.
Concern | Semarchy xDM | SDP Self-Hosted |
|---|---|---|
Infrastructure monitoring | Host, JVM, database | Cluster, nodes, pods, PostgreSQL, Kafka, OpenSearch |
Platform logs | Server and trace logs | Log Explorer, backed by OpenSearch |
Log retention | Your file rotation policy | Lifecycle policy installed by default in the search engine; size OpenSearch for your retention needs |
Application health checks | Built by your team | Platform CLI probes |
Backup scope | Repository and data location databases | PostgreSQL, data location databases, Helm values and secrets |
Disaster recovery | Designed by your team | Designed by your team, across Kubernetes and the four external services |
Semarchy does not set backup, restore or recovery-time objectives for self-hosted deployments: you define them for each component.
Backup and restore highlights
The persistent state of the platform is held in three places: the PostgreSQL databases for the repository and Keycloak, your data locations, and your data products and configuration in source control. A typical approach uses the native backup and point-in-time recovery of your managed PostgreSQL service for the repository and Keycloak, a similar service from your database provider for the data locations, and rebuilds the Kubernetes deployment from Git and your container images. The Kafka topic carries only transient user events, so it does not need a separate backup.
Also protect the following items:
- Kubernetes secrets created before installation: PostgreSQL, Kafka, OpenSearch, mail, registry pull and TLS or CA secrets. Keep them in a secrets manager, not only in the cluster.
- The Helm values file and the license key. When you rebuild, keep
global.site_idandglobal.site_nameunchanged, because they identify the Keycloak realm and the setup state. - The license binding to the namespace. A production license is tied to the UID of your Kubernetes namespace. A namespace rebuilt during disaster recovery gets a new UID, so the license must be re-registered to that UID in the License portal before Keycloak starts. Make sure that a person with License portal access is available during recovery.
- Search engine: it holds technical logs and the license usage metrics. Losing it does not stop the platform, but you lose log and usage history. Take snapshots if you need either.
- A consistent restore point: restore the repository and Keycloak databases to the same point in time, and as close as possible to the restore point of the data locations. This prevents mismatches between users, roles and deployed data products.
Expected: a full restore, including identity data in PostgreSQL and the search store, has been tested in a non-production environment before go-live. A rebuild from your container registry and Git usually takes less than an hour. The two steps most likely to extend it are re-registering the license and recovering secrets that are missing from your secrets manager.
Important: helm rollback is not supported for SDP Self-Hosted. To return to an earlier version, restore from your backups.
8. Security, certificates and network administration
Security work shifts toward Kubernetes secrets, credentials between services, and certificates for every connection between the platform and its backing services. In the Kubernetes setup, sensitive connection details are stored in Kubernetes secrets: PostgreSQL roles and passwords, SMTP credentials and registry credentials. If OpenSearch uses a private or custom certificate authority, create a secret that holds the CA certificate before deployment. The external services guide has you create dedicated Kafka users with SASL authentication and a dedicated OpenSearch role and user, so that each service has its own least-privilege identity.
Topic | Semarchy xDM | SDP Self-Hosted |
|---|---|---|
Secrets | Application server and datasource settings | Kubernetes secrets and Helm values |
TLS | Application server or reverse proxy | Ingress with cert-manager in the reference setup, or your own certificates |
Service accounts | Repository database account | Separate PostgreSQL, Kafka and OpenSearch identities |
Private certificate authority | Java truststore | CA certificate secret referenced in the Helm values |
Network flows | Users to server, server to databases | Users to ingress, cluster to the four services, and cluster to the Semarchy image registry (except in an air-gapped deployment) |
Hardening | Operating system, JVM, application server | Cluster policies, node images, network policies and service-level security |
The reference setup installs Traefik as the ingress controller and cert-manager for TLS certificates, and requires a wildcard DNS record that points to the ingress load balancer. Review the Helm chart defaults against your security policy. If you do not use cert-manager, plan certificate rotation with another method. SDP supports Kubernetes-compatible secret management solutions such as Azure Key Vault or AWS Secrets Manager.
Expected: every secret is stored outside the cluster as well, each backing service has its own identity, and certificate renewal is automated or scheduled.
9. Licensing, support and version lifecycle
SDP Self-Hosted is activated with a license key and uses its own version numbering (major.minor.patch) instead of the xDM year.quarter.patch scheme. Each platform version, from 1.0.0 to 1.4.2 at the date of this article, has its own release notes and upgrade page. Semarchy gives you access to the License portal, where you retrieve your registry credentials and your license key.
Topic | Semarchy xDM | SDP Self-Hosted |
|---|---|---|
Versioning | Year.quarter.patch, labeled LTS or MS | Platform version major.minor.patch, with module versions inside each platform version |
License | License key | License key |
Upgrade obligation | Stay within the support window | Stay within the support window of your platform version |
Pricing | Contact your Semarchy account team | Contact your Semarchy account team |
The license key expires every year. On the expiration date:
- In the License portal, check that the expiration date has been extended.
- Copy the new license key into your
values.yamlfile. - Restart the Keycloak pod so that the platform uses the new key.
Expected: the new license key is in values.yaml and the platform is running with it after the Keycloak pod restarts.
Checklist
Area | Expected | |
|---|---|---|
☐ | Data products | Each xDM model mapped to a data product with an owner |
☐ | Architecture | Kubernetes, PostgreSQL, Kafka, OpenSearch and SMTP provisioned and sized |
☐ | Installation and upgrades | Values file under change control, upgrade rehearsed on a test cluster |
☐ | Identity and access | Identity providers and role mappings re-created, sign-in tested per user profile |
☐ | Environments | Data products deployed with the Design CLI from a pipeline using a service identity |
☐ | Administration | Configuration exported as JSON per environment, |
☐ | Monitoring | Log Explorer reachable, Platform CLI probes used for health checks |
☐ | Backup and recovery | Full restore tested, including identity data and the search store |
☐ | License and recovery runbook | License re-registration to the new namespace UID documented, License portal contact available |
☐ | Security | Secrets stored outside the cluster, certificate renewal automated or scheduled |
☐ | License renewal | Yearly key renewal scheduled before the expiration date |