Author: Derek Leung
Product version: 2024.1.X
Need
Customer wants to install Semarchy xDI Analytics on AWS ECS to enable scalable, flexible, and cost-effective deployment of this data management platform in cloud environment.
Summarized Solution
On a high level this installation needs to setup a RDS cluster where the database will be hoisted and a setup of xDI Analytics docker image on ECS container. To access both DB and xDI, the respective inbound rules are needed to open the connection ports.
This article provides instruction on how to install RDS with or without SSL encryption and assumes your are creating an HTTP based ECS container. If you require HTTPS to be enabled on your instance browse related articles in our KB or on AWS support site to get instructions.
Detailed Solution
Prerequisites and requirements
Review the information in this section before you begin the installation.
You will need the following to install Semarchy xDI on AWS:
- An AWS subscription. If you don’t have an AWS subscription, create a free account before you begin. This user requires administrator privileges.
- The AWS command-line interface (CLI). It is installed and configured with your AWS account. For more information, see Getting started with the AWS CLI.
- One AWS Virtual Private Cloud (VPC). It is configured with at least two subnets from different availability zones. For more information, see Create an IPv4-enabled VPC and subnets using the AWS CLI.
- A SQL management tool, such as DBeaver, pgAdmin or another one.
Step 1. Check the AWS credentials
To access AWS services with the AWS CLI, you need at minimum an AWS account and IAM credentials.
Installing or updating to the latest version of the AWS CLI
https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
Create an access key and secret token to configure AWS.
This is the only time that the secret access key can be viewed or downloaded. You cannot recover it later. However, you can create a new access key any time.
1. Go to IAM - Users - <Your user> --> Create access key
To check all iam users – use the following command:
aws iam list-users
To check only your user, you can use the following command:
aws iam get-user
Step 2. Create the Aurora RDS instance
Note: the below scripts use the default vpc and subnets. Ensure your vpc and subnets are properly configured and if you are not using default you might need to add those parameters in the script
Note: across all items in this article we are using the following names that you can change for your installation:
DB cluster ID: uk-xdi
Master user name: postgres
Master user password: <your_password>
Database name: postgres
DB cluster ID: uk-xdi
Master user name: postgres
Master user password: <your_password>
Database name: postgres
aws rds create-db-cluster \
--db-cluster-identifier uk-xdi \
--engine aurora-postgresql \
--master-username postgres\
--master-user-password <your_password>\
--database-name postgres \
--tags Key=xdi-ecs2. Run the following command to create the writer instance:
aws rds create-db-instance \
--db-cluster-identifier uk-xdi \
--db-instance-identifier instance-1 \
--db-instance-class db.t4g.medium \
--engine aurora-postgresql \
--db-subnet-group-name default \
--tags Key=xdi-ecs4. Follow the cluster and instance creation progress using the AWS console and wait for their status to change to Available (estimated time: 10 minutes). Note the endpoint URLs for later:
Step 3. Create an inbound rule for DB instance accessibility
1. Go to the DB instance on RDS
2. Go to Connectivity & Security and under Security section click on link for VPC security groups.
3. Go to Security group ID
4. Add an inbound rule for port 5432
5. Using any DB client try to connect to DB
You can get connect details as below.
6. For Host name and port go to the DB instance RDS > Databases > uk-xdi > instance-1 > Connectivity and Security
7. Go to Configuration for the DB Name
8. Create a schema in DB based on official documentation.
CREATE SCHEMA extensions; GRANT USAGE ON SCHEMA extensions TO PUBLIC; ALTER DEFAULT PRIVILEGES IN SCHEMA extensions GRANT EXECUTE ON FUNCTIONS TO PUBLIC; ALTER DATABASE postgres SET SEARCH_PATH TO "$user",public,extensions; CREATE USER analytics WITH PASSWORD 'analytics'; GRANT analytics to postgres; CREATE SCHEMA analytics AUTHORIZATION analytics;
Step 4. Create EFS for ECS
The xDI Runtime Container running on ECS Cluster needs to mount an Amazon EFS for storing the configuration and modules.
Consult IT for the following privileges to create EFS:
elasticfilesystem:TagResource
elasticfilesystem:DeleteFileSystem
elasticfilesystem:TagResource
elasticfilesystem:DeleteFileSystem
For the Security Group used by this EFS, configure an inbound rule to enable NFS port 2049
To upload configuration onto the EFS, we need an EC2 instance. First,
make sure the EFS is using the same Security Group as the EC2 instance
Add the inbound rules to allow NFS traffic within the same security group
SSH to an EC2 instance, create the mount-point /mnt/efs-xdi
ubuntu@ip-172-31-26-138:/mnt$ sudo mkdir efs-xdi ubuntu@ip-172-31-26-138:/sbin$ sudo apt install nfs-common Reading package lists... Done Building dependency tree... Done Reading state information... Done The following additional packages will be installed: keyutils libnfsidmap1 rpcbind Suggested packages: watchdog The following NEW packages will be installed: keyutils libnfsidmap1 nfs-common rpcbind 0 upgraded, 4 newly installed, 0 to remove and 80 not upgraded. Need to get 381 kB of archives. After this operation, 1447 kB of additional disk space will be used. Do you want to continue? [Y/n] Y Get:1 http://eu-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnfsidmap1 amd64 1:2.6.1-1ubun tu1.2 [42.9 kB] Get:2 http://eu-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 rpcbind amd64 1.2.6-2build1 [46.6 kB] Get:3 http://eu-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 keyutils amd64 1.6.1-2ubuntu3 [50.4 kB] Get:4 http://eu-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 nfs-common amd64 1:2.6.1-1ubuntu 1.2 [241 kB] Fetched 381 kB in 0s (10.7 MB/s) Selecting previously unselected package libnfsidmap1:amd64. (Reading database ... 146465 files and directories currently installed.) Preparing to unpack .../libnfsidmap1_1%3a2.6.1-1ubuntu1.2_amd64.deb ... Unpacking libnfsidmap1:amd64 (1:2.6.1-1ubuntu1.2) ... Selecting previously unselected package rpcbind. Preparing to unpack .../rpcbind_1.2.6-2build1_amd64.deb ... Unpacking rpcbind (1.2.6-2build1) ... Selecting previously unselected package keyutils. Preparing to unpack .../keyutils_1.6.1-2ubuntu3_amd64.deb ... Unpacking keyutils (1.6.1-2ubuntu3) ... Selecting previously unselected package nfs-common. Preparing to unpack .../nfs-common_1%3a2.6.1-1ubuntu1.2_amd64.deb ... Unpacking nfs-common (1:2.6.1-1ubuntu1.2) ... Setting up libnfsidmap1:amd64 (1:2.6.1-1ubuntu1.2) ... Setting up rpcbind (1.2.6-2build1) ... Created symlink /etc/systemd/system/multi-user.target.wants/rpcbind.service → /lib/systemd/system/rpcbind.serv ice. Created symlink /etc/systemd/system/sockets.target.wants/rpcbind.socket → /lib/systemd/system/rpcbind.socket. Setting up keyutils (1.6.1-2ubuntu3) ... Setting up nfs-common (1:2.6.1-1ubuntu1.2) ... Creating config file /etc/idmapd.conf with new version Creating config file /etc/nfs.conf with new version Adding system user `statd' (UID 117) ... Adding new user `statd' (UID 117) with group `nogroup' ... Not creating home directory `/var/lib/nfs'. Created symlink /etc/systemd/system/multi-user.target.wants/nfs-client.target → /lib/systemd/system/nfs-client .target. Created symlink /etc/systemd/system/remote-fs.target.wants/nfs-client.target → /lib/systemd/system/nfs-client. target. auth-rpcgss-module.service is a disabled or a static unit, not starting it. nfs-idmapd.service is a disabled or a static unit, not starting it. nfs-utils.service is a disabled or a static unit, not starting it. proc-fs-nfsd.mount is a disabled or a static unit, not starting it. rpc-gssd.service is a disabled or a static unit, not starting it. rpc-statd-notify.service is a disabled or a static unit, not starting it. rpc-statd.service is a disabled or a static unit, not starting it. rpc-svcgssd.service is a disabled or a static unit, not starting it. rpc_pipefs.target is a disabled or a static unit, not starting it. var-lib-nfs-rpc_pipefs.mount is a disabled or a static unit, not starting it. Processing triggers for man-db (2.10.2-1) ... Processing triggers for libc-bin (2.35-0ubuntu3.8) ... Scanning processes... Scanning candidates... Scanning linux images... Restarting services... systemctl restart irqbalance.service packagekit.service polkit.service Service restarts being deferred: /etc/needrestart/restart.d/dbus.service systemctl restart networkd-dispatcher.service systemctl restart unattended-upgrades.service No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host. ubuntu@ip-172-31-26-138:/sbin$ sudo mount -t nfs4 fs-0d61960da9d1cb572.efs.eu-west-2.amazonaws.com:/ /mnt/efs-xdi
Create the subdirectory structure under /mnt/efs-xdi
/mnt/efs-xdi/analytics/
/mnt/efs-xdi/analytics/analytics-webapp-home/
/mnt/efs-xdi/analytics/analytics-webapp-home/jdbc/
/mnt/efs-xdi/analytics/tomcat-additional-libraries/
/mnt/efs-xdi/analytics/tomcat-configuration-files/
Grant 777 to all the sub-folders
ubuntu@ip-172-31-26-138:/mnt/efs-xdi$ sudo chmod 777 *
Follow here: https://www.semarchy.com/tutorials-content/quick-install-docker-xdi/#3
Use FileZilla to upload runtime modules
For the tomcat-configuration-files, because ECS can only mount to directories but not files, so you have to:
1. Download all the files from the container path /usr/local/tomcat/conf to your local machine
Step 5. Deploy xDI runtime on ECS
Consult IT for the following privileges to create EFS:
ecs:ListClusters
Go to Amazon Elastic Container Service > Clusters.
Click on Create cluster. Add cluster name and Default namespace.
Keep other configuration as is or change as per your requirements. Click on Create.
It will take few minutes to create the cluster. Wait until it is successfully created.
Now we have to create a new Task Definition, click “Create new task definition”
Provide a task definition name and in Infrastructure requirements add the configuration as per you requirements e.g Launch Type, Operating System, CPU and Memory.
The Image URI can be found in hub.docker.com, search “semarchy/xdi-analytics”
Configure the Environment Variables.
Most of these key/value pairs are referencing the environment variables in docker-compose.yml in the xDI Tutorial. For the JDBC URL, it’s pointing to the RDS we created in Step 2.
XDI_ANALYTICS_DB_JDBC_DRIVER_CLASS_NAME=org.postgresql.Driver XDI_ANALYTICS_DB_JDBC_URL= jdbc:postgresql://instance-1.c3ik0bebqwwr.eu-west-2.rds.amazonaws.com:5432/postgres XDI_ANALYTICS_DB_JDBC_USERNAME=postgres XDI_ANALYTICS_DB_JDBC_PASSWORD=<your password> XDI_ANALYTICS_DEFAULT_ADMIN_USERNAME=semarchy XDI_ANALYTICS_DEFAULT_ADMIN_PASSWORD=<your password> XDI_ANALYTICS_DEFAULT_VIEWER_USERNAME=viewer XDI_ANALYTICS_DEFAULT_VIEWER_PASSWORD=<your password>
We need to create 3 volumes and 3 mount points for the xDI analytics container
Create the Task Definition
Now we have to create a service. Click on cluster which is just created and then go to services section. Click on Create.
Under environment section keep the configurations as is.
In the Deployment Configuration section, choose the Task Definition from the “Family” dropdown.
In the Networking section, ensure that the Public IP is turned ON. And add appropriate Security Group.
Keep other configuration as it is or update as per your requirements and click on Create.
The deployment will take some time.
Once the deployment is complete, you will see a service created.
Go to the Cluster > Services > Task > Logs, you should see the Server is started
Step 6. Create an inbound rule to open port 8080
Configure an inbound rule for port 8080, this is for the xDI analytics to be accessible by outsider
Step 7. Access the xDI Analytics
Go back to Cluster > Service > Tasks > Configuration and note down the public ip.
Access http://serverip:8080/semarchy-xdi-analytics/, login with semarchy/semarchy 
You should see the xDI Analytics main page
