AWS Installation Guide
This guide provides detailed, step-by-step instructions for installing and configuring Obsrv on AWS, utilizing Terraform, Terragrunt, and Helm.
Infrastructure Requirements
1. System Specifications
CPU Requirements:
Minimum: 19 CPUs.
Optimal Configuration: 5 nodes with 4 cores each, totaling 80GB of RAM.
The installation package includes both lakehouse and real-time OLAP storage by default. If the lakehouse component is not required, only the real-time OLAP storage can be installed, reducing requirements to 16 CPUs and 64GB of RAM.
In this case, we recommend using 2 nodes with 8 cores each, totaling 64GB of RAM, by selecting the t2.2xlarge
AWS instance type.
Availability Zones: All instances should be within the same availability zone to minimize cross-zone data transfer costs. The Obsrv installer will automatically create the EKS (Elastic Kubernetes Service) cluster for you.
2. Networking Setup
CIDR Block: Use a
/23
CIDR range (512 IPs) for your environment.Example: A VPC with
10.0.0.0/23
provides IPs from10.0.0.0
to10.0.1.255
.
Subnets: Ensure subnets are created in all availability zones within your AWS region.
Prerequisites
Before beginning the installation, make sure the following tools are installed on your Linux-based system:
Tool
Version
Installation Command
Official Documentation
Terraform
1.5.x or earlier
curl "https://releases.hashicorp.com/terraform/1.5.2/terraform_1.5.2_linux_amd64.zip" -o terraform.zip && unzip terraform.zip && sudo mv terraform /usr/local/bin/ && rm terraform.zip
Terragrunt
0.48 or later
curl -OL https://github.com/gruntwork-io/terragrunt/releases/download/v0.49.0/terragrunt_linux_amd64 && sudo mv terragrunt_linux_amd64 /usr/local/bin/terragrunt && sudo chmod +x /usr/local/bin/terragrunt
Helm
3.10.2 or later
curl https://get.helm.sh/helm-v3.10.2-linux-amd64.tar.gz -o helm.tar.gz && tar -zxvf helm.tar.gz && sudo mv linux-amd64/helm /usr/local/bin/
AWS CLI
2.10 or later
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && unzip awscliv2.zip && sudo ./aws/install
Installation Steps
1. Clone the Obsrv Repository
Start by cloning the Obsrv automation repository and checkout to either latest release tag or master.
2. Configure the Kubernetes Cluster
By executing the following commands which will bring up the kubernetes cluster in the AWS environment of configured region.
Navigate to the Configuration Directory:
Update Configuration Files:
Open
cluster_overides.tf
and modify the configuration values to match your environment.
Configure S3 for Cluster State:
Open
obsrv.conf
and update your AWS credentials and bucket names.
3. Run the Installation Script
Make the Script Executable:
Run the Installation:
To start the installation, run the script:
If you want the installer to automatically handle dependencies, set
install_dependencies=true
.
4. Verify the Cluster
Once the installation completes, verify that your Kubernetes cluster is up and running:
This should show the nodes in your Kubernetes cluster.
Helm Chart Configuration
1. Navigate to the Helm Chart Directory
2. Update AWS Cloud Configuration
Modify global-cloud-values-aws.yaml
with the appropriate values for your environment:
3. Update Domain Configuration
In global-values.yaml
, replace <domain>
with your actual domain or Elastic IP:
4. Update Private and Public Keys
Follow these steps to generate and configure the private and public keys for the web console and dataset API:
Step 1: Generate a Private Key
Run the following command to generate a private key for the web console:
Open the generated
private_key.pem
file.Copy its contents and update the
USER_TOKEN_PRIVATE_KEY
field in the following file:obsrv-automation/helmcharts/services/web-console/values.yaml
Example:
Step 2: Generate a Public Key
Using the private key generated above, create a public key with the following command:
Open the generated
public_key.pem
file.Copy its contents and update the
user_token_public_key
field in the following file:obsrv-automation/helmcharts/services/dataset-api/values.yaml
Example:
5. Install Obsrv
Make the script executable and set the environment variables and run the installation:
Post-Installation Verification
After completing the installation, follow these steps to verify that all components are running correctly:
1. Check Kubernetes Components
Verify all pods are running:
All pods should be in
Running
state. Common namespaces to check:flink
: Core Pipelinemonitoring
: Monitoring stackdataset-api
: Dataset APIsweb-console
: Dataset Management console
Check Services:
Verify that essential services have external IPs assigned, particularly the Kong service.
If any component fails these checks, refer to the component-specific logs:
Upgrade Steps
Pull the Latest Code:
Update Configurations: Review and update configuration values as needed.
Run Terraform for Upgrade:
Upgrade with Updated Cloud Values:
By following these steps, you will ensure a successful installation and configuration of Obsrv on AWS.
Last updated