Comment on page
Getting Started with Obsrv Deployment Using Helm

Sunbird Obsrv is a high-performance, cost-effective data stack with several components such as ingestion, querying, processing, backup, visualisation and monitoring. Obsrv 2.0 can be either installed using Terraform (Infrastructure as Code tool) or using Helm (Kubernets Package Manager).
Prerequisites
Obsrv runs completely on a Kubernetes cluster. A completely functional Kubernetes cluster is expected for a seamless Obsrv installation.
Hardware
Obsrv can support a volume of 5 million events per day with an average size of each event to be around 5 kb with the following specifications.
Kubernetes version of 1.25 or greater
Minimum of 16 cores of CPU
Minimum of 64 GB of RAM
PersistentVolume support in the Kubernetes cluster
Support for LoadBalancer service to externally expose some of the Obsrv services. Popular implementations such as MetalLB or Traefik can be used to expose the services using external IPs.
Software
Helm
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
Helm Dependencies
Run the following helm repo add command to download the required dependencies for running Obsrv.
# Example
helm repo add prometheus https://prometheus-community.github.io/helm-charts
monitoring - https://prometheus-community.github.io/helm-charts
redis - https://charts.bitnami.com/bitnami
loki (version - 4.8.0 ) - https://grafana.github.io/helm-charts
promtail (version - 6.9.3 ) - https://grafana.github.io/helm-charts
velero (version - 3.1.6 ) - https://vmware-tanzu.github.io/helm-charts
Source Code
Clone the obsrv-automation github repository. The required list of helm charts to deploy Obsrv will be under the terraform/modules/helm directory.
git clone https://github.com/Sunbird-Obsrv/obsrv-automation.git
cd obsrv-automation/terraform/modules/helm
Resources and Services
Please be advised that the list of resources will be completely different for different cloud service providers.
Common
The following list of buckets/containers need to be created for different services to store the data. This is applicable to Object Storage such as MinIO/Ceph as well.
flink-checkpoints
velero-backup
obsrv
AWS
1.IAM role with AmazonS3FullAccess policy. Services such as Api, Druid, Flink, Secor need to read and write access to S3 buckets.
2.Velero is a service which provides backups of the entire Obsrv cluster state through snapshots. Velero backup service needs a restricted user access to upload the snapshot state onto S3. The following IAM role policy needs to be attached to user created for velero backup. The access keys needs to be generated for the velero backup user as well.
{
"Statement": [
    {
    "Action": [
        "ec2:DescribeVolumes",
        "ec2:DescribeSnapshots",
        "ec2:CreateTags",
        "ec2:CreateVolume",
        "ec2:CreateSnapshot",
        "ec2:DeleteSnapshot"
    ],
    "Effect": "Allow",
    "Resource": "*"
    },
    {
    "Action": [
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:PutObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
    ],
    "Effect": "Allow",
    "Resource": [
        "arn:aws:s3:::<velero-s3-container-name>/*"
    ]
    },
    {
    "Action": [
        "s3:ListBucket"
    ],
    "Effect": "Allow",
    "Resource": [
        "arn:aws:s3:::<velero-s3-container-name>"
    ]
    }
],
"Version": "2012-10-17"
}
3.Serive Accounts: Service accounts enable access of the S3 object storage without the need for the access keys. If you prefer to use keys instead, you can skip the creation of service accounts. The list of service accounts needed
Dataset API with the name dataset-api-sa
Druid with the name druid-raw-sa
Flink with the name flink-sa
Secor with the name secor-sa
Deployment Instructions
Helm package manager provides an easy way to install specific components using a generic command. Configurations can be overriden by updating the values.yaml file in the respective Helm charts.
helm upgrade --install --atomic <release_name> <chart_name> -n <namespace> -f <path/values.yaml> --create-namespace --debug
Prerequisites
Kubernetes Cluster Access
Helm package manager needs access to the Kubernetes cluster. The path to the KUBECONFIG file needs to be exported as an environment variable, either in the current shell or in environment configuration files such as .bashrc
export KUBECONFIG=<path_to_kubeconfig file>
Postgres
Postgres is a RDBMS database which is used as the metadata store
helm upgrade --install --atomic postgresql postgresql/postgresql-helm-chart -n postgresql --create-namespace --debug
Redis
Redis is an in-memory key-value store primarily used as a distributed cache
helm upgrade --install --atomic obsrv-redis redis/redis -n redis -f redis/values.yaml --create-namespace --debug
Prometheus
Prometheus is a monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach.
helm upgrade --install --atomic monitoring monitoring/kube-prometheus-stack -n monitoring -f monitoring/values.yaml --create-namespace --debug
Kafka
Apache Kafka is a distributed event store and stream-processing platform.
helm upgrade --install --atomic kafka kafka/kafka-helm-chart -n kafka --create-namespace --debug
The following list of kafka topics are created by default. If you would like to add more topics to the list, you can do so by adding it to provisioning.topics configuration in the values.yaml file.
dev.ingest
masterdata.ingest
Druid
Druid is a high performance, real-time analytics database that delivers sub-second queries on streaming and batch data at scale
Druid CRD
helm upgrade --install --atomic druid-operator druid_operator/druid-operator-helm-chart -n druid-raw --create-namespace --debug
Druid Cluster
Druid requires the following set of configurations to be provided for specific storage systems such as AWS S3, Azure Blob Storage, GCP Storage or MinIO/Ceph
AWS
druid_deepstorage_type: s3
druid.extensions.loadList: ["druid-s3-extensions"]
# S3 Access keys
s3_access_key: ""
s3_secret_key: ""
s3_bucket: "obsrv"
MinIO/Ceph
druid_deepstorage_type: s3
druid.extensions.loadList: ["druid-s3-extensions"]
# S3 Access keys
s3_access_key: ""
s3_secret_key: ""
# Use the ClusterIP of the MinIO service instead of the Kubernetes service name
# We have noticed that the service names don't resolve properly
druid_s3_endpoint_url: http://172.20.126.232:9000/
s3_bucket: "obsrv"
druid_s3_endpoint_signingRegion: "us-east-2"
Azure
druid.extensions.loadList: ["druid-azure-extensions"]
druid_deepstorage_type: azure
azure_storage_account_name: ""
azure_storage_account_key: ""
azure_storage_container: "obsrv"
GCP
druid.extensions.loadList: ["druid-google-extensions"]
druid_deepstorage_type: google
# Google cloud credentials json file where the access_token and credentials are stored.
google_application_credentials: 
gcs_bucket: "obsrv"
​Hadoop​
druid_deepstorage_type: "hdfs"
# Include the "druid-hdfs-storage" extension as part of the existing the extensions list
druid.extensions.loadList: ["druid-hdfs-storage"]
druid.indexer.logs.directory: "/druid/indexing-logs"
druid.storage.storageDirectory: "/druid/segments"
​
helm upgrade --install --atomic druid-raw druid_raw_cluster/druid-raw-cluster-helm-chart -n druid-raw --create-namespace --debug
API
This service provides metadata APIs related to various resources such as datasets/datasources in Obsrv. The following configurations need to be specified in the values.yaml file.
AWS
exhaust_service.CONTAINER: obsrv
exhaust_service.CONTAINER_STORAGE_PROVIDER: aws
exhaust_service.CONTAINER_STORAGE_REGION: us-east-2
helm upgrade --install --atomic dataset-api dataset_api/dataset-api-helm-chart -n dataset-api --create-namespace --debug 
Flink Streaming Jobs
Flink jobs are used to process and enrich the data ingested into Obsrv in near-realtime.
Configuration Overrides
AWS
checkpoint_store_type: s3
# S3 Access keys
s3_access_key: ""
s3_secret_key: ""
# Under base_config in the values.yaml
base.url: s3://flink-checkpoints
MinIO/Ceph
checkpoint_store_type: s3
# S3 Access keys
s3_access_key: ""
s3_secret_key: ""
# Use the ClusterIP of the MinIO service instead of the Kubernetes service name
# We have noticed that the service names don't resolve properly
s3_endpoint:  http://172.20.126.232:9000/
# Under base_config in the values.yaml
base.url: s3://flink-checkpoints
Azure
checkpoint_store_type: azure
azure_account: ""
azure_secret: ""
# Under base_config in the values.yaml
base.url: blob://flink-bucket
GCP
checkpoint_store_type: gcp
# Google cloud credentials json file where the access_token and credentials are stored.
google_application_credentials: ""
base.url: blob://flink-bucket
​Hadoop​
checkpoint_store_type: hdfs
# Under base_config in the values.yaml
base.url: hdfs:///flink-bucket/
Flink Merged Pipeline Job
helm upgrade --install --atomic merged-pipeline flink/flink-helm-chart -n flink --set image.registry=sunbird --set image.repository=sb-obsrv-merged-pipeline --create-namespace --debug
Flink Master Data Processor Job
helm upgrade --install --atomic master-data-processor flink/flink-helm-chart -n flink --set image.registry=sunbird --set image.repository=sb-obsrv-master-data-processor --create-namespace --debug
Backup Processes
Secor
Configuration Overrides
AWS
# S3 upload manager which is responsible to upload backup to deepstorage.
upload_manager: com.pinterest.secor.uploader.S3UploadManager
cloud_store_provider: S3
aws_access_key: ""
aws_secret_key: ""
aws_region: us-east-2
MinIO/Ceph
# S3 upload manager which is responsible to upload backup to deepstorage.
upload_manager: com.pinterest.secor.uploader.S3UploadManager
cloud_store_provider: S3
aws_access_key: ""
aws_secret_key: ""
# Use the ClusterIP of the MinIO service instead of the Kubernetes service name
# We have noticed that the service names don't resolve properly
aws_endpoint: http://172.20.126.232:9000/
aws_region: us-east-2
Azure
upload_manager: com.pinterest.secor.uploader.AzureUploadManager
cloud_store_provider: Azure
azure_account_name: ""
azure_account_key: ""
GCP
upload_manager: com.pinterest.secor.uploader.GsUploadManager
# Credentials path where access token and secrets are stored.
gs_credentials_path: google_app_credentials.json
Hadoop
upload_manager: com.pinterest.secor.uploader.HadoopS3UploadManager
# Ensure the secor.s3.filesystem property is updated with the `hdfs` value
cloud_store_provider=hdfs
cloud_storage_bucket=namenode-host:8020/dir_path
# For More details please check here - https://github.com/pinterest/secor/issues/129
Secor backups are performed from various kafka topics which are part of the data processing pipeline. The following list of backup names need to be replaced in the below mentioned command.
List of backup names
ingest-backup
extractor-duplicate-backup
extractor-failed-backup
raw-backup
failed-backup
invalid-backup
unique-backup
duplicate-backup
denorm-backup
denorm-failed-backup
system-stats
system-events
helm upgrade --install --atomic <backup_name> secor/secor-helm-chart -n secor --create-namespace
Velero
helm upgrade --install --atomic velero velero/velero -n velero -f velero/values.yaml --create-namespace --debug --version 3.1.6
Monitoring Services
Monitoring Dashboards
helm upgrade --install --atomic grafana-configs grafana_configs/grafana-configs-helm-chart -n monitoring --create-namespace --debug
Monitoring Alert Rules
helm upgrade --install --atomic alertrules alert_rules/alert-rules-helm-chart -n monitoring --create-namespace --debug
Druid Exporter
helm upgrade --install --atomic druid-exporter druid_exporter/druid-exporter-helm-chart -n druid-raw --create-namespace --debug
Kafka Exporter
helm upgrade --install --atomic kafka-exporter kafka_exporter/kafka-exporter-helm-chart -n kafka --create-namespace --debug
Postgres Exporter
helm upgrade --install --atomic postgresql-exporter postgresql_exporter/postgresql-exporter-helm-chart -n postgresql --create-namespace --debug
Loki
helm upgrade --install --atomic loki loki/loki -n loki -f loki/values.yaml --create-namespace --debug --version 4.8.0
Promtail
helm upgrade --install --atomic promtail promtail/promtail -n loki -f promtail/values.yaml --create-namespace --debug --version 6.9.3
Ingestion
This helm chart is used to submit the default ingestion tasks required for the system statistics events
helm upgrade --install --atomic submit-ingestion submit_ingestion/submit-ingestion-helm-chart -n submit-ingestion --create-namespace --debug 
Visualization
Superset
helm upgrade --install --atomic superset superset/superset-helm-chart -n superset --create-namespace --debug
Loadbalancers
Following is a list of services which are exposed as a LoadBalancer service.
Component
Service Name
Description
Dataset API
service/dataset-api-service
Meta APIs
Superset
service/superset
Data Visualization Tool
Post Deployment
Please find documentation related to various application level functionalities in Obsrv below
​Create Datasets​
​Data Ingestion​
​Data Querying​
Component	Service Name	Description
Dataset API	service/dataset-api-service	Meta APIs
Superset	service/superset	Data Visualization Tool
Use - Previous
Obsrv 2.0 Installation Guide
Next - Use
System Requirements
Last modified 3mo ago