Data Pipeline
Group of real-time stream processing jobs that processes the event stream of telemetry data generated from client apps and micro services. The telemetry data goes through a series of steps such as validation, de-duplication, transformation and denormalization of metadata. The transformed data is then stored in a consumable format that can be used for further analysis.
Analytics Data Pipeline

Key Features:

  1. 1.
    Lambda Architecture: A hybrid approach of using both batch-processing and stream-processing methods to process massive data sets.
  2. 2.
    Loose coupling: Data processing jobs are loosely coupled as they only communicate with a durable queue such as Apache Kafka.
  3. 3.
    Easy chaining: Data processing jobs can be chained easily by only configuring the input and output data sources they consume from. This allows easy introduction of new jobs required for processing custom workflows.
  4. 4.
    Data Sync points: The stream of data is synced to a configurable cloud storage which acts as a persistent data store with durability. The data sync points allow the capability to replay data from a specific stage in the pipeline.
  5. 5.
    Resiliency: The data pipeline guarantees AT LEAST ONCE processing semantics and ensures no data loss.
  6. 6.
    Monitoring: The data pipeline jobs has the capability to emit standard and custom metrics to monitor the health and also allows you to perform an audit of the system at various stages.
  7. 7.
    Auto-scaling: The data processing pipeline offers support for auto-scaling out of the box. This helps the pipeline to adapt to changes in the incoming data volume.
  8. 8.
    Real-time analytics: The data processing pipeline offers out of the box support with Apache Druid, an analytics data store design for fast slice-and-dice analytics.

Installation Configuration Reference:

Host or IP addresses of Kafka brokers for consumption of data
Host or IP addresses of Kafka brokers for publishing data
A boolean variable to enable checkpointing on cloud storage
Host or IP address of Redis cache used for metadata caching
Number of threads to consume data in parallel
Number of threads to process data in parallel
Directory path for JSON schema files to validate the telemetry data
Acceptable size of each event in bytes
1 Mb
The index of the device data store in Redis cache
The index of the user data store in Redis cache
The index of the content data store in Redis cache
The index of the dialcode data store in Redis cache


GitHub - project-sunbird/sunbird-data-pipeline: Repository for set of real-time streaming jobs to process and enrich the telemetry data generated by various user devices. The repository also consists of ansible provisioning playbooks to automate data pipeline related infrastructure provisioning and deployment playbooks to automate deployment of various components related to data analytics.
Data Pipeline source code
Copy link
Edit on GitHub