Sunbird Obsrv
  • Introduction
    • The Value of Data
    • Data Value Chain
    • Challenges
    • The Solution: Obsrv
  • Core Concepts
    • Obsrv Overview
    • Key Capabilities
    • Datasets
    • Connectors
    • High Level Architecture
    • Tech Stack
    • Monitoring
  • Explore
    • Roadmap
    • Case Studies
      • Agri Climate Advisory
      • Learning Analytics at Population Scale
      • IOT Observations Infra
      • Data Driven Features in Learning Platform
      • Network Observability
      • Fraud Detection
    • Performance Benchmarks
  • Guides
    • Installation
      • AWS Installation Guide
      • Azure Installation Guide
      • GCP Installation Guide
      • OCI Installation Guide
      • Data Center Installation Guide
    • Dataset Management APIs
    • Dataset Management Console
    • Connector APIs
    • Data In & Out APIs
    • Alerts and Notification Channels APIs
    • Developer Guide
    • Example Datasets
    • Connectors Developer Guide
      • SDK Assumptions
      • Required Files
        • metadata.json
        • ui-config.json
        • metrics.yaml
        • alerts.yaml
      • Obsrv Base Setup
      • Dev Requirements
      • Interfaces
        • Stream Interfaces
        • Batch Interfaces
      • Classes
        • ConnectorContext Class
        • ConnectorStats Class
        • ConnectorState Class
        • ErrorData Class
        • MetricData Class
      • Verifying
      • Packaging Guide
      • Reference Implementations
    • Coming Soon!
  • Community
  • Previous Versions
    • SB-5.0 Version
      • Overview
      • USE
        • Release Notes
          • Obsrv 2.0-Beta
          • Obsrv 2.1.0
          • Obsrv 2.2.0
          • Obsrv 2.0.0-GA
          • Obsrv 5.3.0-GA
          • Release V 5.1.0
          • Release V 5.1.2
          • Release V 5.1.3
          • Release V 5.0.0
          • Release V 4.10.0
        • Installation Guide
        • Obsrv 2.0 Installation Guide
          • Getting Started with Obsrv Deployment Using Helm
        • System Requirements
      • LEARN
        • Functional Capabilities
        • Dependencies
        • Product Roadmap
        • Product & Developer Guide
          • Telemetry Service
          • Data Pipeline
          • Data Service
          • Data Product
            • On Demand Druid Exhaust Job
              • Component Diagram
              • ML CSV Reports
              • Folder Struture
          • Report Service
          • Report Configurator
          • Summarisers
      • ENGAGE
        • Discuss
        • Contribute to Obsrv
      • Raise an Issue
  • Release Notes
    • Obsrv 1.1.0 Beta Release
    • Obsrv 1.2.0-RC Release
Powered by GitBook
On this page
  • Introduction
  • What does connectors do?
  • Available Connectors

Was this helpful?

Edit on GitHub
  1. Core Concepts

Connectors

Another key construct of Obsrv which allows it to connect to wide variety of data sources & formats

Introduction

Connectors are one of the core construct of Obsrv in addition to datasets. While datasets provide a construct to manage your data processing , storage and querying layers, connectors provide a construct to manage your data ingress and outgress.

What does connectors do?

While Obsrv automatically enables data "push" into the platform there are many use-cases where one has to pull the data from sources. Trying to write custom scripts/jobs to read from many sources and pushing into Obsrv is a complex problem and is prone to quality and reliability issues. "Connectors" as a concept is used to solve the problem of pulling data efficiently and reliably. Connectors solve quite a few design problems of new age data platforms:

  1. Decoupled: Enables Obsrv to be decoupled with the data ingestion from source systems. One source system cannot effect the data flowing in from another source system

  2. Source Data Management: Data quality, volume and lineage can be segregated by source and can be managed independently.

  3. Efficient Scalability: Only the specific connector that processes large volume needs to be scaled independently rather than the entire data platform

  4. Pluggability: Just swap out the out of the box connector with your own custom connector or a market-place connector without any impact to the data platform

  5. Extensability: Future proof where extensibility is guaranteed by design. Your data has a unique source - build your own connector using the connector framework. Connectors not only pull data from sources but can also sync data to choice of your destination (reverse ETL)

Connectors fall into 4 broad categories:

  1. Database: Any connector pulling data from a OLTP or NoSQL database.

  2. Stream/Event: Any connector pulling data from streams or event driven systems (like Kafka, RabbitMQ etc). The connectors of this type can process data in real-time.

  3. File: Any connector pulling data from file systems or object stores like S3, Azure Blob, GCS, MinIO etc

  4. Application: Custom application specific connector. For ex: A SAP connector to pull data from SAP system.

Available Connectors

Following are the connectors available out of the box. Connector with blue are available and orange are under incubation and will be available in future releases

Connector Type
Connector Source

Database

Postgres, MySQL, DB2, MariaDB

Oracle, SQL Server, Amazon RDS, Azure SQL, Google Cloud SQL

MongoDB, Cassandra, ElasticSearch

Stream

Kafka, Postgresql Debezium, MySQL Debezium, Neo4j Transactions, DB2 Debezium, Oracle Debezium, SQL Server Debezium, MongoDB Debezium, Cassandra Debezium

File

AWS S3, Azure Blob Storage, MinIO, Google Cloud Storage

PreviousDatasetsNextHigh Level Architecture

Last updated 1 year ago

Was this helpful?