Dataset Management Console
The Management Console is a sophisticated and intuitive platform designed to give you complete control over your datasets, connectors, and system operations. With a user-friendly interface, it streamlines the processes of dataset creation, configuration, and monitoring, while offering advanced features for data transformation, health monitoring, and alert management. Built-in metrics and real-time dashboards provide a comprehensive overview of your system's performance, ensuring efficient and seamless data workflows.
Key Features
Dataset Management: Create, update, and organize datasets with ease.
Connector Configuration: Integrate connectors effortlessly to enable automatic data synchronization between sources and destinations.
Data Transformation: Apply advanced transformations, including masking, encryption, and JSONAT, to ensure data privacy and compliance.
Dataset Joining: Link datasets with master datasets to enrich data analysis capabilities.
Duplicate Record Removal: Ensure data integrity by identifying and removing duplicate records.
System Monitoring: Access real-time dashboards and metrics for monitoring system performance and health.
Custom Alerts: Configure alerts and notifications to proactively manage dataset and connector issues.
Accessing the Management Console
To use the Management Console, navigate to the configured ingress URL or use port forwarding to access the application. Log in using the default credentials:
Username:
obsrv_adminPassword:
enDoPvTAxFSd
Upon logging in, you can explore the extensive features for managing data workflows, transforming data, and monitoring system health.
Key Workflows in the Management Console
Datasets Creation
The Management Console provides a streamlined workflow for creating datasets.
1. Connectors
Manage and configure connectors for seamless data integration.
Register connectors via API and view them in the console.
Explore default open-source connectors such as JDBC, Kafka, Object Store, and Knowlg.
Add custom connectors to meet specific requirements.
During dataset creation, select and configure the appropriate connector by filling in the required details.
2. Provide Basic Information
Enter the name of the dataset.
Upload a sample data file or schema file if available.
3. Ingestion Configuration
Review the schema configuration and make modifications if needed.
Resolve any data type conflicts automatically with recommendations provided by the console.
4. Processing Configuration
Choose the schema evaluation type.
Apply data transformations like masking and encryption for PII fields.
Automatically detect and address data type conflicts.
Enable settings to prevent duplicate record ingestion.
5. Storage Configuration
Configure storage type, such as lakehouse or real-time OLAP storage.
Set up storage configurations, including partition keys, timestamp keys, and primary keys.
Once all sections are configured, review the dataset settings and publish the dataset to make it live.
6. Dataset Management
View all datasets with their status, metrics, and health information on a consolidated list.
Click on a dataset to view its detailed configuration and insights.
Retire unused datasets to maintain an organized environment.
Alerts and Notifications
Create custom alerts to monitor specific events related to datasets and connectors.
Set up notifications for events, ensuring timely action when needed.
Configure alerts to notify you via channels like Slack, email, and Discord for real-time updates.
Dashboards
Monitor the overall system health with infra cluster and health dashboards.
Gain insights into ingestion, processing, storage, and querying workflows through dedicated dashboards.
The Management Console empowers you to manage your datasets, ensure data integrity, monitor system performance, and stay proactive with alerts and notifications—all in one streamlined interface.
Last updated
