Open Telekom Cloud for Business Customers

Data Lake Insight (DLI)

Data Lake Insight (DLI) is a serverless big data query and analysis service fully compatible with Apache Spark and Apache Flink ecosystems. DLI supports standard SQL and is compatible with Spark and Flink SQL. It also supports multiple access modes and is compatible with mainstream data formats. DLI supports SQL statements and Spark applications for heterogeneous data sources, including CloudTable, RDS, DWS, CSS, OBS, custom databases on ECSs, and offline databases.

Spark is a unified analysis engine that is ideal for large-scale data processing. It focuses on query, compute, and analysis. DLI optimizes performance and reconstructs services based on open-source Spark. It is compatible with the Apache Spark ecosystem and interfaces and improves performance by 2.5x when compared with open-source Spark. That way, DLI enables you to perform query and analysis of EB's of data within hours.

Flink is a distributed compute engine that is ideal for batch processing, i.e., for processing static data sets and historical data sets. You can also use it for stream processing, i.e., processing real-time data streams and generating data results in real time. DLI enhances features and security based on the open-source Flink and provides the Stream SQL feature required for data processing.

Woman working in front of several screens in a data center

Reasons for DLI in the Open Telekom Cloud

Icon with server and hand

Ease of use

DLI lets you easily explore entire terabytes in your data lake in seconds using standard SQLs with zero O&M burden.

Icon with pie chart and speech bubble with bullet list

One-stop analysis

Fully compatible with Apache Spark and Flink; stream & batch processing and interactive analysis in one place.

Icon with gear and arrow symbol for scalability

Scalable resources

On-demand, shared access to pooled resources, flexible scaling based on preset priorities.

Icon with cloud and server

Cross-source connection

Easy cross-source data access for collaborative analysis with DLI datasource connections, no need for data migration.


Key Features of DLI

Woman with pen in hand works in front of several screens displaying different data

Full SQL compatibility

You do not need a background in big data to conduct big data analyses. You only need to know SQL, and you are good to go. The SQL syntax is fully compatible with the standard ANSI SQL 2003.

 
Icon: Puzzle

Serverless Spark/Flink

Seamlessly migrate your offline applications to the cloud with serverless technology. DLI is fully compatible with Apache Spark, Apache Flink, and Presto ecosystems and APIs.

Icon: Diagram

Cross-source analysis

Analyze your data across databases. No migration required. A unified view of your data gives you a comprehensive understanding of your data and helps you innovate faster. There are no restrictions on data formats, cloud data sources, or whether the database is created online or off.

Icon: Hierarchical structure

Enterprise multi-tenant

Manage compute or resource related permissions by project or by user. Enjoy fine-grained control that makes it easy to maintain data independence for separate tasks.

Icon: Cloud-Datenbank

Storage-compute decoupling

DLI decouples storage from computing so that you can use lower costs while improving resource utilization.

Icon: Time lapse

O&M-free and high availability

DLI frees you from the hassle of complicated O&M and upgrade operations while you enjoy high data availability with dual-AZ deployment.


Identity and Access Management

DLI has a comprehensive permission control mechanism and supports fine-grained authentication through Identity and Access Management (IAM). You can create policies in IAM to manage DLI permissions. You can use both the DLI's permission control mechanism and the IAM service for permission management.

Application Scenarios of IAM Authentication

When using DLI on the cloud, enterprise users need to manage DLI resources (queues) used by employees in different departments, including creating, deleting, using, and isolating resources. In addition, data of different departments needs to be managed, including data isolation and sharing.

DLI uses IAM for refined enterprise-level multi-tenant management. IAM provides identity authentication, permissions management, and access control, helping you securely access to your cloud resources.

With IAM, you can use your cloud account to create IAM users for your employees and assign permissions to the users to control their access to specific resource types. For example, some software developers in your enterprise may need to use DLI resources but should not delete them or perform any high-risk operations. To guarantee this result, you can create IAM users for the software developers and grant them only the permissions required for using DLI resources.

 

DLI system permissions

Roles: A type of coarse-grained authorization mechanism that defines permissions related to user responsibilities. This mechanism provides only a limited number of service-level roles for authorization. When using roles to grant permissions, you need to also assign other roles on which the permissions depend to take effect. However, roles are not an ideal choice for fine-grained authorization and secure access control.

Policies: A type of fine-grained authorization mechanism that defines permissions required to perform operations on specific cloud resources under certain conditions. This mechanism allows for more flexible policy-based authorization, meeting requirements for secure access control. For example, you can grant DLI users only the permissions for managing a certain type of ECSs.
 

Role/Policy Name

Description

Category

DLI FullAccess

All permissions for DLI

System defined policy

DLI ReadOnlyAccess

DLI read permissions

System defined policy

Tenant Administrator

Tenant administrator

  • Administer permissions for managing and accessing all cloud services. After a database or a queue is created, the user can use the Access Control List (ACL) to assign rights to other users.
  • Scope: project-level service

System defined role

DLI Service Admin

DLI administrator

  • Administer permissions for managing and accessing the queues and data of DLI. After a database or a queue is created, the user can use the Access Control List (ACL) to assign rights to other users.
  • Scope: project-level service

System defined role

 

DLI service permissions

Permission Type

Subtype

SQL Syntax

Queue Permissions

Queue management permissions

Queue usage permission

None

Data Permissions

Database permissions

Table permissions

Column permissions

For details, see SQL Syntax of Batch Jobs > Data Permissions Management > Data Permissions List in the Data Lake Insight SQL Syntax Reference.

Job Permissions

Flink job permissions

None

Package Permissions

Package group permissions

Packe permissions

None

Datasource Connection Permissions

Datasource connection permissions

None

 

For details, see Permission-related APIs > Granting Users with the Data Usage Permission in the Data Lake Insight API Reference.

 

DLI console features

SQL Editor

Job Management

Resources in Queue Management

Data Management

Job Template

Datasource Connections

Global Configuration

 

Application scenarios

Analytics

E-Commerce

Large Enterprises

Genetics

Finance

Geography

 
 
  • Communities

    The Open Telekom Cloud Community

    This is where users, developers and product owners meet to help each other, share knowledge and discuss.

    Discover now

  • Telefon

    Free expert hotline

    Our certified cloud experts provide you with personal service free of charge.

     0800 3304477 (from Germany)

     
    +800 33044770 (from abroad)

     
    24 hours a day, seven days a week

  • E-Mail

    Our customer service is available free of charge via E-Mail

    Write an E-Mail