DLI offers new features and new functionalities

11/24/2023

Following features and functionalities are available.

New functionalities:

Feature	Description
Load-based elastic resource pool, queue, and job scaling	SQL jobs can preempt resources based on loads (graceful scaling). SQL queues can preempt resources based on loads. In a SQL queue, the number of CUs applied for by the pod in the QuotaManager information can be customized.
Automatic scaling of general-purpose queues based on job priorities and resource loads	The job waiting list of the general queue is sorted by time and priority. The ClusterAgent component of the cluster can collect backpressure metrics of Flink jobs. The ClusterAgent can collect the number of Spark executors.
Permissions on DLI database tables can be granted by tag	Permissions on DLI database tables can be granted and authenticated by tag. APIs that support granting permissions by tag are added. On the DLI console, permissions on DLI tables can be granted by tag.
Fließende Skalierung von elastischen Ressourcenpools	Elastische DLI-Ressourcenpools können problemlos skaliert werden. Allgemeine Warteschlangen können in elastische Ressourcenpools migriert werden. Verschiedene Flavors in elastischen Ressourcenpools können zugewiesen und freigegeben werden.
DLI can connect to multiple catalogs	dli-catalog-client connects to DLI catalogs using independent configuration items. Spark images connect to DLI catalogs. The Spark Extension plugin connects to DLI catalogs using new configuration items.

New features:

On the Spark job management page, jobs can be filtered by date.
DLI permissions can be granted across tenants.
The Spark tenant plane can be interconnected with Quick BI through Livy.
Spark and Scala versions can be specified when interactive jobs are submitted using Livy.
DLI catalog metadata can be accessed through Spark Jar jobs.
The OLC feature allows DLI JobManager to dynamically control the flow at the microservice level.
DLI tenant clusters can be quickly switched with just one click. In some scenarios, jobs in existing clusters can be smoothly migrated to new clusters.
DLI SDKs and catalog clients can use temporary AKs/SKs and security tokens for authentication.
Flink1.12 OpenSource SQL jobs support datasource authentication.
DLI Spark jobs support 3.1*.
DLI overload detection can identify overload sources and top subscribers and APIs that cause overload.
DLI Catalog Client supports AK/SK authentication.
Pods in CCE clusters support balanced job scheduling.
DLI Spark datasource jobs support Spark 3.1*.

Further information can be found in Data Lake Insight HelpCenter.

*By default Spark 3.1 is disabled and customers need to open a support ticket to get enabled this version in their tenant.

Back to overview Release Notes

Do you have questions?

We answer your questions about testing, booking and use – free of charge and individually. Try it!
Hotline: 24 hours a day, 7 days a week
0800 3304477 from Germany / 00800 33044770 from abroad

Write an E-mail