Distributed Message Service (DMS)

The Distributed Message Service (DMS) in the Open Telekom Cloud is based on Apache Foundation's Kafka solution and enables communication between different applications, for example in order to synchronize data and processes. The DMS works like a postal service within the cloud: senders and receivers do not need to be active at the same time. Applications that send messages in the form of JSON objects do not need to wait for acknowledgment of receipt from other applications – the DMS creates a processing queue and sends the message at a later point. This enables coordination between individual components of a cloud application, for instance in order to synchronize the status of data, making DMS the perfect solution for large, distributed application landscapes.

Reasons for DMS in the Open Telekom Cloud

Two stylised sheets of paper in front of server rack

High availability

DMS stores queued messages on different servers, and each message has multiple replicas, achieving high reliability and availability.

High reliability

Data replication and synchronous flushing to disks ensure up to 99.99999999% data reliability, while clustered and cross-AZ deployments ensure up to 99.95% service availability.

Stylised green speedometer needle in front of diagram

High performance

DMS queues can support millions of messages without compromising performance and can reach a throughput of 100,000 concurrent messages per second. Message delivery time is accurate to the millisecond.

Security

DMS interacts with Cloud Trace Service (CTS) to record and audit tenant management operations. Encrypted message storage protects them from unauthorized access.

Key Features of DMS

Laptop screen with cloud circled by interconnected envelopes

Message features

Message filtering
Consumers can use labels to filter the messages they want to retrieve from the chosen queue.
Message tracking
Messages that have already been retrieved can be retrieved again from the specified time or position.
Intentional delay delivery
Messages can be delivered after a specified delay.
Message broadcasting
The same message can be delivered to all consumers in a consumer group.
Message re-delivery
Messages that will not be immediately retrieved can be returned to the original queues. Consumers can retrieve them when they wish.

Blue circle with three arrows pointing to the right

Queue types and access protocols

DMS is compatible with native Kafka queues, so you can migrate your message services without any modifications.

Multiple queue types
Multi-protocol access

Weißes Klemmbrett mit dei Linien und HAufzählungshäkchen auf blauem Kreis

Queue features

DMS queues support dead letters queues, queue sharing and authorization.

Dead letter queue

Use Cases

DMS can be used in various fields, such as enterprise applications, online payments, telecommunications, e-commerce, logistics, marketing, social networking, Instant Messaging (IM), mobile gaming, video, Internet of Things (IoT), and Internet of Vehicles (IoV).

Distributed Message Service is useful in the following scenarios:

Service decoupling

DMS provides message notifications for supplementary services that are dependent on other systems. It decouples supplementary services from key services, allowing key services to proceed without waiting for other systems.

For example, the order processing (OP) system of an e-commerce website puts order information in DMS message queues during promotional activities. The inventory and delivery management systems will read the order information from the queues later.

Eventual consistency

In trading or payment systems, the transaction status (success or failure) must be consistent across subsystems or modules. Reliable message transmission is required between subsystems or modules to ensure service continuity. DMS provides highly reliable data transmission between subsystems or modules to ensure transaction statuses consistency at lower costs.

For example, if a bank customer buys wealth management products by using the deposit, the gains of wealth management products may not be included in the customer's deposit account. This is because a wealth management system usually processes transactions at the end of each day. To avoid reconciliation inconsistency between the banking system and the wealth management system, purchasing and payment data of wealth management products can be stored in DMS, ensuring the eventual consistency between the deposit balance and the wealth management gains.

Off-peak traffic control

In e-commerce systems or other large-scale websites, there is a processing capability gap between upstream and downstream systems. Traffic bursts from upstream systems with high processing capabilities may have a large impact on downstream systems with low processing capabilities.

For example, online sales promotions involve a huge amount of traffic flooding into e-commerce systems. DMS can help to buffer orders and other information, relieving pressure on downstream systems. It provides a three-day buffer for hundreds of millions of messages, allowing message consumption systems to process them during off-peak periods.

Log synchronization

Applications asynchronously send log messages to DMS over reliable transmission channels. Other components can read log messages from message queues for further analysis, either in real time or offline. In addition, DMS can collect required key log information to monitor applications. DMS's log synchronization process includes the following steps:

The log collection client collects log data from a user application service and writes the log data to message queues.
Message queues receive, store, and forward the log data.
Log processing applications subscribe to and consume log messages stored in message queues.

Structure and Function

DMS works in distributed and highly scalable computing clusters and comes with standardized RESTful APIs that are used to access the generated messages. The queued messages are stored on different partitions, and the cloud-native service can be used without any additional hardware or software resources. The DMS can be controlled via the web console, OTC Terraform Provider, or the OpenStack command-line interface. The generated message queues or Apache Kafka Premium clusters are processed in sequence in accordance with the first-in-first-out principle. In addition, the DMS supports dead letter queues (in which messages that could not be delivered are stored).

Step 1

A producer sends message M to a message queue. Message M is redundantly distributed in the queue.

Graphic that describes who a consumer receives a message from the queue

Step 2

A consumer receives message M from the queue.

While message M is being retrieved, it remains in the queue. It cannot be retrieved again within 30s since the start of retrieval. If message M fails to be acknowledged within this period, it can be retrieved again.

Diagram When message is confirmed, it can be retrieved to Queue

Step 3

Once message M is acknowledged, it can no longer be retrieved by consumers from the same consumer group.

However, it can still be retrieved by consumers from other consumer groups. It remains in the queue for at least 72 hours (unless the queue is deleted) and will be deleted after this period.

Distributed Message Services as a shared or dedicated solution

The Open Telekom Cloud offers message queues as part of a shared model, meaning that computing resources are shared between customers and costs are assigned based on the number of API calls and queues triggered.

Apache Kafka Premium, on the other hand, offers a dedicated, managed cluster solution with maximum availability at the touch of a button. Compared to message queues, Kafka Premium offers a guaranteed data throughput for messages, which is essential for enterprise solutions. Kafka Premium can be used both internally via the secure OTC network, and externally via the internet. The cluster costs depend on your desired cluster size and are incurred on an hourly basis, regardless of the number of API calls and queues. This makes it easy for you to calculate the cost of your business case.

Kafka Premium instances use physically isolated computing, storage, and bandwidth resources. You can customize partitions and replicas for Kafka topics in the instances, and configure the network bandwidth as required. The instances can be used right out of the box, taking off the deployment and O&M pressure for you so that you can focus on developing your services.

New Features

07/16/2024DMS Kafka creates new SSL certificatesView Details

08/19/2024DMS support Smart-ConnectView Details

10/02/2024DMS 3.x beta phase completedView Details

Don't want to miss any updates?Visit our portfolio roadmap and discover new services and updates.
Learn more

view all release notes

Find out more

Book now and claim starting credit of EUR 250* (code: 4UOTC250)

Book now

Take advantage of our consulting services!
Our experts will be happy to help you.

We will answer any questions you have regarding testing, booking and usage – free and tailored to your needs. Try it out today!

Hotline: 24 hours a day, seven days a week

0800 3304477from Germany

+800 33044770from abroad

Write an E-mail

* Voucher can be redeemed until December 31, 2024. Please contact us when using the voucher for booking. The discount is only valid for customers with a billing address in Germany and expires two months after conclusion of the contract. The credit is deducted according to the valid list prices as per the service description. Payment of the credit in cash is excluded.