Distributed Message Service (DMS)
The Distributed Message Service (DMS) in the Open Telekom Cloud is based on Apache Foundation's Kafka solution and enables communication between different applications, for example in order to synchronize data and processes. The DMS works like a postal service within the cloud: senders and receivers do not need to be active at the same time. Applications that send messages in the form of JSON objects do not need to wait for acknowledgment of receipt from other applications – the DMS creates a processing queue and sends the message at a later point. This enables coordination between individual components of a cloud application, for instance in order to synchronize the status of data, making DMS the perfect solution for large, distributed application landscapes.
DMS stores queued messages on different servers, and each message has multiple replicas, achieving high reliability and availability.
Data replication and synchronous flushing to disks ensure up to 99.99999999% data reliability, while clustered and cross-AZ deployments ensure up to 99.95% service availability.
DMS queues can support millions of messages without compromising performance and can reach a throughput of 100,000 concurrent messages per second. Message delivery time is accurate to the millisecond.
DMS interacts with Cloud Trace Service (CTS) to record and audit tenant management operations. Encrypted message storage protects them from unauthorized access.
DMS is compatible with native Kafka queues, so you can migrate your message services without any modifications.
DMS can be used in various fields, such as enterprise applications, online payments, telecommunications, e-commerce, logistics, marketing, social networking, Instant Messaging (IM), mobile gaming, video, Internet of Things (IoT), and Internet of Vehicles (IoV).
Distributed Message Service is useful in the following scenarios:
DMS provides message notifications for supplementary services that are dependent on other systems. It decouples supplementary services from key services, allowing key services to proceed without waiting for other systems.
For example, the order processing (OP) system of an e-commerce website puts order information in DMS message queues during promotional activities. The inventory and delivery management systems will read the order information from the queues later.
In trading or payment systems, the transaction status (success or failure) must be consistent across subsystems or modules. Reliable message transmission is required between subsystems or modules to ensure service continuity. DMS provides highly reliable data transmission between subsystems or modules to ensure transaction statuses consistency at lower costs.
For example, if a bank customer buys wealth management products by using the deposit, the gains of wealth management products may not be included in the customer's deposit account. This is because a wealth management system usually processes transactions at the end of each day. To avoid reconciliation inconsistency between the banking system and the wealth management system, purchasing and payment data of wealth management products can be stored in DMS, ensuring the eventual consistency between the deposit balance and the wealth management gains.
In e-commerce systems or other large-scale websites, there is a processing capability gap between upstream and downstream systems. Traffic bursts from upstream systems with high processing capabilities may have a large impact on downstream systems with low processing capabilities.
For example, online sales promotions involve a huge amount of traffic flooding into e-commerce systems. DMS can help to buffer orders and other information, relieving pressure on downstream systems. It provides a three-day buffer for hundreds of millions of messages, allowing message consumption systems to process them during off-peak periods.
Applications asynchronously send log messages to DMS over reliable transmission channels. Other components can read log messages from message queues for further analysis, either in real time or offline. In addition, DMS can collect required key log information to monitor applications. DMS's log synchronization process includes the following steps:
DMS works in distributed and highly scalable computing clusters and comes with standardized RESTful APIs that are used to access the generated messages. The queued messages are stored on different partitions, and the cloud-native service can be used without any additional hardware or software resources. The DMS can be controlled via the web console, OTC Terraform Provider, or the OpenStack command-line interface. The generated message queues or Apache Kafka Premium clusters are processed in sequence in accordance with the first-in-first-out principle. In addition, the DMS supports dead letter queues (in which messages that could not be delivered are stored).
Step 1
A producer sends message M to a message queue. Message M is redundantly distributed in the queue.
Step 2
A consumer receives message M from the queue.
While message M is being retrieved, it remains in the queue. It cannot be retrieved again within 30s since the start of retrieval. If message M fails to be acknowledged within this period, it can be retrieved again.
Step 3
Once message M is acknowledged, it can no longer be retrieved by consumers from the same consumer group.
However, it can still be retrieved by consumers from other consumer groups. It remains in the queue for at least 72 hours (unless the queue is deleted) and will be deleted after this period.
The Open Telekom Cloud offers message queues as part of a shared model, meaning that computing resources are shared between customers and costs are assigned based on the number of API calls and queues triggered.
Apache Kafka Premium, on the other hand, offers a dedicated, managed cluster solution with maximum availability at the touch of a button. Compared to message queues, Kafka Premium offers a guaranteed data throughput for messages, which is essential for enterprise solutions. Kafka Premium can be used both internally via the secure OTC network, and externally via the internet. The cluster costs depend on your desired cluster size and are incurred on an hourly basis, regardless of the number of API calls and queues. This makes it easy for you to calculate the cost of your business case.
Kafka Premium instances use physically isolated computing, storage, and bandwidth resources. You can customize partitions and replicas for Kafka topics in the instances, and configure the network bandwidth as required. The instances can be used right out of the box, taking off the deployment and O&M pressure for you so that you can focus on developing your services.
* Voucher is inwisselbaar tot 31.12.2024. Neem contact met ons op voor het bedrag van de voucher bij de boeking. Het kortingsvolume is alleen geldig voor klanten met een factuuradres in Duitsland en vervalt 2 maanden na het afsluiten van het contract. Het tegoed wordt verrekend met de geldige catalogusprijzen volgens de servicebeschrijving. Een uitbetaling is uitgesloten.
