Open Telekom Cloud: Top ratings in the price/performance benchmark

In this article you will read about,

• the results of the performance and price/performance benchmark between Open Telekom Cloud and US hyperscalers
• which tools and methods the analysts from Cloud Mercato used for the benchmarking
• why the analysts rate the CPU flavors from the Open Telekom Cloud as the best offers

Who offers the best performance and the best price/performance ratio in the infrastructure as a service (IaaS) segment? Who maps the workloads of their customers with the greatest possible efficiency? The independent analyst firm Cloud Mercato took a close look at cloud computing providers. The benchmark focused on the providers’ virtual machines (VM).

Like other hyperscalers, in 2020 the Open Telekom Cloud added new VM products to its range that are based on the current generation of processors (CPUs) and offer more performance. This gives customers extra computing power for their money. Among the new VM offerings, the flavors s3 and c4 succeed the previous generations s2 and c3. Unlike s2, c3 offers users dedicated resources without allowing virtual machines to take CPU capacities from the same pool.

In the benchmark, the Cloud Mercato experts compared various CPU flavors with 16 CPUs as well as block storage from the Open Telekom Cloud with leading hyperscalers such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). The following table lists the corresponding VMs of the providers.

For a virtual machine to work, block storage must be allocated. In order to enable a fair comparison, the required memory was measured with a rule of thumb: 50 GB per CPU. The following diagram shows which block storage was used per provider in the benchmark.

The performance of VMs depends on the computing power of the underlying CPU equipment. The metrics specified by CPU manufacturers such as MIPS (million instructions per second) or FLOPS (floating point operations per second) are not sufficient to measure the performance of a workload and compare the performance of individual VMs.

Cloud Mercato therefore used the Geekbench 5 benchmark suite to determine comparable performance values. This software runs workloads with integer, floating point, and cryptographic domains. Various methods were used to determine the performance, such as compression tests, machine learning, and computer vision.

The result is clear: The VMs from the Open Telekom Cloud outperform those of the other hyperscalers – the previous generation of s2 and c3 flavors already offer a higher performance. With the new c4 and s3 flavors, the Open Telekom Cloud offers twice the multi-threading performance of the comparable AWS VM. The following table illustrates the differences:

CPU steal, which can occasionally occur in cloud environments, did not play a role in the tests. The performance of a VM can sometimes drop considerably when VMs are not accurately separated from each other. This can lead to time delays in the processing of tasks. The hyperscalers examined by Cloud Mercato all showed only minimal CPU steal and correspondingly low time losses (see the following table).

For some use cases, the available RAM is even more important than the CPU performance. These include memory-intensive workloads with in-memory databases such as SAP HANA. To measure RAM performance, the analysts used Sysbench and tested the bandwidth between processor and memory with read and write modes. The best performer was the c4 flavor from the Open Telekom Cloud, which is equipped with dedicated CPUs.

Cloud providers generally have high-performance internal networks that guarantee a certain service level for all customers. To evaluate the maximum bandwidth of the respective hyperscalers, the analysts used iPerf3 and set up a test with two identical VMs from the same regional availability zone. Google achieved the best result, followed by c4 VMs from the Open Telekom Cloud.

Persistent block storage is primarily defined by its input/output per second (IOPS) capacities. This allows the maximum number of possible transactions to be determined. Like network bandwidth, storage is a shared resource that is capped by the providers. Each provider defines its own rules for IOPS. For the test, the analysts used Flexible I/O Tester (FIO). The result: AWS and Azure cap at around 3,000 IOPS, while GCP and the Open Telekom Cloud throttle bandwidth above 20,000 IOPS.

Performance and price are two sides of the same coin. Using these two criteria is the best way to assess how users benefit from a cloud provider's offer. The analysts therefore put the performance values determined for storage and computing in relation to the list prices of the providers. (The US dollar to euro exchange rate was: 1 US$ = 0.858 €).

To determine the price/performance ratio, the analysts used the multi-score values of the tested VMs and CPUs determined via Geekbench. Since the providers' billing models differ, Cloud Mercato compared both the hourly (on-demand usage) and the annual consumption (long-term usage).

While the previous generation of VMs from the Open Telekom Cloud was already on par with the best hyperscalers, the new generation clearly outperforms all the other VMs tested. It also offers an unrivalled price/performance ratio. By far the best offering that delivers customers the most value for money is s3, followed by c4 – regardless of whether customers opt for hourly or annual billing. 

To determine the storage ranking, the analysts compared the respective hourly prices for VMs and storage volumes, including the values for read and write access (see following chart). 

The results reflect the large differences in performance and the capping of bandwidths. Apart from Google, the Open Telekom Cloud shows the best price/performance ratio in the area of storage.

The independent tests and calculations carried out by Cloud Mercato show: The Open Telekom Cloud has once again significantly improved its performance with the introduction of the latest CPU generation. The s3 flavor outperforms the next best offering, AWS r5s, by 61 percent, the c4 flavor is even 65 percent above AWS r5s – thus the Open Telekom Cloud’s VMs show an outstanding performance. 

In terms of price/performance ratio, customers also get the best in class solution with the Open Telekom Cloud compared to the leading hyperscalers: s3 flavors offer a 40 percent better price/performance ratio and c4 flavors a 12 percent better price/performance ratio than the next best offering, AWS r5. These results confirm surveys conducted by Cloud Spectator's performance analysts. While the current generation of c4 VMs delivers outstanding performance, s3 is the best choice for cost-conscious customers seeking the best price/performance ratio.