AWS EBS performance

I have been meaning to document a couple key items to consider when looking at EBS volume performance. Here is a brief example from 4 volumes attached to an EC2 instance, in which the application in the OS was equally distributing the data across all four volumes. 

Each volume is 40 GB. To determine the IOPs, you simply multiply the volume size by 3, giving each volume 120 IOPs.

To calculate bandwidth, it is IOPs multiplied by I/O size. In the metrics you can see it’s averaging about 100 KB for each write to the volume. So 120 IOPs X 100KB = 12,000 KB  = 1200 MB/s of bandwidth available for each volume. From the graph, we can see the average is 2500 KBs which equals 250 MBs.

Also to note, latency and queue are fine. Queue length is basically how much work is waiting to be done, which you actually don’t want it to be zero as that would mean the volumes stand around with nothing to do. Too much work though and the latency goes up.

So, the per volume bandwidth average of 250 MBs is not maxing out the 1200MBs. However, The IOPs are being maxed out as you can see from the graph, where it keeps bursting above 120 and dropping back down.

Just some brief notes on AWS data flow to EBS volumes, and about optimizing AWS PIOPs (provisioned IOPs) EBS volumes.

Let me attempt to use the freeway analogy to define the terms. 

1. Application data = amount of traffic leaving their house driveway
2. Data block size = the width of each vehicle
3. Compute speed = how fast traffic gets from driveway to on ramp
4. Queue depth = how many vehicles are currently lined up waiting on the ramp to the freeway
5. Volume IOPs = how many vehicles can be on the freeway at once / quantity of freeway lanes
6. I/O size = the width of each lane
7. Volume bandwidth in MiB/s = total width of freeway
8. Latency = time for vehicles to reach destination after entering freeway (this would be a combincation of length of destination

To calculate max IOPs per volume, divide the volume throughput by the I/O size. For example [1], (16 KiB I/O) = 0.015625 MiB. Now take a volume throughput of 1,000 MiB/s and you are given 64,000 max IOPS for that volume.

Also [2], "to determine the optimal queue length for your workload on SSD-backed volumes, we recommend that you target a queue length of 1 for every 1000 IOPS available....for example, a volume with 3,000 provisioned IOPS should target a queue length of 3......Increasing the queue length is beneficial until you achieve the provisioned IOPS.

Average IO size calculation [2]

(Sum of VolumeWriteBytes over 5 minute period) / (Sum of VolumeWriteOps over 5 minute period)) / 1024

25,000,000,000 / 100,000 = 250,000 / 1024 = 244 Average Write Size KiB/op

IOPs calculation [2]

(over 5 minute period VolumeConsumedReadWriteOps) / period in seconds

1,500,000,000 / (60*5) = 5000 IOPs

Throughput

IOPs * IO size = throughput

5000 * 249856 bytes (244 KiB) = 1,249,280,000 bytes = 1,191 MiB/s

References:

[1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html

[2] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_cloudwatch_ebs.html

[3]https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/benchmark_procedures.html#UnderstandingQueueLength

[4] https://aws.amazon.com/premiumsupport/knowledge-center/ebs-calculate-optimal-io-size/

[5] https://aws.amazon.com/premiumsupport/knowledge-center/optimize-ebs-provisioned-iops/

[6] https://dzone.com/articles/understanding-amazon-ebs-io