Taking Control of Your AWS Bill

We learn from the RightScale (now Flexera) 2019 State of the Cloud Report that cloud adoption has become pretty much universal, and public cloud adoption is growing significantly. Some of the key report takeaways are:

• Organizations of all sizes are running nearly 80% of their workloads in the cloud, with SMBs running 43% of their workloads in the public cloud (versus private cloud).
• Companies will spend 24% more on the public cloud in 2019 than in 2018, with public cloud spend growing three times faster than private cloud spend.
• One out of every two enterprises spends more than $1.2 million annually on public cloud, with 13% spending $12+ million each year.

The same report also highlights that the top cloud challenge for the third year in a row is managing cloud spend, while optimizing cloud costs is the #1 priority of central IT. The respondents themselves estimate that 27% of their cloud spend is wasted, while Flexera has measured actual waste among their customers at 35%. But managing cloud budgets effectively requires near real-time visibility into actual costs—often across multiple accounts, numerous projects, and a multitude of self-provisioning users.

This is a two-part blog post series. In this first post, we explore the Amazon Web Services (AWS) suite of cost. Later next week, we’ll explore the usage of management tools that can help you take control of your AWS bill.

The Monthly Bill 101

As of early 2018, the AWS Cost and Usage report has become the default AWS tool for tracking and billing cloud usage. Downloaded from the Amazon Simple Storage Service (Amazon S3) console, this CSV report contains a line item for each usage of an AWS resource or service. Three times a day, a cumulative update is delivered to a user-specified Amazon S3 bucket. At the end of the month, the report is finalized into an invoice.

The AWS Cost and Usage report is free of charge (up to five reports per AWS account). With line items aggregated on either an hourly or daily basis (as specified by the account owner), a report can contain literally millions of lines. In this section, we review how the report is structured and the key cost elements that it tracks.

Identifying the Product

A number of fields in the report are dedicated to precisely identifying the AWS product for which the charge has been incurred. These fields include the instance type (such as Amazon EC2), the instance type family (such as t2 or m4), the operating system, the product family (such as compute or storage), and the geographic AWS Region in which the usage took place. In addition, a unique SKU is generated for each product by combining three line item details (see Line Items: Defining Billing and Usage Parameters below): ProductCode, UsageType, and Operation.

Line Items: Defining Billing and Usage Parameters

The line item fields provide detailed information on the type of operation, its relevant costs, and various high-level billing parameters, such as the legal billing entity, the billing currency, the taxes to be applied, and the type of charge (which could also be a refund or a credit).

For example, the three line item fields that are used to generate the product SKU (see Identifying the Product above) are:

• ProductCode, such as aws ec2 for Amazon Elastic Compute Cloud (Amazon EC2).
• UsageType: Region + type of usage + instance. For example, USW2-BoxUsage:m2.2xlarge refers to the total machine resource-hours consumed by a request (QUERY, GET, PUT, etc.) in the US West (Oregon) Region on a high-memory, double extra large instance. 
• Operation, such as RunInstances for an EC2 instance.

For on-demand resources (versus Reserved Instances, see Tracking Reserved Instances below), usage duration metrics are provided by three fields:

• UsageStartDate and UsageEndDate: The date and time in UTC format when the operation was initiated and terminated.
• UsageAmount: Calculated using the first two fields, but this also takes into account AWS minimum usage thresholds. For example, for an Amazon EC2 instance charged on an hourly basis, the UsageAmount will be a minimum of 60 minutes, even if there was only 10 minutes of actual usage per the start and end times.

The cost of on-demand usage for a single account is shown in the UnblendedCost field, which is calculated by multiplying the UnblendedRate (the current AWS pricing rate for the resource or service) by the UsageAmount. See Consolidated Billing below for an explanation of blended costs across multiple accounts when using the AWS Organizations service. 

Tracking Reserved Instances

Reserved Instances (RIs) are pre-provisioned instance-hours for AWS resources such as Amazon EC2 or Amazon Relational Database Service (Amazon RDS) instances. In general, the usage costs for pre-provisioned resources are significantly lower than for on-demand resources. The depth of the RI discount is determined by the term (one or three years) and the payment model (All Upfront, Partial Upfront, or No Upfront). If the RIs are all in the same Availability Zone, they are also a capacity reservation. In addition, Linux or Unix Amazon EC2 RIs in the same Region (and configured for shared tenancy) are size-flexible. In other words, the RI discount, which is normally applicable only to the specified instance size, can be applied on the fly to any instance size in the instance family. 

It can be very tricky to track the actual cost of an RI in any given billing period. The AWS Cost and Usage report uses the following fields to give as clear a picture as possible of RI costs:

• AmortizedUpfrontCostForUsage: The initial upfront payment (for Full and Partial Upfront RIs) amortized for actual RI usage time to date. For No Upfront RIs, the value is 0.
• AmortizedUpfrontFeeForBillingPeriod: The initial upfront payment amortized over the current billing month. For No Upfront RIs, the value is 0.
• RecurringFeeForUsage: For Partial Upfront and No Upfront RIs, the recurring fee payment amortized for usage time. The value for All Upfront RIs is 0.
• EffectiveCost: The total effective cost of usage with RI discounts applied (i.e., the sum of AmortizedUpfrontCostForUsage + RecurringFeeForUsage).

In addition to the effective RI costs incurred, the AWS Cost and Usage report also shows RI capacity that has not been used yet during the month or, in the case of the final invoice, was not used during the monthly billing period.

Consolidated Billing

An organization can consolidate billing across all its AWS accounts into a single invoice and thus better leverage volume pricing tiers and RI discounts. The master account pays for all the charges aggregated by the linked member accounts, but each member account also gets its own bill for showback or chargeback purposes.

When billing is consolidated, the AWS Cost and Usage report uses blended rates (i.e., the averaged rates of the RIs and the on-demand instances used by all the member accounts). In the Line Item details, this will be shown in the following two fields:

• BlendedRate: The average cost incurred for each unique SKU across all the member accounts. For example, the blended rate for Amazon S3 standard storage is calculated by dividing the total cost of storage by the amount of data stored per month.
• BlendedCost: This is calculated by multiplying the BlendedRate by the UsageAmount.

Stay tuned next week for the second part of this series where we explore the usage of management tools that can help you take control of your AWS bill.

As an AWS Advanced Consulting Partner, Media Temple can help you get the most from your AWS cloud. Reach out anytime.