AWS Cloud Cost Management Dashboard

CUDOS Dashboard: AWS Cost Intelligence Solution

In today's cloud landscape, managing AWS costs is key to efficiency and cutting waste. The CUDOS Dashboard (Cost and Usage Dashboard Operations Solution) is an open-source tool in AWS Cloud Intelligence Dashboards. It uses AWS Cost and Usage Reports (CUR) to offer detailed insights, recommendations, and visuals to optimize costs. This blog guides you through deploying CUDOS, its architecture, core services, and customizing it with user-defined tags, based on practical use and official AWS resources.

Introduction to CUDOS

CUDOS transforms raw CUR data into actionable intelligence using services like AWS Glue for data integration, Athena for querying, and QuickSight for visualizations. It offers high-level overviews alongside deep dives into usage patterns, supporting multi-account setups in AWS Organizations. Key benefits include ML-driven recommendations for rightsizing, reservations, and AWS Marketplace spend analysis.

The CUR provides comprehensive metadata on services, pricing, Reserved Instances, and Savings Plans, making it the foundation for CUDOS. AWS Glue prepares this data, Athena analyzes it, and QuickSight delivers interactive dashboards.

Prerequisites

Before diving in, ensure you have:

No costs beyond standard AWS usage apply. QuickSight's SPICE engine efficiently handles queries.

Architecture Overview

CUDOS architecture centers on CUR data flowing through a serverless pipeline:

Architecture of the CUDOS dashboard implementation
fig: Architecture of the CUDOS dashboard implementation

For multi-account scenarios, replicate CUR data from the Payer Account to the Data Collection Account using S3 replication rules.

Deployment Workflow

Deployment is streamlined via Terraform wrappers around CloudFormation, ensuring reproducibility.

Step 1: Set Up CUR in the Payer Account

Create an S3 bucket and CUR report:

module "cur_payer" {
  source = "../../modules/cur-source"
}

This provisions the bucket with necessary permissions for multi-payer support.

Step 2: Deploy CUDOS in the Data Collection Account

Use the CID module:

module "cid_dashboard" {
  source          = "../../modules/cudos-dashboard"
  stack_name      = "CUDOS-STACK"
  template_bucket = "your-template-bucket" # Must exist
  stack_parameters = {
    "PrerequisitesQuickSight"            = "yes"
    "PrerequisitesQuickSightPermissions" = "yes"
    "QuickSightUser"                     = "your-quicksight-user"
    "DeployCUDOSv5"                      = "yes"
    "DeployCostIntelligenceDashboard"    = "no"
    "DeployKPIDashboard"                 = "no"
    "CURBucketPath"                      = "s3://your-cur-bucket/path"
  }
}

This deploys Glue crawlers, Athena views, Lambda functions, and QuickSight dashboards. Wait 24 hours for initial data population. Schedules for crawlers and datasets ensure regular updates.

Post-deployment, access QuickSight, select the "CUDOS Dashboard v5," and explore visualizations. Avoid paginated reports to control costs.

Validate Deployment

# Check outputs
terraform output

# In QuickSight
- Search: "CUDOS Dashboard v5"
- Open → Verify data in visuals

# In Athena
SELECT bill_payer_account_id, line_item_usage_account_id, sum(line_item_unblended_cost) as cost
FROM summary_view
GROUP BY 1, 2
LIMIT 10;
CUDOS Dashboard
fig: CUDOS Dashboard (Executive Summary View)

Services Under the Hood

CUDOS orchestrates several AWS services seamlessly. Here's a walkthrough:

AWS QuickSight

Dashboards: Access "CUDOS Dashboard v5" for visuals.

Datasets: Sourced from Athena views; scheduled refreshes ensure freshness. Manually refresh if needed.

Data Sources: Connected to Athena workgroup "CID."

CUDOS Dashboard v5
fig: CUDOS Dashboard v5

AWS Athena

Workgroup: "CID" with tables and views (e.g., summary_view). Avoid deleting views.

Queries: Run SQL to explore CUR data.

Athena Overview
fig: Athena Overview
Athena Overview
fig: Athena Overview

AWS Glue

Databases: Contains CUR tables.

Crawlers: Scheduled to index S3 data; view IAM roles and run history.

Tables: Check schemas for partitioned data.

Glue Database
fig: Select the Glue Database
a. Database:

In the left panel, click Database. You will see the database name; if you followed the steps, it will be your database. Otherwise, you will see your input. Click it to check for tables.

Glue Crawler
fig: Select the Database
Glue Crawler
fig: Glue Database Overview
c. Crawlers:

Navigate to left panel and click on Crawlers.

AWS Glue Crawler
fig: Glue Crawler Overview

To view the details of the crawler, click on the crawler.

Glue Crawler Details
fig: Glue Crawler Details

Here you can see the IAM roles it is using and other details like crawler run, schedule and many more.

AWS Lambda

Functions: Handle automation, like dashboard creation via cid-cmd.

Lambda Function
fig: Lambda Function Overview
a. Function:

Click on Functions in the left panel to see the list of functions created.

Lambda Function List
fig: Lambda Function List
b. Details:

Click on the function to see the details of the function.

Lambda Function Details
fig: Lambda Function Details

AWS S3

Buckets: Store CUR files (e.g., cid-account-number-local). Directories include manifests, partitions, and status files.

S3 Bucket
fig: S3 Bucket
a. Bucket:

Navigate to the left panel and click on buckets and search your local

Note: If you followed along, your bucket names will look like cid-account-number-local and aws-athena-query-results-cid-account-id-region.

S3 Bucket Overview
fig: S3 Bucket Overview
b. bucket directories:

Click on object and click the directory

S3 Bucket Directories
fig: S3 Bucket Directories

AWS CloudFormation

Stacks: Manage resources; view events for deployment history. Update via Terraform to avoid conflicts.

CloudFormation Stack
fig: CloudFormation Stack Overview
CloudFormation Stack
fig: CloudFormation Stack Overview
CloudFormation Stack
fig: CloudFormation Stack Overview
CloudFormation Stack
fig: CloudFormation Stack Overview

AWS Billing and Cost Management

Data Exports: Configure CUR reports; view details for delivery settings.

CUR Report
fig: CUR Report Details
a. Data export:

In the left panel, click Data Export and select the report. If you deployed via Terraform with default values, the report name remains the same; otherwise, select your custom name.

CUR Report Details
fig: CUR Report Details

Here you can see the details of the report.

CUR Report Details
fig: CUR Report Details

These services integrate to provide end-to-end data flow without manual intervention.

Adding User-Defined Tags

Customize CUDOS by incorporating tags for grouping (e.g., by application or business unit) using cost allocation tags or categories.

Prerequisites

Step-by-Step Guide

Add Cost Categories

In Billing and Cost Management:

  1. Navigate to Cost Categories > Create.
  2. Define rules (e.g., by account or tag values).
  3. Set lookback periods if needed.
  4. Review and create.

Modify Queries in Athena

Select AwsDataCatalog and your database.

Edit views (e.g., summary_view): Add tag columns (e.g., lineitem_resourcetags_user_silo AS silo).

Update GROUP BY clauses.

-- Add new column
line_item_resource_tags_user_team AS team,

-- Update GROUP BY
GROUP BY 1, 2, 3, ..., team

Run and verify.

Repeat for hourly_view and resource_view.

Exploring the CUDOS Dashboard

Once deployed, dive into CUDOS Dashboard v5 in QuickSight. Organized into interactive sheets, it provides granular, recommendation-driven analysis of CUR data. Use global filters (top-right) for time ranges, accounts, regions, or tags. Here's a hands-on tour of key sheets and features:

Best Practices and Troubleshooting

Organizations using CUDOS often achieve 15-25% cost savings through insights.

Conclusion

Key Takeaway: CUDOS Dashboard transforms AWS cost management from reactive to proactive, providing the visibility and intelligence needed to optimize cloud spending effectively.

Implementing CUDOS provides a robust tool for AWS cost management. From deployment with Terraform to custom tagging, this guide covers the essentials to start. Beyond visibility, CUDOS converts raw CUR data into actionable intelligence, offering ML-powered savings recommendations, resource-level insights, and accurate forecasting.