CloudWatch-ing:

Creating More Useful Logs & Alerts with AWS

Rhia Dixon

Thank You Sponsors!

Sponsors
Rhia Dixon headshot

Rhia Dixon

@TheFakeRiRi

  • VeriShip Software Engineer by day
  • Everything else by night
  • Previously afflicted by over-logging

Why should you listen to me?

Over 1 Year Playing with CloudWatch

Visibility was my 1st real project

I could gush about this for DAYS!

Story Time

Back in the day...

Things were good...

...or so we thought.

Great Debacle of 2018

We were definitely doing it all wrong

Seriously.

    Who

    What

    When

    Where

    Why

    How

Enter AWS CloudWatch!

What & Why


Previously

  • Logging everything
  • Unformatted
  • Unsearchable

vs.

Currently

  • Log levels
  • Formatted
  • Searchable

Where & How


Previously

  • Local Text File
  • Per App/Machine
  • Batch Appended

vs.

Currently

  • Logs in Cloud
  • Groups & Streams
  • Constant Push

Before CloudWatch

After CloudWatch

My Favorite Things

Ease of Use & Integrations

Centralized Logging

Searchable Logs

Cross-App Metrics

SHOW & TELL

Terraform Metrics

Basic Metric Filter


							resource "aws_cloudwatch_log_metric_filter" "unique-name-of-your-filter-resource" {
							     name    = "name-of-your-filter-pattern"
							     pattern = "\"the actual pattern you are looking for\""
							     log_group_name = "logroup/app.name.orwhatevs/"

							     metric_transformation {
							          namespace = "${var.metric_namespace}"
							          name  = "Easy-To-Spot-Name-Of-Your-Metric"
							          value = "1"
							          default_value = "0"
							      }
							}
						

Parsed Metric Filter


							resource "aws_cloudwatch_log_metric_filter" "unique-name-of-another-filter-resource" {
							     name    = "your-new-filter-pattern-name"
							     pattern = "[..., user=*root*, spacer=*-*, colx=*Some* && coly=*Web*, count]"
							     log_group_name = "logroup/app.name.orwhatevs/"

							     metric_transformation {
							          namespace = "${var.metric_namespace}"
							          name  = "Easy-To-Spot-Name-Of-Your-Metric"
							          value = "$count"
							          default_value = "0"
							      }
							}
						

Basic Metric Alarm


							resource "aws_cloudwatch_metric_alarm" "unique-name-of-your-alarm-resource" {
							     alarm_name = "Easy-To-Spot-Name-Of-Your-Metric"
							     alarm_description   = "10 errors in 10 minutes Error connecting to SQL Server"
							     comparison_operator = "GreaterThanOrEqualToThreshold"
							     metric_name         = "Easy-To-Spot-Name-Of-Your-Metric"
							     namespace           = "${var.metric_namespace}"
							     period              = "900"
							     statistic           = "Sum"
							     threshold           = "10"
							     evaluation_periods  = "1"
							     datapoints_to_alarm = "1"
							     treat_missing_data  = "notBreaching"
							     alarm_actions       = [
							          "${data.aws_sns_topic.yada.arn}","${data.aws_sns_topic.blah.arn}"]
							}
						

Resources

CloudWatch Documentation

Terraform with CloudWatch

Grafana Documentation

Major Keys

Major Keys

Log Levels Matter

Centralize Your Logs

Log Strategically

Send Alerts to People

Metrics Make a World of Difference

This could be YOU!

Questions?

Questions
Slides available at: https://www.rhiadixon.com/CLOUDWATCHING
@TheFakeRiRi #CLOUDWATCHING #prdcwpg2019 @prairiedevcon