CloudWatch Alarms and Metric Filters

CloudWatch allows you to search through the vast quantity of logs likely generated in your AWS account using metric filters. This can be especially useful when carrying out investigations into what has been happening in your environment. CloudWatch alarms let you automate notification when logs that match these filters occur, making the path to improving incident response that much quicker.

CloudWatch metric filters

Metric filters are one of the features of CloudWatch Logs. Metric filters tell CloudWatch to find log files that match a specific pattern and then publish numerical metrics when it detects matching logs. Once you've stored your logs in CloudWatch, metric filters allow you to automate observations to reduce the time it takes to search through them and generate data. You can apply filters at a log group level, meaning they apply to all log streams in that log group.

The syntax

The exact syntax for the metric filter depends upon the log files being put in the log group. You can use a simple text filter for all log types. For example, a filter ERROR will return all log files with the word ERROR in them (case-sensitive). If you want to match multiple terms, you can put them in the filter separated by spaces (the equivalent of and logic), and if you want to match exact phrases, you can put them in quotation marks.

we’ll be using logs forwarded by CloudTrail, which arrive as JSON log events. This allows us to use more precise string-based metric filters.

If we wanted to create a metric filter to find all events where a bucket was created, we can use the following metric syntax:

{$.eventName = CreateBucket}

When filtering JSON, curly braces always surround the pattern. To define the key we want to match with, we use $. and then specify the name of the key from the JSON.

We can handle nested values by adding keys to the filter pattern. For example, if we wanted to filter all events attributed to the user with the username Alice, we would use the following metric syntax:

{$.userIdentity.userName = Alice}

You can then use compound expressions to be more selective in your filter patterns. The and expression lets you specify that you want to match both patterns you’ve been given. To filter all events where the user Alice creates a bucket, we can use the following:

{$.eventName = CreateBucket && $.userIdentity.userName = Alice}

If you want the filter to look for one expression or the other, we can use the or operator. The following filter pattern will filter log events attributed to Alice, or where a user has created a bucket:

{$.eventName = CreateBucket || $.userIdentity.userName = Alice} Other operators that can be used in metric filters are detailed in the CloudWatch documentation.

To add a metric filter to your log group, select it in the CloudWatch console and click on Create metric filter from the Actions drop-down menu.

You'll then be able to enter your filter pattern using the syntax discussed in the previous section. You can also use a test event log to see if it works as expected.

The next screen will take you through the configuration options for your metric filter. The namespace specifies where CloudWatch will store the metric. This is to ensure that similar metrics are kept together. The metric name will be what the metric is called within the namespace and should be unique.

The Metric value is what CloudWatch publishes when it finds logs that match the filter pattern. In this lab, we'll be using it to post a value of 1 every time it finds a matching log event. Other values can be used when you're directly monitoring values in the logs. For example, if you're monitoring specific IP addresses, you may want to publish the IP address for the metric. The default value is what CloudWatch publishes when it sees a log event that doesn’t match the pattern – we’ll be leaving this blank so that nothing gets published if the log file doesn’t fit.