Sending Alert Text Messages Based on CloudWatch Log Patterns

Published on December 24, 2021

By Hyuntaek Park

Senior full-stack engineer at Twigfarm

Introduction

Monitoring or getting an alert when unexpected happens is essential to keep our system reliable. System faults should be notified to appropriate person as early as it can be. This can be achieved easily with CloudWatch.

In this post, I create a simple lambda function and a couple of tests with a normal condition and one with an error condition. Once the error condition happens SMS text message is sent to my cell number all the way from North Virginia (us-east-1) to Seoul, Korea (ap-northeast-2).

The idea is simple:

• Exception happens during lambda execution
• Logging error in the CloudWatch associated with that lambda
• CloudWatch invokes another lambda function which sends an SMS text message if a predefined expression is in the log text

Creating a simple lambda function

Go to Lambda Functions and click Create function. Then create a simple function with any name you like.

The lambda code does simple division. Our goal is to get notified if DivideByZeroException occurs.

// cloudwatch-alert-test lambda code
exports.handler = async (event) => {
  console.log("event: ", JSON.stringify(event));
  const { x, y } = event;

  try {
    const result = divide(x, y);
    return result;
  } catch (e) {
    console.error(e);
    return null;
  }
};

const divide = (dividend, divisor) => {
  if (divisor === 0) {
    throw Error("DivideByZeroException");
  }
  return dividend / divisor;
};

Test input (OK case)

{
  "x": 10,
  "y": 2
}

Logs for OK test case

Test input (Error case)

{
  "x": 10,
  "y": 0
}

Logs for error test case

SMS sender lambda function

Let’s create another lambda function, called sms-sender, which gets triggered if the CloudWatch log contains certain pattern. For now this lambda function does nothing but prints out the input so that we are able to identify what the CloudWatch input format is like.

sms-sender

exports.handler = (event, context) => {
  console.log("event: ", JSON.stringify(event));
};

CloudWatch Subscription Filters

Now CloudWatch Subscription Filter connects following two lambda functions:

• cloudwatch-alert-test
• sms-sender

Go to CloudWatch –> Logs –> Log groups –> /aws/lambda/cloudwatch-alert-test

Then click on Subscription filters tab then choose Create Lambda subscription filter

Input as following:

• Lambda function: sms-sender
• Log format: Other
• Subscription filter pattern: DivideByZeroException
• Subscription filter name: ANY_NAME_YOU_LIKE

Click on Start streaming button to finish CloudWatch subscription filter setup.

This means that sms-sender lambda function gets invoked, if and only if the CloudWatch log contains string DivideByZeroException.

Testings

We created two tests for cloudwatch-alert-test. Run each of them and check the CloudWatch log for sms-sender to see if the function is invoked only when DivideByZeroException is happened.

Input passed from CloudWatch to sms-sender

What do you see in sms-sender CloudWatch log? Was the function triggered only on error test case? Probably yes. Now take a look at the event parameter value in the log. You would see some data value like the following:

What is it? We have to unzip to decode the data.

Decoding input from CloudWatch

Following code takes care of unzipping the input from CloudWatch.

const zlib = require("zlib");

exports.handler = (event, context) => {
  console.log("event: ", JSON.stringify(event));
  const payload = Buffer.from(event.awslogs.data, "base64");

  zlib.gunzip(payload, async (err, result) => {
    if (err) {
      console.error(err);
      context.fail(err);
    } else {
      const stringResult = result.toString("utf-8");
      console.log("stringResult: ", stringResult);
      context.succeed();
    }
  });
};

Variable stringResult shows stringified version of the input, which looks like:

If your alert method requirement is webhook notifications, then you can just add a few lines of code for HTTP POST call in the lambda function. End of story.

Since SMS text message is a means of alert requirement, one more setup is required.

Setting Amazon Simple Notification Service (SNS) for SMS text messaging

As of December 2021, mobile text messaging (SMS) feature is not supported in Seoul region (ap-northeast-2). So I changed my region to complete text messaging setup in SNS.

Supported Regions are listed here: https://docs.aws.amazon.com/sns/latest/dg/sns-supported-regions-countries.html

After changing region to us-east-1, go to Amazon SNS –> Mobile –> Text messaging (SMS). If Text messaging (SMS) is now showing in your region, it is likely that your region is not supporting this feature.

Under the Sandbox destination phone numbers, click Add phone number for adding recipients’ phone numbers. By default, you are in a sandbox mode which has a few restrictions such as:

• Recipients’ phone numbers have to be verified
• 1 USD is the maximum you can spend. (it suddenly stops sending messages after reaching the limit)

Note: you can always request limit increase here: https://aws.amazon.com/premiumsupport/knowledge-center/sns-sms-spending-limit-increase/

AWS support will get you out of sandbox as well upon the limit request.

After you add a recipient’s phone number for testing, now let’s go back to sms-sender lambda function in order to add some code for text messaging.

Publishing text message code in sms-sender

Following is the complete lambda code for sms-sender, in which message sending feature is added.

const zlib = require("zlib");

const AWS = require("aws-sdk");
AWS.config.update({ region: "us-east-1" });

exports.handler = (event, context) => {
  console.log("event: ", JSON.stringify(event));
  const payload = Buffer.from(event.awslogs.data, "base64");

  zlib.gunzip(payload, async (err, result) => {
    if (err) {
      console.error(err);
      context.fail(err);
    } else {
      const stringResult = result.toString("utf-8");
      console.log("stringResult: ", stringResult);
      await sendMessages(JSON.parse(stringResult));
      context.succeed();
    }
  });
};

const sendMessages = async (data) => {
  const sns = new AWS.SNS();
  const recipientNumbers = [YOUR_NUMBERS];

  try {
    for (const PhoneNumber of recipientNumbers) {
      console.log("sending to: " + PhoneNumber);
      await sns
        .publish({
          Message: `Please check ${data.logGroup}\n${data.logStream}`,
          PhoneNumber,
        })
        .promise();
    }
  } catch (err) {
    console.error("ERROR: ", err);
  }
};

Almost done. One last adjustment is left. Our lambda role needs a permission to send text messages.

IAM role for text messaging

Let’s move to IAM role for sms-sender lambda function.

In the lambda role, click Attach policies –> check AmazonSNSFullAccess –> click Attach policy button.

Final testing

Now all of our components are set up and ready. Go back to cloudwatch-alert-test lambda function and run error test case. Wait a few seconds, you should receive a text message if you set up everything correctly. You can modify the message to any way that you like to. Note that if your message is longer than 160 characters, they are split into separate messages.

Conclusion

You can design your alert system in various ways by applying different log patterns without much hassle utilizing CloudWatch subscription filters.

Getting alerts if something goes wrong is very useful. But it might be annoying if our phones vibrate too frequently than it should be. Yes, it is very tricky to get the balance right. I know DivideByZeroException is not an appropriate case to get SMS text messaging alert :)

Each system has different reliability requirements. When to fire alarm should be designed carefully according to the requirements. Avoiding false alarms is crucial point to consider as well.

Happy Christmas and to our reliable systems!