Slack Notification with CloudWatch Alarms & Lambda

ChatOps has emerged as one of the most effective techniques to implement DevOps. Hence, it will be great to receive notifications and infrastructure alerts into collaboration messaging platforms like Slack & HipChat.

AWS CloudWatch Alarms and SNS are a great mix to build a real-time notification system as SNS supports multiple endpoints (Email, HTTP, Lambda, SQS). Unfortunately SNS doesn’t support out of the box sending notifications to tools like Slack.

CloudWatch will trigger an alarm to send a message to an SNS topic if the monitoring data gets out of range. A Lambda function will be invoked in response of SNS receiving the message and will call the Slack API to post a message to Slack channel.

To get started, create an EC2 instance using the AWS Management Console or the AWS CLI:

1
2
3
4
5

aws ec2 run-instances --image-id ami_ID
    --count 1 --instance-type t2.micro
    --key-name KEY_NAME --security-group-ids SG_ID
    --region us-east-1 
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=demo}]'

Next, create a SNS topic:

1

aws sns create-topic --name cpu-utilization-alarm-topic --region us-east-1

Then, setup a CloudWatch alarm when the instance CPU utilization is higher than 40% and send notification to the SNS topic:

1
2
3
4
5
6
7
8
9
10
11
12

aws cloudwatch put-metric-alarm --region us-east-1 
    --alarm-name "HighCPUUtilization" 
    --alarm-description "High CPU Utilization" 
    --actions-enabled 
    --alarm-actions "TOPIC_ARN" 
    --metric-name "CPUUtilization" 
    --namespace AWS/EC2 --statistic "Average" 
    --dimensions "Name=InstanceName,Value=demo" 
    --period 60 
    --evaluation-periods 60 
    --threshold 40 
    --comparison-operator "GreaterThanOrEqualToThreshold"

As a result:

To be able to post messages to slack channel, we need to create a Slack Incoming WebHook. Start by setting up an incoming webhook integration in your Slack workspace:

Note down the returned WebHook URL for upcoming part.

The Lambda handler function is written in Go, it takes as an argument the SNS message. Then, it parses it and queries the Slack API to post a message to the Slack channel configured in the previous section:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"log"
	"net/http"
	"os"

	"github.com/aws/aws-lambda-go/lambda"
)

type Request struct {
	Records []struct {
		SNS struct {
			Type       string `json:"Type"`
			Timestamp  string `json:"Timestamp"`
			SNSMessage string `json:"Message"`
		} `json:"Sns"`
	} `json:"Records"`
}

type SNSMessage struct {
	AlarmName      string `json:"AlarmName"`
	NewStateValue  string `json:"NewStateValue"`
	NewStateReason string `json:"NewStateReason"`
}

type SlackMessage struct {
	Text        string       `json:"text"`
	Attachments []Attachment `json:"attachments"`
}

type Attachment struct {
	Text  string `json:"text"`
	Color string `json:"color"`
	Title string `json:"title"`
}

func handler(request Request) error {
	var snsMessage SNSMessage
	err := json.Unmarshal([]byte(request.Records[0].SNS.SNSMessage), &snsMessage)
	if err != nil {
		return err
	}

	log.Printf("New alarm: %s - Reason: %s", snsMessage.AlarmName, snsMessage.NewStateReason)
	slackMessage := buildSlackMessage(snsMessage)
	postToSlack(slackMessage)
	log.Println("Notification has been sent")
	return nil
}

func buildSlackMessage(message SNSMessage) SlackMessage {
	return SlackMessage{
		Text: fmt.Sprintf("`%s`", message.AlarmName),
		Attachments: []Attachment{
			Attachment{
				Text:  message.NewStateReason,
				Color: "danger",
				Title: "Reason",
			},
		},
	}
}

func postToSlack(message SlackMessage) error {
	client := &http.Client{}
	data, err := json.Marshal(message)
	if err != nil {
		return err
	}

	req, err := http.NewRequest("POST", os.Getenv("SLACK_WEBHOOK"), bytes.NewBuffer(data))
	if err != nil {
		return err
	}

	resp, err := client.Do(req)
	if err != nil {
		return err
	}
	defer resp.Body.Close()

	if resp.StatusCode != 200 {
		fmt.Println(resp.StatusCode)
		return err
	}

	return nil
}

func main() {
	lambda.Start(handler)
}

As Go is a compiled language, build the application and create a Lambda deployment package using the bash script below:

1
2
3
4
5
6
7
8
9
10

#!/bin/bash

echo "Building binary"
GOOS=linux GOARCH=amd64 go build -o main main.go

echo "Create deployment package"
zip deployment.zip main

echo "Cleanup"
rm main

Once created, use the AWS CLI to deploy the function to Lambda. Make sure to override the Slack WebHook with your own:

1
2
3
4
5
6

aws lambda create-function --function-name SlackNotification
    --zip-file fileb://./deployment.zip
    --runtime go1.x --handler main
    --role IAM_ROLE_ARN
    --environment Variables={SLACK_WEBHOOK=https://hooks.slack.com/services/TOKEN}
    --region AWS_REGION

Note: For non Gophers you can download the zip file directly from here.

From here, configure the invoking service for your function to be the SNS topic we created earlier:

1
2
3

aws sns subscribe --topic-arn TOPIC_ARN --protocol lambda
    --notification-endpoint LAMBDA_ARN
    --region AWS_REGION

Lambda Dashboard:

Let’s test it out, connect to your instance via SSH, then install stress which is a tool to do workload testing:

1

sudo yum install -y stress

Issue the following command to generate some workloads on the CPU:

1

stress --cpu 2 --timeout 60s

You should receive a Slack notification as below:

Note: You can go further and customize your Slack message.

Drop your comments, feedback, or suggestions below — or connect with me directly on Twitter @mlabouardy.