How can we use a Lambda function to receive SNS alerts to Slack when an AWS Glue job fails a retry?

Notification Structure

As you see the structure above, we will learn how we can use and implement these.

1. Create Lambda Function

What is the Amazon Lambda?

Lambda is a compute service that lets you run code without provisioning or managing servers. Lambda runs your code on a high-availability compute infrastructure and performs all of the administration of the compute resources, including server and operating system maintenance, capacity provisioning and automatic scaling, code monitoring and logging. With Lambda, you can run code for virtually any type of application or backend service.

Firstly, we need to create a lambda function that sends an alert message to Slack.

Go Lambda and Click Create Function.
Give Function Name and Create Function.

Then we can add this python script.
This script is able to send notifications to your slack channel that you add webhook URL
Note: Be sure to replace webhook_uri with the webhook uri of your webhook and channel.

As you see, we have just created our Lambda Function.

We will use Function ARN to configure SNS.

2. Create SNS

What is the Amazon SNS (Simple Notification Service)

Amazon Simple Notification Service (Amazon SNS) is a fully managed messaging service for both application-to-application (A2A) and application-to-person (A2P) communication.

The A2A pub/sub functionality provides topics for high-throughput, push-based, many-to-many messaging between distributed systems, microservices, and event-driven serverless applications. Using Amazon SNS topics, your publisher systems can fanout messages to a large number of subscriber systems, including Amazon SQS queues, AWS Lambda functions, HTTPS endpoints, and Amazon Kinesis Data Firehose, for parallel processing. The A2P functionality enables you to send messages to users at scale via SMS, mobile push, and email.

Go to the Amazon SNS section and click Create Topic.

Enter the Name and Create Topic button.

As you see, we have just created the Topic.
Click Subscription.

The protocol should be AWS Lambda. (Our python script.)

The endpoint is our lambda function ARN. Copy our lambda function ARN.

Then click Create Subscription.

We created a subscription successfully.

3. Create Amazon EventBridge

What is the Amazon EventBridge

Amazon EventBridge is a serverless event bus service that you can use to connect your applications with data from a variety of sources. EventBridge delivers a stream of real-time data from your applications, software as a service (SaaS) applications, and AWS services to targets such as AWS Lambda functions, HTTP invocation endpoints using API destinations, or event buses in other AWS accounts.

Go to Amazon EventBridge -> Rules -> Create Rules

Add Name and Click Next

Event source ->AWS events or EventBridge partner events

Sample event -> AWS events

Event source -> Aws services

Aws service -> Glue

Event type -> Glue Job State Change

Then click Edit pattern

We need to add this detailed information.

jobName is our AWS Glue Job Name.

Click Next.

Note: If you don’t know how you can create an AWS Glue Job Name, you can look at this article.

We will add our topic as a target for EventBridge.
Because, when we get an error from our glue job, eventBridge triggers our SNS topic.

Target types -> AWS service
Target -> SNS topic
Topic -> Select the topic that you created.

The last part is reviewing and Creating Rule.

I run my job then it was a failure.

When I looked at the slack channel, I saw my alert message.

Slack Webhook message

Of course, you can write the lambda function more complex. I have explained it here with simple examples for simplicity.

for more information,

Amazon SNS

Amazon Lambda

Thank you :)




Java Developer @XPoint-Tech

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Understanding the Power of Firebase Security Rules — Part 1

This week in #Scala (Mar 23, 2020)

Pearls of Wisdom for the Junior Developer

Avoiding Boilerplate Code With MapStruct, Spring Boot and Kotlin

Things you must do after Apps development

Flutter Introduction Screen [Intro slides]

meters seems effective when you are standing still inside or even outside with

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Hasan Kadir Demircan

Hasan Kadir Demircan

Java Developer @XPoint-Tech

More from Medium

Need a storage which is very easy to manage in AWS?

How to release Lambda functions to AWS using Azure DevOps Release Pipelines?

AWS DMS Heartbeat — Avoid Critical Production DB Issues

Data Pipeline using AWS S3, Glue Crawler, IAM and Athena