How can we use a Lambda function to receive SNS alerts to Slack when an AWS Glue job fails a retry?
As you see the structure above, we will learn how we can use and implement these.
1. Create Lambda Function
What is the Amazon Lambda?
Lambda is a compute service that lets you run code without provisioning or managing servers. Lambda runs your code on a high-availability compute infrastructure and performs all of the administration of the compute resources, including server and operating system maintenance, capacity provisioning and automatic scaling, code monitoring and logging. With Lambda, you can run code for virtually any type of application or backend service.
Firstly, we need to create a lambda function that sends an alert message to Slack.
Go Lambda and Click Create Function.
Give Function Name and Create Function.
Then we can add this python script.
This script is able to send notifications to your slack channel that you add webhook URL
Note: Be sure to replace webhook_uri with the webhook uri of your webhook and channel.
As you see, we have just created our Lambda Function.
We will use Function ARN to configure SNS.
2. Create SNS
What is the Amazon SNS (Simple Notification Service)
Amazon Simple Notification Service (Amazon SNS) is a fully managed messaging service for both application-to-application (A2A) and application-to-person (A2P) communication.
The A2A pub/sub functionality provides topics for high-throughput, push-based, many-to-many messaging between distributed systems, microservices, and event-driven serverless applications. Using Amazon SNS topics, your publisher systems can fanout messages to a large number of subscriber systems, including Amazon SQS queues, AWS Lambda functions, HTTPS endpoints, and Amazon Kinesis Data Firehose, for parallel processing. The A2P functionality enables you to send messages to users at scale via SMS, mobile push, and email.
Go to the Amazon SNS section and click Create Topic.
Enter the Name and Create Topic button.
As you see, we have just created the Topic.
Click Subscription.
The protocol should be AWS Lambda. (Our python script.)
The endpoint is our lambda function ARN. Copy our lambda function ARN.
Then click Create Subscription.
We created a subscription successfully.
3. Create Amazon EventBridge
What is the Amazon EventBridge
Amazon EventBridge is a serverless event bus service that you can use to connect your applications with data from a variety of sources. EventBridge delivers a stream of real-time data from your applications, software as a service (SaaS) applications, and AWS services to targets such as AWS Lambda functions, HTTP invocation endpoints using API destinations, or event buses in other AWS accounts.
Go to Amazon EventBridge -> Rules -> Create Rules
Add Name and Click Next
Event source ->AWS events or EventBridge partner events
Sample event -> AWS events
Event source -> Aws services
Aws service -> Glue
Event type -> Glue Job State Change
Then click Edit pattern
We need to add this detailed information.
jobName is our AWS Glue Job Name.
Click Next.
Note: If you don’t know how you can create an AWS Glue Job Name, you can look at this article.
We will add our topic as a target for EventBridge.
Because, when we get an error from our glue job, eventBridge triggers our SNS topic.
Target types -> AWS service
Target -> SNS topic
Topic -> Select the topic that you created.
The last part is reviewing and Creating Rule.
I run my job then it was a failure.
When I looked at the slack channel, I saw my alert message.
Of course, you can write the lambda function more complex. I have explained it here with simple examples for simplicity.
for more information,
EventBridge
Thank you :)
