Email notification for AWS batch

Goal: Get an email notification when an AWS batch (single or array) job is completed. The problem (1) is to get email notifications only for the user that launched the job and (2) for array jobs only get one email notification when all the children jobs are completed.

Solution: For the array job problem the solution is to create a rule specific to the parental job. For the email notifications to only one user, the solution is to create a topic per job launched.

1. Create an AWS array job (called here sleep.json)

> cat sleep.json
{
    "jobName": "sleep-job-1",
    "jobQueue": "job-queue-1",
    "jobDefinition": "job-def-1",
    "arrayProperties": {
        "size": 3
    },
    "containerOverrides": {
        "command": [
            "sleep",
            "30"
        ]
    },
    "timeout": {
        "attemptDurationSeconds": 7200

    }
}

2. Create bash script that call aws batch job and fetch the jobId (called here sleep.sh)

> cat sleep.sh

#!/bin/bash

cmd="aws batch submit-job"
cmd="$cmd --cli-input-json file://sleep.json"

jobid=$(eval $cmd | \
        grep jobId | \
        sed -r 's|.+jobId\": \"(.+)\"$|\1|g')

3. Create an event rule (i.e. pattern to be matched) that is specific to the AWS array job

> cat rule.json

{

"Name": "rule-1",

"EventPattern": "{\"source\":[\"aws.batch\"],

\"detail-type\":[\"Batch Job State Change\"],

\"detail\":{\"jobId\":[\"JOBID\"],

\"status\":[\"FAILED\",\"SUCCEEDED\"]}}",

"State": "ENABLED",

"Description": "rule for specific jobId",

"EventBusName": "default"

}

> tail sleep.sh

sed -ri "s|\"jobId\\\\\":[^,]+|\"jobId\\\\\":[\\\\\"${jobid}\\\\\"]|g" \

rule.json

# register rule
aws events put-rule \
    --cli-input-json file://rule.json

4. Create a topic (i.e. communication channel) for that rule

> tail spleep.sh

# register topic

cmd="aws sns create-topic"

cmd="$cmd --name 'job-${jobid}"

topicArn=$(eval $cmd | \

grep TopicArn | \

sed -r 's|.+TopicArn\": \"(.+)\"$|\1|g')

5. Add iam role to allow notification

> cat sleep.sh
attributeValue="{\"Version\":\"2012-10-17\",

                 \"Id\":\"__default_policy_ID\",

                 \"Statement\":[{

                    \"Sid\":\"__default_statement_ID\",

                    \"Effect\":\"Allow\",

                    \"Principal\":{\"AWS\":\"*\"},

                    \"Action\":[

                      \"SNS:GetTopicAttributes\",

                      \"SNS:SetTopicAttributes\",

                      \"SNS:AddPermission\",

                      \"SNS:RemovePermission\",

                      \"SNS:DeleteTopic\",

                      \"SNS:Subscribe\",

                      \"SNS:ListSubscriptionsByTopic\",

                      \"SNS:Publish\",

                      \"SNS:Receive\"],

               \"Resource\":\"${topicArn}\",

              {\"Sid\":\"AWSEvents_rule-1_1\",

               \"Effect\":\"Allow\",

               \"Principal\":{\"Service\":\"events.amazonaws.com\"},

               \"Action\":\"sns:Publish\",

               \"Resource\":\"${topicArn}\"}]}"

aws sns set-topic-attributes \

   --topic-arn "$topicArn" \

   --attribute-name "Policy" \

   --attribute-value "$attributeValue"

6. create target (i.e. resource to be invoked) for the rule

> cat sleep.sh

# add target to rule

aws events put-targets \

--rule "$ruleName" \

    --targets "Id"=1,"Arn"="$topicArn"

PS: multiple rules can be linked to the same topic using different targets

7. Create subscription (mode of notification)
> cat sleep.sh

# create subscription

aws sns subscribe \

 --topic-arn "$topicArn" \

 --protocol "email" \

 --notification-endpoint "YourEmailAddress"