> aws s3 mb s3://rv398-20220712
> aws s3 cp /mnt/efs/Joana3/Data s3://rv398-20220712/Data --recursive
3. Change bucket lifecycle
Goal: Get an email notification when an AWS batch (single or array) job is completed. The problem (1) is to get email notifications only for the user that launched the job and (2) for array jobs only get one email notification when all the children jobs are completed.
Solution: For the array job problem the solution is to create a rule specific to the parental job. For the email notifications to only one user, the solution is to create a topic per job launched.
1. Create an AWS array job (called here sleep.json)# register rule
aws events put-rule \
--cli-input-json file://rule.json
4. Create a topic (i.e. communication channel) for that rule> Please authenticate with Globus here:
> ------------------------------------
> https://auth.globus.org/v2/oauth2/authorize?client_id=...
6. Install Globus personal server
> wget https://downloads.globus.org/globus-connect-personal/v3/linux/stable/globusconnectpersonal-latest.tgz
> tar -xzvf globusconnectpersonal-latest.tgz
7. create an endpoint
> ./globusconnectpersonal -setup
Globus Connect Personal needs you to log in to continue the setup process.
We will display a login URL. Copy it into any browser and log in to get a
single-use code. Return to this command with the code to continue setup.
Login here:
-----
https://auth.globus.org/v2/oauth2/authorize?...
Input a value for the Endpoint Name: aws
registered new endpoint, id: ...
8. start endpoint
> ./globusconnectpersonal -start -restrict-paths rw/mnt/efs &
9. print all endpoints by current user
> globus endpoint search --filter-scope my-endpoints
10. Directory Listing
> globus ls 'endpointUUID:/'
11. Start transfer
> globus transfer shared-endpoint:/ myendpoint:/
a. GitHub > Settings
b. Settings > Developer settings
c. Developer settings > Personal access tokens
1. Create an S3 bucket > aws s3 mb s3://rv398-20220712 2. Copy EFS files to the S3 bucket > aws s3 cp /mnt/efs/Joana3/Data s3://rv398-...