Globus transfer to AWS EFS

Topic: I needed to download files shared via Globus (globus.org) to AWS EFS drive. Globus provides an option to share a directory to AWS S3 bucket but not directly to EFS; however, this requires having a paid account with Globus (do not work with personal endpoint). This solution below (installing Globus CLI and downloading the file from an EC2 instance with the EFS mounted) works even with Globus's personal endpoint. If Globus CLI is already installed on your EC2 frontend, you can skip to step 6.

1. Create EC2 instance

I suggest an EC2 instance with high bandwidth for a faster download (ex. c5n with up to 25 Gbps network bandwidth)

> aws ec2 run-instances \
--image-id ami-0beaa649c482330f7 \
--count 1 \
--instance-type c5n.2xlarge \
--key-name sfourat \
--security-group-ids sg-0d0e3364014a7fc7e sg-0173b74c97e48493e \
--subnet-id subnet-04f3da868a634843d \
--profile "tki-aws-account-310-rhedcloud/RHEDcloudAdministratorRole"

2. Mount EFS

> sudo yum -y update
> sudo yum -y install amazon-efs-utils
> sudo yum -y install nfs-utils
> sudo mkdir -p /mnt/efs
> sudo mount -t efs -o tls fs-57e8702f:/ /mnt/efs

3. Install Globus on your AWS EC2 frontend
> pip3 install globus-cli

4. Connect to Globus

> globus login --no-local-server

5. Authentify in local browser and get Native App Authorization Code

> Please authenticate with Globus here:

> ------------------------------------

> https://auth.globus.org/v2/oauth2/authorize?client_id=...

6. Install Globus personal server

> wget https://downloads.globus.org/globus-connect-personal/v3/linux/stable/globusconnectpersonal-latest.tgz

> tar -xzvf globusconnectpersonal-latest.tgz 

7. create an endpoint

> ./globusconnectpersonal -setup

Globus Connect Personal needs you to log in to continue the setup process.

We will display a login URL. Copy it into any browser and log in to get a

single-use code. Return to this command with the code to continue setup.

Login here:

-----

https://auth.globus.org/v2/oauth2/authorize?...

Input a value for the Endpoint Name: aws

registered new endpoint, id: ...

8. start endpoint

> ./globusconnectpersonal -start -restrict-paths rw/mnt/efs &

9. print all endpoints by current user

> globus endpoint search --filter-scope my-endpoints

10. Directory Listing

> globus ls 'endpointUUID:/'

11. Start transfer

> globus transfer shared-endpoint:/ myendpoint:/