Access an S3 Bucket using SFTP and password-based authentication

By default, a new AWS Transfer Family endpoint uses the service-managed, internal user directory for SSH key-based authentication (and not password-based authentication). However, you can instead use an IdP (Identity Provider) of your choice and authenticate users using a password (or a combination of password and key authentication). This solution explains how to access an S3 bucket using SFTP in scenarios where only password-based authentication is available (for instance, when using a legacy SFTP that uses passwords to authenticate and it's not possible to change the client).

AWS Transfer Family provides the following options for working with custom identity providers:

• Use AWS Lambda to connect your identity provider: https://docs.aws.amazon.com/transfer/latest/userguide/custom-lambda-idp.html
• Use Amazon API Gateway to connect your identity provider: https://docs.aws.amazon.com/transfer/latest/userguide/authentication-api-gateway.html
  

I'm going to explain how to set up the solution provided by the sample "AWS Secrets Manager stack template" presented in the "Lambda function examples" section of this page: https://docs.aws.amazon.com/transfer/latest/userguide/custom-lambda-idp.html#authentication-lambda-examples

Unfortunately, the official documentation page does not clearly provide all the steps to make this solution work, but I'm going to provide the missing information.

Deploying the solution

You start by creating a new bucket to host the files (unless you already have an existing bucket you want to reuse). You can create it on the AWS Management Console or by running the equivalent AWS CLI Command (use your own unique bucket name):

aws s3 mb s3://sftp-s3bucket-javier

Then you deploy the provided CloudFormation template: https://s3.amazonaws.com/aws-transfer-resources/custom-idp-templates/aws-transfer-custom-idp-secrets-manager-lambda.template.yml

This template creates the Lambda function that will authenticate users by comparing the information provided in the connection request with the data stored in a Secrets Manager secret (it uses one secret per user). Optionally, the stack can create a new AWS Transfer Family-managed SFTP server.

NOTE: The deployment will take around 4-5 minutes to finish.

Let's review the properties of the SFTP server that was automatically created:

To allow password authentication only, we should edit the Identity Provider details:

Also, in the "Additional details" section, we should define the CloudWatch Logs Log group that will store all connection activities and errors.

NOTE: You must create the Log group or re-use an existing one.

Once you have correctly set up the S3 bucket, SFTP server and Lambda function, it's time to define the permissions that the SFTP user will use when interacting with S3. AWS Transfer Family assumes an IAM role to access Amazon S3 on behalf of the connecting user.

The beauty of the Lambda function's implementation with Secrets Manager is that each user can have his/her own IAM Role (and, therefore, IAM Policy) and even S3 bucket/folder. Note that you can even use Session Policies per user (not shown in this solution).

To create an IAM Role, you first add a new Customer managed policy (let's call it sftp-s3-fullaccess-policy) with the following sample permissions (the user can upload, download and delete objects within the associated bucket):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::sftp-s3bucket-javier"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObjectVersion",
                "s3:DeleteObject",
                "s3:GetObjectVersion"
            ],
            "Resource": "arn:aws:s3:::sftp-s3bucket-javier/*"
        }
    ]
}

To create the IAM Role (let's call it sftp-s3-fullaccess-role), we must enable trust in the Transfer service and attach the newly created IAM Policy.

NOTE: You can create a hierarchy of home folders for different SFTP users. I've only shown the option where the whole bucket is accessible by the connecting user. Take note of the IAM Role's ARN. It will look like: arn:aws:iam::<ACCOUNT_ID>:role/sftp-s3-fullaccess-role.

The last step of the configuration process is to create a secret in Secrets Manager.

In the AWS Secrets Manager console, create a new secret by choosing "Store a new secret" and choose "Other type of secret".

Create the following key-value pairs. The key names are case-sensitive:

Save the secret in the following format: aws/transfer/server-id/username, for example: aws/transfer/s-1332abb90ade4f4c9/testuser.

NOTE: It is critical that the secret's name and the keys and values used follow the specified format, particularly the "/" before and after the bucket name.

Choose all the default actions in the remaining screens to create the secret.

Testing the solution

Once the secret has been created, you can test that everything works. To test graphically, go to the details of the SFTP server and use the Actions drop-down menu.

Or, alternatively, using the AWS CLI:

aws transfer test-identity-provider --server-id s-1332abb90ade4f4c9 --user-name testuser --user-password thisismypassword --server-protocol SFTP --source-ip 127.0.0.1 --region eu-west-1

{
    "Response": "{\"HomeDirectory\":\"/sftp-s3bucket-javier/\",\"HomeDirectoryType\":\"PATH\",\"Role\":\"arn:aws:iam::9XXXXXXXXXX5:role/sftp-s3-fullaccess-role\",\"UserName\":\"testuser\",\"IdentityProviderType\":\"AWS_LAMBDA\",\"EncryptionOptions\":{\"EncryptionType\":\"BUCKET_ENCRYPTION\"},\"IdentityProviderRequestId\":\"75b765a3-9af1-49e9-9094-281334287057\"}",
    "StatusCode": 200,
    "Message": ""
}

Finally, you can use any SFTP client to interact with S3, for instance, Filezilla:

The CloudWatch Log group will contain any errors and activity logs. These are some sample log entries:

{
    "role": "arn:aws:iam::9XXXXXXXXXX5:role/sftp-s3-fullaccess-role",
    "identity-provider-request-id": "257ac309-f920-462e-bab5-65035cc83156",
    "activity-type": "CONNECTED",
    "macs": "<implicit>,<implicit>",
    "ciphers": "aes256-gcm@openssh.com,aes256-gcm@openssh.com",
    "client": "SSH-2.0-FileZilla_3.67.0",
    "source-ip": "66.XX.XX.179",
    "resource-arn": "arn:aws:transfer:eu-west-1:9XXXXXXXXXX5:server/s-1332abb90ade4f4c9",
    "home-dir": "/sftp-s3bucket-javier/",
    "user": "testuser",
    "kex": "curve25519-sha256",
    "session-id": "0030174f92fcc5c23c36"
}

{
    "mode": "CREATE|TRUNCATE|WRITE",
    "path": "/sftp-s3bucket-javier/VPC.tif",
    "activity-type": "OPEN",
    "resource-arn": "arn:aws:transfer:eu-west-1:9XXXXXXXXXX5:server/s-1332abb90ade4f4c9",
    "session-id": "0030174f92fcc5c23c36"
}

Pricing Considerations

IMPORTANT: The following is a rough estimation of the costs, but please note that the actual costs can vary based on your usage.

1. AWS Transfer Family costs: AWS charges $0.30 per hour for each fully-managed SFTP endpoint. If you run your server 24/7 for a month (approximately 730 hours), this would cost around $219.

2. S3 costs: S3 pricing depends on the amount of data stored, the number of requests made (GET, PUT, COPY, etc.), and data transfer costs. For example, in the US East (N. Virginia) region, the first 50 TB stored per month costs $0.023 per GB. If you stored 100GB of data for a month, it would cost around $2.30.

3. Secrets Manager costs: AWS Secrets Manager costs $0.40 per secret per month for the first 10,000 secrets. If you're storing one secret, it would cost $0.40 per month.

4. Lambda costs: AWS Lambda charges based on the number of requests and the duration of code execution. The first 1 million requests per month are free, and after that, it's $0.20 per 1 million requests. Duration is calculated from the time your code begins executing until it returns or otherwise terminates, rounded up to the nearest 1ms. The price depends on the amount of memory you allocate to your function. If your Lambda function runs for less than 1 million times and executes within 1GB memory and less than 400,000 GB-seconds, it would be free.

So, adding up these costs, a rough estimate would be around $221.70 per month. This is a very basic estimation and your actual costs may vary. You should use the AWS Pricing Calculator to get a more accurate estimate based on your expected usage.