Add support for AWS S3 Endpoint URLs

Advertisement

archie
Guest

Add support for AWS S3 Endpoint URLs

AWS SS has support for S3 bucket access over a private link by using an AWS S3 facility called Endpoint URLs.

Support for this would be most beneficial for networks without internet access and also more secure.

Using a private SFTP front-end in front of AWS S3 is considerably more expensive and complex. AWS S3 endpoints are very simple to use via AWS-CLI and Python boto3 libraries so shouldn't be too challenging to implement in WinSCP.

Reply with quote

Advertisement

martin
Site Admin
martin avatar
Joined:
Posts:
41,441
Location:
Prague, Czechia

Re: Add support for AWS S3 Endpoint URLs

What is AWS SS? Can you post some links/references to the feature? Also maybe an example how to use the feature via CLI/Python.

Reply with quote

Guest

Thank you for your continued interest in my feature request Martin.

Below is simple bash script that I use to list the top level folder of an S3 bucket using endpoint URLs.
#!/bin/sh
 
export AWS_ACCESS_KEY_ID="MyAwsAccessKeyValue"
export AWS_SECRET_ACCESS_KEY="MyAwsSecretAccessKeyValue"
 
# List bucket via endpoint URL.
/usr/local/bin/aws --region=ap-southeast-2 --endpoint-url https://bucket.vpce-0a123456789-d321zyz.s3.ap-southeast-2.vpce.amazonaws.com s3 ls s3://my-s3-bucket-name/
 
# List bucket via public internet.
/usr/local/bin/aws --region=ap-southeast-2 s3 ls s3://my-s3-bucket-name/

Here's a Python boto3 example
import boto3
 
session = boto3.session.Session()
 
client = session.client(
    service_name='s3',
    aws_access_key_id='MyAwsAccessKeyValue',
    aws_secret_access_key='MyAwsSecretAccessKeyValue',
    endpoint_url='https://bucket.vpce-0a123456789-d321zyz.s3.ap-southeast-2.vpce.amazonaws.com',
)
print(client.list_buckets())
The value for the endpoint-url parameter is derived independently to your S3 buckets and comes from an AWS component called S3 Endpoint of type Interface. There is also an S3 Endpoint of type Gateway but that is not relevant here as it is used primarily for AWS IAM policy. An attribute of the S3 Endpoint of type Interface is the URL above.

Creating S3 Endpoints of type Interface assumes the end-user is already familiar with this AWS specific technology as it involves medium level networking skills around using private IP address space.

Cheers, A.

Reply with quote

Advertisement

martin
Site Admin
martin avatar

Thanks. So what happens, when you put bucket.vpce-0a123456789-d321zyz.s3.ap-southeast-2.vpce.amazonaws.com to WinSCP's Host name box?

Reply with quote

Guest

Hi Martin,

I just tried your suggestion in the office which is locked down from the internet and it worked. Thank you very much for your patience with this and your valueable advice.

A.

Reply with quote

Advertisement

You can post new topics in this forum