Friday, December 8, 2023

AWS Tips - Create an Interface VPC Endpoint to Access S3

 

Scenario


As a feature service in AWS, Simple Storage Service (S3) stores data as object comprising object data and metadata. S3 organizes objects with bucket, simply put, a bucket is a container where you store objects. A bucket has a unique name in the Region, and an object is identified by an object key. It may be a bit surprising that you can use SQLs to query contents of objects. S3 itself supports S3 Select, and you can submit standard SQL queries through Amazon Athena, an analytical service. You may think S3 is not a simple container. Right, it in effect can store snapshots for EBS and Amazon RDS, can be used as backup storages for AWS Backup, data stores for AWS Kinesis and AWS Glue, and it can also host a static website. S3 comes with a range of storage classes such as S3 Standard, S3 Intelligent-Tiering, S3 Standard-IA, S3 One Zone-IA, S3 Glacier Instant Retrieval, S3 Glacier Flexible Retrieval, S3 Glacier Deep Archive, etc. 

By default, the access to S3 goes through the Internet. In the meanwhile, S3 also supports private accesses via a Gateway VPC endpoint or an Interface VPC endpoint in the same Region. The traffic is not leaving AWS network, as illustrated by the diagram below. So, the services in the VPC can work with it in a more secure way. 



In this demo, we’ll create an interface VPC endpoint and test the access to a specific S3 bucket through it.



Examples


Create an Interface Endpoint

Sign in the AWS Management Console, choose VPC service, click Endpoints in the left pane. The home page of the Endpoints is presented where you can view the list of currently available endpoints in the VPC.

Click Create endpoint.



In the Name tag field, enter if-endpoint. You can leave it blank as it is optional.

For Service category, choose AWS services.

For Services, select S3 Interface.



Under VPC, choose the destination VPC from the dropdown list where you would like to create the endpoint, for example, the default VPC in the AWS Region.

For Subnets, we choose a public subnet and a private for the demo.

For Security groups, choose the one named by sg_public which allows all traffic.



For Policy associated with this endpoint, choose Full access.

Click Create endpoint.



The Console navigates back to the home page of the Endpoints. You can find the newly created interface endpoint on the list. An interface endpoint comes with a Regional DNS name and one or multiple Zonal DNS names depending on how many Availability Zones it is attached to. 



Moving to Subnets tab, you may notice that AWS has created a network interface for each subnet and assigned it a private IP address. As designed, the endpoint connects to S3 via the AWS PrivateLink.




Test Connectivity Using AWS CLI


We are about to carry out a few tasks on the test bucket tempfortest via the interface endpoint that we just created, vpce-05c5c15c114775198.

Log in the public instance and list up the contents in the bucket tempfortest.

aws s3 ls s3://tempfortest/ --region ap-northeast-1 --endpoint-url https://bucket.vpce-05c5c15c114775198-1zb58c4z.s3.ap-northeast-1.vpce.amazonaws.com

 The text in bold is the endpoint’s regional DNS name.



Log into the private instance and view the bucket as well. To connect to the private instance, we log in the public instance first and use it a transit server to log in the private one via SSH.



We upload another file, anotherfile.txt, to the test bucket.

aws s3 cp --endpoint-url https://bucket.vpce-05c5c15c114775198-1zb58c4z.s3.ap-northeast-1.vpce.amazonaws.com anotherfile.txt s3://tempfortest



We go back to the public instance and check the bucket again. The uploaded file appears there.




Endpoint Policy


We didn’t touch on more details about the Endpoint policy here. In nature, it is a JSON format document in the IAM policy language. You can restrict access to a specific bucket or a specific account. For example, the Full access we granted to the demo endpoint is stated as below.

{

    "Statement": [

{

"Effect": "Allow",

"Principal": "*",

"Action": "*",

"Resource": "*"

}

]

}

Moreover, you can set up a bucket policy on S3 side to further strength access control to directory level, object level. If we look more broadly, we can also design user/role-based access control according to the use cases. You can find plenty of examples in AWS help documentation.












No comments:

Post a Comment

React - Makeover in React: W3Schools How To

When it comes to building a React app, I get used to searching for a UI library to build the front pages, a straightforward way to achieve f...