I am attempting to upload a file into a S3 bucket, but I don't have access to the root level of the bucket and I need to upload it to a certain prefix instead. The following code:

import boto3
s3 = boto3.resource('s3')
open('/tmp/hello.txt', 'w+').write('Hello, world!')
s3_client.upload_file('/tmp/hello.txt', bucket_name, prefix+'hello-remote.txt')

Gives me an error:

An error occurred (AccessDenied) when calling the PutObject operation: Access Denied: ClientError Traceback (most recent call last): File "/var/task/tracker.py", line 1009, in testHandler s3_client.upload_file('/tmp/hello.txt', bucket_name, prefix+'hello-remote.txt') File "/var/runtime/boto3/s3/inject.py", line 71, in upload_file extra_args=ExtraArgs, callback=Callback) File "/var/runtime/boto3/s3/transfer.py", line 641, in upload_file self._put_object(filename, bucket, key, callback, extra_args) File "/var/runtime/boto3/s3/transfer.py", line 651, in _put_object **extra_args) File "/var/runtime/botocore/client.py", line 228, in _api_call return self._make_api_call(operation_name, kwargs) File "/var/runtime/botocore/client.py", line 492, in _make_api_call raise ClientError(parsed_response, operation_name) ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

bucket_name is in the format abcd while prefix is in the format a/b/c/d/. I'm not sure if the error is due to the slashes being wrong or if there's some way you can specify the prefix elsewhere, or if I don't have write permissions (although I supposedly do).

This code executes without any errors:

for object in output_bucket.objects.filter(Prefix=prefix): print(object.key)

Although there is no output as the bucket is empty.

2

5 Answers

I'm assuming you have all this set up:

1. AWS Access Key ID and Secret Key set up (typically stored at ~/.aws/credentials
2. You have access to S3 and you know your bucket names & prefixes (subdirectories)

According to the Boto3 S3 upload_file documentation, you should upload your upload like this:

upload_file(Filename, Bucket, Key, ExtraArgs=None, Callback=None, Config=None)

import boto3
s3 = boto3.resource('s3')
s3.meta.client.upload_file('/tmp/hello.txt', 'mybucket', 'hello.txt')

The key to note here is s3.meta.client. Don't forget that--it worked for me!

I hope that helped.

1

import boto3
s3 = boto3.resource('s3')
s3.meta.client.upload_file( 'csv1.csv', "bucketname", "prefixna/csv1.csv")

Turns out I needed SSE:

transfer = S3Transfer(s3_client)
transfer.upload_file('/tmp/hello.txt', bucket_name, prefix+'hello-remote.txt', extra_args={'ServerSideEncryption': "AES256"})

2

Below is an alternative to John Adjei's answer. This is also taken from the Boto3 S3 upload_file documentation. Because the Client is low-level (low abstraction / closer to machine code) it can improve performance - especially if you a dealing with Big Data.

import boto3
s3 = boto3.client('s3')
with open("FILE_NAME", "rb") as f: s3.upload_fileobj(f, "BUCKET_NAME", "OBJECT_NAME")

1

Here is my answer:

import boto3
s3_client = boto3.client(service_name='s3', region_name='ap-southeast-1', aws_access_key_id='AWS_ACCESS_KEY_ID', aws_secret_access_key='AWS_SECRET_ACCESS_KEY')
dest_bucket = 'data-lake'
dest_prefix = 'datamart/my_file_name/'
file_name = 'my_file_name'+ '.parquet'
s3.meta.client.delete_object(Bucket=dest_bucket,Key=dest_prefix + file_name)

2

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password Submit

Post as a guest

Post Your Answer Discard

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy