boto
Boto3 Error: botocore.exceptions.NoCredentialsError: Unable to locate credentials
try specifying keys manually s3 = boto3.resource(‘s3′, aws_access_key_id=ACCESS_ID, aws_secret_access_key= ACCESS_KEY) Make sure you don’t include your ACCESS_ID and ACCESS_KEY in the code directly for security concerns. Consider using environment configs and injecting them in the code as suggested by @Tiger_Mike. For Prod environments consider using rotating access keys: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_RotateAccessKey
AWS: Boto3: AssumeRole example which includes role usage
Here’s a code snippet from the official AWS documentation where an s3 resource is created for listing all s3 buckets. boto3 resources or clients for other services can be built in a similar fashion. # create an STS client object that represents a live connection to the # STS service sts_client = boto3.client(‘sts’) # Call … Read more
Move files between two AWS S3 buckets using boto3
If you are using boto3 (the newer boto version) this is quite simple import boto3 s3 = boto3.resource(‘s3’) copy_source = { ‘Bucket’: ‘mybucket’, ‘Key’: ‘mykey’ } s3.meta.client.copy(copy_source, ‘otherbucket’, ‘otherkey’) (Docs)
How to save S3 object to a file using boto3
There is a customization that went into Boto3 recently which helps with this (among other things). It is currently exposed on the low-level S3 client, and can be used like this: s3_client = boto3.client(‘s3’) open(‘hello.txt’).write(‘Hello, world!’) # Upload the file to S3 s3_client.upload_file(‘hello.txt’, ‘MyBucket’, ‘hello-remote.txt’) # Download the file from S3 s3_client.download_file(‘MyBucket’, ‘hello-remote.txt’, ‘hello2.txt’) print(open(‘hello2.txt’).read()) … Read more
How to handle errors with boto3?
Use the response contained within the exception. Here is an example: import boto3 from botocore.exceptions import ClientError try: iam = boto3.client(‘iam’) user = iam.create_user(UserName=”fred”) print(“Created user: %s” % user) except ClientError as e: if e.response[‘Error’][‘Code’] == ‘EntityAlreadyExists’: print(“User already exists”) else: print(“Unexpected error: %s” % e) The response dict in the exception will contain the … Read more
Read a file line by line from S3 using boto?
Here’s a solution which actually streams the data line by line: from io import TextIOWrapper from gzip import GzipFile … # get StreamingBody from botocore.response response = s3.get_object(Bucket=bucket, Key=key) # if gzipped gzipped = GzipFile(None, ‘rb’, fileobj=response[‘Body’]) data = TextIOWrapper(gzipped) for line in data: # process line