pandas uses boto for read_csv, so you should be able to: import boto data = pd.read_csv(‘s3://bucket….csv’) If you need boto3 because you are on python3.4+, you can import boto3 import io s3 = boto3.client(‘s3′) obj = s3.get_object(Bucket=”bucket”, Key=’key’) df = pd.read_csv(io.BytesIO(obj[‘Body’].read())) Since version 0.20.1 pandas uses s3fs, see answer below.
Jaminto did a great job of answering the question, but I recently went through the process myself and wanted to expand on Jaminto’s answer. I’m assuming that you already have an EC2 instance created and have associated an Elastic IP Address to it. Step #1: Install vsftpd SSH to your EC2 server. Type: > sudo … Read more
I have the same needs and created the following function that download recursively the files. The directories are created locally only if they contain files. import boto3 import os def download_dir(client, resource, dist, local=”/tmp”, bucket=”your_bucket”): paginator = client.get_paginator(‘list_objects’) for result in paginator.paginate(Bucket=bucket, Delimiter=”https://stackoverflow.com/”, Prefix=dist): if result.get(‘CommonPrefixes’) is not None: for subdir in result.get(‘CommonPrefixes’): download_dir(client, resource, … Read more
There are now 3 ways to get this done: via the AWS Console, via the command line, or via the s3cmd command line tool. AWS Console Instructions This is now the recommended solution. It is straight forward, but it can take some time. Log in to AWS Management Console Go into S3 bucket Select all … Read more
I’m not sure if you have already solved it. I had the same issue. It’s because AWS S3 is looking for that folder (projects) to serve from, which you don’t have. Simply point Error document to index.html. This worked for me. You can handle error pages in react-router.
I think what you want is Browser-Based Uploads Using POST. Basically, you do need server-side code, but all it does is generate signed policies. Once the client-side code has the signed policy, it can upload using POST directly to S3 without the data going through your server. Here’s the official doc links: Diagram: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingHTTPPOST.html Example … Read more
After two days of debugging, I finally discovered the problem… The key I was assigning to the object started with a period i.e. ..\images\ABC.jpg, and this caused the error to occur. I wish the API provides more meaningful and relevant error message, alas, I hope this will help someone else out there!
There is no concept of folders or directories in S3. You can create file names like “abc/xys/uvw/123.jpg”, which many S3 access tools like S3Fox show like a directory structure, but it’s actually just a single file in a bucket.
You’ll just need two models. One for the Post and the other would be for the Images. Your image model would have a foreignkey to the Post model: from django.db import models from django.contrib.auth.models import User from django.template.defaultfilters import slugify class Post(models.Model): user = models.ForeignKey(User) title = models.CharField(max_length=128) body = models.CharField(max_length=400) def get_image_filename(instance, filename): title … Read more
Wrap the S3 upload() function with the node.js stream.PassThrough() stream. Here’s an example: inputStream .pipe(uploadFromStream(s3)); function uploadFromStream(s3) { var pass = new stream.PassThrough(); var params = {Bucket: BUCKET, Key: KEY, Body: pass}; s3.upload(params, function(err, data) { console.log(err, data); }); return pass; }