Home Ask Login Register

Developers Planet

Your answer is one click away!

Sthe February 2016

AWS S3 large file reverse proxying with golang's http.ResponseWriter

I have a request handler named Download which I want to access a large file from Amazon S3 and push it to the user's browser. My goals are:

  • To record some request information before granting the user access to the file
  • To not buffer the file into memory too much. Files may become too large.

Here is what I've explored so far:

func Download(w http.ResponseWriter, r *http.Request) {

    sess := session.New(&aws.Config{
        Region:             aws.String("eu-west-1"),
        Endpoint:           aws.String("s3-eu-west-1.amazonaws.com"),
        S3ForcePathStyle:   aws.Bool(true),
        Credentials:        cred,
    })

    downloader := s3manager.NewDownloader(sess)
    // I can't write directly into the ResponseWriter. It doesn't implement WriteAt. 
    // Besides, it doesn't seem like the right thing to do.
    _, err := downloader.Download(w, &s3.GetObjectInput{
        Bucket: aws.String(BUCKET),
        Key: aws.String(filename),
    })
    if err != nil {
        log.Error(4, err.Error())
        return
    }

}

I'm wondering if there isn't a better approach (given the goals I'm trying to achieve).

Any suggestions are welcome. Thank you in advance :-)

Answers


fl0cke February 2016

If the file is potentially large, you don't want it to go trough your own server. The best approach (in my opinion) is to have the user download it directly from S3.

You can do this by generating a presigned url:

func Download(w http.ResponseWriter, r *http.Request) {

    ...

    sess := session.New(&aws.Config{
        Region:             aws.String("eu-west-1"),
        Endpoint:           aws.String("s3-eu-west-1.amazonaws.com"),
        S3ForcePathStyle:   aws.Bool(true),
        Credentials:        cred,
    })

    s3svc := s3.New(sess)
    req, _ := s3svc.GetObjectRequest(&s3.GetObjectInput{
        Bucket: aws.String(BUCKET),
        Key: aws.String(filename),
    })

    url, err := req.Presign(5 * time.Minute)
    if err != nil {
        //handle error
    }

    http.Redirect(w, r, url, http.StatusTemporaryRedirect)
}

The presigned url is only valid for a limited time (5 minutes in this example, adjust to your needs) and takes the user directly to S3. No need to worry about downloads anymore!

Post Status

Asked in February 2016
Viewed 3,923 times
Voted 10
Answered 1 times

Search




Leave an answer


Quote of the day: live life