Gotchas migrating rails file_column to use Amazon S3

If you’re already using file_column, or thinking of migrating file_column to use S3, here’s the gotchas I’ve come across so far in migrating the photos for Grows on You, our gardening website.

Limit on file_column uploads

file_column puts all uploaded files in one directory, so if you hit 32k photos (or whatever) you hit an operating system limit and it barfs with this error…

Errno::EMLINK (Too many links -/some/file/column/storage/directory).

This means you have to delete the local files after putting them on S3 if you want to handle more than 32,000 uploads. Which leads nicely onto gotcha number 2…

file_column plugin still needs the local files (kinda)

If you move files to S3 and then start removing them locally (to free up disk space or stay below 32k files) validations that require the file to exist will fail. In our case, we have a file_column validation on filesize. Since the validation requires the local file to check file size, it errors if the file isn’t there.

To get round this, we set a flag (we called it stored_remotely) on the Photo object. This is checked in the file_column validations code and the validation is skipped if it’s set to true. We set the flag after the initial upload and we’ve confirmed it’s on S3. Which is where gotcha numer 2 blends seemlessly into gotcha number 3…

S3 gets confused over plus signs in keys

Trying to confirm an object was stored to S3 using S3Object.find fails if the key you used had a ‘+’. Say you you have an S3 object with a ‘+’ in the key; if you try to do an S3Object.find for an object with ‘+’ in the key, it fails. However, it finds the object if you replace the ‘+’ with a space. When you list objects, the returned keys also has spaces instead of any plus signs. To work around you need to replace the ‘+’ with a space.

Here’s the weird thing: If the object is an image and you’re serving it directly from S3, having plus signs in the uri works just fine, for example:

<img src =”http://bucketname.s3.amazonaws.com/key+name_with_a_plus.jpg”/&gt; serves the image just fine, but…

AWS::S3::S3Object.find(http://bucketname.s3.amazonaws.com/key+name_with_a_plus.jpg , bucket_name) fails to fnd the object but it does find the object using a key name of “key name_with_a_plus.jpg”

Hope this saves you some time and hair pulling.

Advertisements

About this entry