Direct File Uploads to S3: Part 3 Discussion
Awesome as usual Chris.
How similar do you think this process would be to using Carrierwave Direct?
The main difference is that with CarrierWaveDirect you generate the HTML form with fields for S3 request parameters, which you do while rendering the page, while with Shrine the JavaScript requests the S3 request parameters from your app dynamically (in JSON format) when a file is attached.
Among other things, this allows you to do multiple file uploads with Shrine, because an S3 presign can be requested for each selected file. With CarrierWaveDirect multiple uploads aren't really possible, because it can only generate HTML forms, it doesn't enable you to return these S3 request parameters in JSON format so that the JavaScript can just make the AJAX S3 request(s) itself.
Chris, can you cover how to upload multiple videos to S3 with progress bars (using Shrine direct upload) + AWS transcoding to web & mobile format after the record has been created in database using ActiveJob ie Sidekiq?
Chris any plans to do a 4th part of this series to add background processing with ActiveJob for large uploads (or multiple uploads) specifically on Heroku?
Thanks Chris. This has been really good. I'd like to learn more about what I can do with Shrine.
Hi Chris, I'm trying to get this working and following along with your files. I didn't do your final step of converting from JSON to js (so I can't copy/paste your source code file). However, when I try to use what we made with your video, I get js errors showing in the console. After I put the code in JSHint, it tells me to add semi-colons on the end of some of the lines, and an extra ")" on the end of the second last line (so it has 2 "))"), I then get an error that says: uploads.self-a6843c1….js?body=1:2 Uncaught TypeError: "[type=file]".fileupload is not a function(…). I'm not very good at understand js or the console output. Can you see if there might be a problem with that line? Thanks a lot :)
If you're getting the fileupload is not a function, then maybe jquery fileupload is not included in your application.js file?
Scott
Great episode! Just curious what are the advantages are of using Shrine vs. direct S3 upload described by Heroku: https://devcenter.heroku.co... ? Thanks!
It's the exact same process if you look into what they do. I recommend Shrine because then you can take advantage of all of the features of it after the file is uploaded such as image processing, thumbnails, background processing, you name it.
There is a subtle, but nasty bug in the video. Because of the way the form is created in the 'done' function, you create a form with a input type=file, which rails will attempt to process by uploading it and then discarding the data before it hits the controller.
If you are uploading a small picture, you might not even notice the extra work and delay, but I was uploading a 400Mb video, which Rails dutifully attempted to process -- adding over a minute of delay before the controller would respond.
The way I solved this problem (I would love to know if there was a better way) was to create a second form on the page that was hidden (this way, I get all the rails support for security and security tokens, etc.) and I used that second form to build up my data in the done function. This form DOES NOT have an input type=file -- it just has the shrine hidden field.
Hey Hal, good find. I think I noticed it but it was subtle enough with small files I didn't catch it.
The solution should be to delete the existing file field from the FormData object before you append the jquery file upload one.
var form = $(this).closest("form");
var form_data = new FormData(form[0]);
form_data.delete($(this).attr("name")); // Remove the existing form field
form_data.append($(this).attr("name"), JSON.stringify(image)); // Add our json version
Austin Klenk
This helped me out a lot, I'm wanting to build a Rails photo gallery app for the company I work for to display all the past jobs and such, currently using Wordpress 😞 Where it takes sweet time to load all the images and such. Though anyway is there a way where you can upload multiple images at once with this uploaded? For like having a gallery with lots of images. Instead of uploading one image at a time?
Yes, you can just add "multiple" HTML attribute to the file field, which enables it to accept multiple files:
<input type="file" name="file" multiple="true">
The files will still be uploaded in individual requests (there is no performance gain in sending multiple files in a single request anyway), but the uploads will happen in parallel. The shrine-rails-example repository demonstrates this flow.
Question: This could gives time outs on Heroku if the request to "/images/upload/cache/presign" takes longer than 30 secs?
The `/images/upload/cache/presign` endpoint is instantaneous, it doesn't make any HTTP requests or anything.
Alejandro Ventura
What if I want not to send the form to Rails automatically after upload? Because I want to fill some other fields and until the users hit the Submit button I want all the fields an the file fields get stored in the DB.
How can I accomplish this?
After the file is uploaded, you can write the JSON data to the hidden attachment field (which has the same "name" attribute as the file field; it's mentioned in the "Quick Start" section of the Shrine README). Then when you submit the form, Shrine will attach the file from the JSON data. So the idea is, you can either send a new file (multipart request via the file field) or send an already uploaded file (JSON data via the hidden field) as the attachment attribute.
How to make the upload.js works for file_fields added dynamically?
After the page has fully loaded.
Dave Woodall
@excid3:disqus Solid Series Sir! (S3 joke) But seriously, this was very timely and I look forward to digging into Shrine. One question - I'm noticing that images are being uploaded to both aws directories (/store and /cache). I can't tell if cache eventually goes away or this is a bug?
Hi @fakefarm:disqus. I had the same question. I found a discussion here: https://github.com/janko-m/... that suggests two options.
1. Use a Shrine plugin called 'moving' to delete the cached S3 file immediately after it is moved to the store bucket.
http://shrinerb.com/rdoc/cl...
I think you would include this line 'Shrine.plugin :moving, storages: [:store]' in 'config/initializers/shrine.rb'
2. Amazon can be set up to delete files in the cache bucket periodically, documentation here: http://docs.aws.amazon.com/...
@excid3:disqus, My question is... Why does Shrine upload to a 'cache' bucket first and then copy it to a 'store' bucket? I haven't been able to find an answer online yet. If you could explain why Shrine or S3 works this way or post a link it would be greatly appreciated!
Chris! Just wanted to say that this series is beautifully done. Thank you so much for your hard work and attention to detail. I love how you explain how everything works the hard way or even the wrong way first which helps us understand how the final implementation decisions were made. It took me two full days to get through it all, but everything is working for me. Excellent. Awesome! Thanks again.
@excid3:disqus @jankomarohnic:disqus I have everything working beautifully here, but when I run my tests I get: uninitialized constant ImageUploader::UploadEndpoint (NameError)
I found this discussion recommending putting the uploader class in the app/uploaders directory, but am still getting the same error.
https://groups.google.com/f...
Any ideas? Thank you so much for your help.
Hi Chris, nice series. I almost done but I'm having an issue. Everything was working fine but after place all javascript code I'm not able to upload the image to AS3. I'm having this error: "Failed to load resource: the server responded with a status of 400 (Bad Request)". I think could be something related with the url and the presing. I checked all the code and google about it, but I can't find a solution.
Here is my repo, I send it because this code is based on your first Shrine video (where you created a post with description and image): https://github.com/carlos99...
https://uploads.disquscdn.c...
Thanks in advance for any help @Chris Oliver
Hi Chris, I followed your series to do the direct upload, which worked for me locally, everything works fine, things are nice. But when i deployed my code to server(eb), direct upload stopped working. No error in the console, it just does not reach uploads.js is what I think, but the normal upload works, I can see the file in S3. As there is no error I cannot figure out what the problem is. Any suggestions? Thank you.
Any ideas on how to go about this without Jquery and/or jQuery file upload?
Hey Chris, do you have a tutorial for implementing Active Storage, Trix and S3 for images for Rails 6? Trying to follow these videos but am running into some issues I cannot find solutions online. Currently the two issues I cannot get past are errors running local server for:
Shrine.plugin :direct_upload
and
mount ImageUploader::UploadEndpoint => "/images/upload"
Looks like the issue was I was using a newer version of Shrine. This version works:
gem 'shrine', '~> 2.3.1'