How do I access files on server with a cron or similar ? (outside of app)
Hi everyone, i'm a bit new to rails and i'm searching for help.
I have a rails app where I receive a zipfile on my server (outside of the app, homedir of new FTP user).
Then I want to unzip the files (xml and img) and do my things to update my database with the xml file and moves images to the right directories.
I already coded all of that, and this works perfectly, but today I have to put the zipfiles on the /public directory. To make this work, I created a route to a controller:action. I just have to call my url.
Now, I want to make this automated with cron.
Problem is : my files can't be in public directory of my app (ftp user for zip/xml don't have rights to put it there)
And I have no idea where to start and how I can do that. Anyone can help me with this ?
PS : I use hatchbox to deploy my app.
Here's a part of my code :
I have a rails app where I receive a zipfile on my server (outside of the app, homedir of new FTP user).
Then I want to unzip the files (xml and img) and do my things to update my database with the xml file and moves images to the right directories.
I already coded all of that, and this works perfectly, but today I have to put the zipfiles on the /public directory. To make this work, I created a route to a controller:action. I just have to call my url.
Now, I want to make this automated with cron.
Problem is : my files can't be in public directory of my app (ftp user for zip/xml don't have rights to put it there)
And I have no idea where to start and how I can do that. Anyone can help me with this ?
PS : I use hatchbox to deploy my app.
Here's a part of my code :
def unzip_files_cron
require 'zip'
FileUtils.rm_rf(Dir.glob('public/cron_images/*'))
File.delete('public/cron/annonces.xml') if File.exist?('public/cron/annonces.xml')
Zip::File.open('public/cron/annonces.zip') do |zipfile|
zipfile.each do |f|
if f.name == 'annonces.xml'
f_path=File.join("public/cron/", f.name)
else
f_path=File.join("public/cron_images/", f.name)
end
FileUtils.mkdir_p(File.dirname(f_path))
zipfile.extract(f, f_path) unless File.exist?(f_path)
end
end
File.delete('public/cron/annonces.zip') if File.exist?('public/cron/annonces.zip')
end
Hey LinaeAzn,
Will the file name always be the same? If so, you could create a symbolic link to the file to your public directory... so something like:
Will the file name always be the same? If so, you could create a symbolic link to the file to your public directory... so something like:
ln -s /home/path/to/file.ftp /path/to/rails/public/
You could also designate a folder in their home directory and then just symlink that directory instead and access any files placed in that directory. Just mind your security since you are putting it in the public directory.
Thanks for the answer,
I'm not really good with servers, but I achieved to put homedir of the restricted FTP user in my deployed app.
I managed to get my files outside of my public directory (root of my app) because of what you said on security.
Then I change the paths of file with :
I'm not really good with servers, but I achieved to put homedir of the restricted FTP user in my deployed app.
I managed to get my files outside of my public directory (root of my app) because of what you said on security.
Then I change the paths of file with :
Rails.root + 'cron/files/...'
This seems to work on url call, which is a good point.
Now I have to install the "whenever" gem to get this cron running for the first time and delete my route to remove access by url.
Do not hesitate to tell me if I'm doing some bad rails design/workflow.
Thank you Jacob for pointing me in the right direction !
Now I have to install the "whenever" gem to get this cron running for the first time and delete my route to remove access by url.
Do not hesitate to tell me if I'm doing some bad rails design/workflow.
Thank you Jacob for pointing me in the right direction !
No problem, we all start somewhere :)
I achieved to put homedir of the restricted FTP user in my deployed app
Do you mean you physically moved the users FTP folder into your rails app folder? Or did you just create the symlink?
If you moved it, then I don't think I'd go that route as the symlink method will give you the exact same access without the negative side effects of moving a user's home directory inside your app.
Let's say the files you're trying to access are located here:
/home/user1/uploads
And you want your app to access the zip file here:
/var/www/your_rails_app/cron/files
Then your symlink command that you'd type in the terminal (not rails console) would look like this:
ln -s /home/user1/uploads /var/www/your_rails_app/cron/files
If you haven't watched it already, Chris has a wonderful video on the Whenever gem you should check out: https://gorails.com/episodes/cron-jobs-with-whenever-gem
Hi,
I finally took the time to makes all the changes this weekend.
I've watched the video on the whenever gem, and managed to make my cron "working".
But i ran into issues :
With your advices I moved back the homedir of the user outside of my app and created the symlink.
1. As I said before, I use hatch from Chris to deploy my app, but the symlink is not "following" the current release.
With :
I finally took the time to makes all the changes this weekend.
I've watched the video on the whenever gem, and managed to make my cron "working".
But i ran into issues :
With your advices I moved back the homedir of the user outside of my app and created the symlink.
1. As I said before, I use hatch from Chris to deploy my app, but the symlink is not "following" the current release.
With :
ln -s /home/my_restricted_user/ftp/files/ /home/deploy/my_app/current/cron/
Where "current" is already a symlink to the last release of the app.
Do you know how can I make this symlink persistent with each deploy ?
2. Even when I got the symlink right with the actual release, my cron had trouble executing. With this error :
Do you know how can I make this symlink persistent with each deploy ?
2. Even when I got the symlink right with the actual release, my cron had trouble executing. With this error :
bundler: failed to load command: bin/rails (bin/rails) Errno::EACCES: Permission denied @ unlink_internal - /home/deploy/my_app/releases/20180225184145/cron/files/my_xml_files.xml
At first I thought this was some permissions issues. So I changed the owner/group of file to deploy:deploy (same issue), then I put chmod 777 on my files (again same error message).
After that, I tried to just delete the symlink, copy the directory, and put 777 on files. And know, it is working.
Do you know how to make those symlinked files accessible from the rails app / cron ?
Thank you for all the previous answer Jacob, it helped me learned some new things !
After that, I tried to just delete the symlink, copy the directory, and put 777 on files. And know, it is working.
Do you know how to make those symlinked files accessible from the rails app / cron ?
Thank you for all the previous answer Jacob, it helped me learned some new things !
As long as the user that your app uses is a member of the same group the original files were assigned and that group has read access to the file then there's nothing more you should have to do.
So it sounds like you need to add your deploy user to the group that the FTP user is a member of and then ensure the file and directory have the proper read access for the assigned group. For this, you may consider making a special group that you assign both your deploy user and your FTP user to. Then create the directory in the FTP users home dir and ensure the folder is assigned to the new group with read access for the group, and read/write access to the FTP user.
Can you post the cron job that whenever generated for you? Just do crontab -e to list them
How are you calling your script from the cron itself? Are you using a rake task? I had a similar issue relatively recently with a temp log file not being able to write to the tmp directory... I'll have to dig through it to see if I can recall what the fix was. I think it had to do with the way I was calling it from the rails console instead of a rake task - but I may be confusing that with something else entirely.
I'm not familiar with Hatch, so maybe Chris will see this and can chime in with how to handle the rolling releases. Maybe you can set an environment variable on deploy with the release version which you could use to rebuild the symlinks? That feels kinda awkward though...
So it sounds like you need to add your deploy user to the group that the FTP user is a member of and then ensure the file and directory have the proper read access for the assigned group. For this, you may consider making a special group that you assign both your deploy user and your FTP user to. Then create the directory in the FTP users home dir and ensure the folder is assigned to the new group with read access for the group, and read/write access to the FTP user.
Can you post the cron job that whenever generated for you? Just do crontab -e to list them
How are you calling your script from the cron itself? Are you using a rake task? I had a similar issue relatively recently with a temp log file not being able to write to the tmp directory... I'll have to dig through it to see if I can recall what the fix was. I think it had to do with the way I was calling it from the rails console instead of a rake task - but I may be confusing that with something else entirely.
I'm not familiar with Hatch, so maybe Chris will see this and can chime in with how to handle the rolling releases. Maybe you can set an environment variable on deploy with the release version which you could use to rebuild the symlinks? That feels kinda awkward though...
`777` is bad news. If you're having to change perms to owner/group/world full read/write/exec something is wrong.
Totally correct jaems, but 777 is an excellent way to tell if it's a permission issue you're dealing with, which based on what LinaeAzn said, was all they were after (at least that's how I interpreted it).
Yes 777 was just to test permissions on files, without success !
I already did add the user deploy to the group of my restricted FTP user.
Here's the crontab -e
I already did add the user deploy to the group of my restricted FTP user.
Here's the crontab -e
# Begin Whenever generated tasks for: my_app at: 2018-02-25 19:04:12 +0100 PATH=/home/deploy/my_app/shared/bundle/ruby/2.3.0/bin:/home/deploy/.rbenv/versions/2.3.3/bin:/home/deploy/.rbenv/libexec:/home/deploy/.rbenv/plugins/ruby-build/bin:/home/deploy/.rbenv/plugins/rbenv-vars/bin:/home/deploy/.rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games 25 18 * * * /bin/bash -l -c 'cd /home/deploy/my_app/releases/20180225190348 && bundle exec bin/rails runner -e production '\''Bien.cron'\''' # End Whenever generated tasks for: my_app at: 2018-02-25 19:04:12 +0100
I had to include "env :PATH" because of previous errors of cron not finding rails/bundler
I do not use a rake task, but a runner calling to Model.method
And my schedule.rb looks like this :
I do not use a rake task, but a runner calling to Model.method
And my schedule.rb looks like this :
env :PATH, ENV['PATH'] every 1.day, at: '06:25 pm' do runner "Bien.cron" end
And to explain what my method do :
- clean specific files (xml) from folder
- unzip a zip file, containing xml + img
- read xml, geolocate some addresses, insert database
There has to be something wrong with the permissions or the way your cron is being executed. I just did a quick test where I created a new user foo on a droplet, created a directory and put a txt file with "hello world" in there and symlinked it to my rails root directory. Then I assigned my deploy user to the foo group and created a simple rake task to read the contents of the text file that was symlinked to the rails root directory. Everything worked as expected.
I've had issues with cron jobs when using a runner so I only use tasks now which took care of everything. Would you mind trying this out:
rails root/lib/tasks/bien.rake
I've had issues with cron jobs when using a runner so I only use tasks now which took care of everything. Would you mind trying this out:
rails root/lib/tasks/bien.rake
task bien: :environment do Bien.cron # I'm assuming this is the model/method you're trying to execute, correct? end
You can then run it by just executing rake bien from your terminal like you do rake db:migrate...
If that works, you can add it to your schedule as:
config/schedule.rb
every 1.day, at: '06:25 pm' do rake "bien" end
Be sure to double check all the permissions and group assignments, especially since you've done so much up to this point something may not have stuck.
I make baby steps.
I changed my call to a rake task as you told me.
Then ran into a really WEIRD issue (at least for me) :
I changed my call to a rake task as you told me.
Then ran into a really WEIRD issue (at least for me) :
deploy@vps:/home/my_ftp_user/ftp/files$ mv my_xml.xml my_xml.xml.old mv: cannot move 'my_xml.xml' to 'my_xml.xml.old': Permission denied deploy@vps:/home/my_ftp_user/ftp/files$ ls -l total 38632 -rwxrwxrwx 1 deploy deploy 87084 Feb 24 15:10 my_xml.xml -rwxrwxrwx 1 deploy deploy 31853133 Feb 26 23:12 my_zip.zip -rwxrwxrwx 1 my_ftp_user my_ftp_user 7614269 Feb 26 23:17 my_second_zip.zip
I can't mv/cp/rm a 777 file that got the right owner...
=> After that, I chmod 2777, then 0777 back, and got it working. I really, don't understand what just happened. (rights et owner are exactly the same now, but deploy user now have the permissions to do things on the files). So it seems to be fixed, wait and see on that one I guess.
The cron now, seems to start, but run into :
=> After that, I chmod 2777, then 0777 back, and got it working. I really, don't understand what just happened. (rights et owner are exactly the same now, but deploy user now have the permissions to do things on the files). So it seems to be fixed, wait and see on that one I guess.
The cron now, seems to start, but run into :
rake aborted! ActiveRecord::NoDatabaseError: FATAL: role "deploy" does not exist
My database is auto generated by hatch deployment, and got a username randomized, and it's not "deploy".
Tomorrow I'll try to search if there's an easy fix for this one.
For the all thing, I have the feeling that I do/did some newbish errors that it make this not as smooth as it should be ! Thanks again for the help and your time Jacob.
Tomorrow I'll try to search if there's an easy fix for this one.
For the all thing, I have the feeling that I do/did some newbish errors that it make this not as smooth as it should be ! Thanks again for the help and your time Jacob.
Check out the following page for the whole 2777 then 0777 bit... https://major.io/2007/02/13/chmod-and-the-mysterious-first-octet
Can you post your updated crontab? Does it still call -production or RAILS_ENV=production?
And can you manually run the rake task or does it give you the error there as well? How about:
Can you post your updated crontab? Does it still call -production or RAILS_ENV=production?
And can you manually run the rake task or does it give you the error there as well? How about:
rake bien RAILS_ENV=production
For the bit change, I failed to understand one thing. My files was "-rwxrwxrwx" (so 0777 if i'm not wrong) when i got permission denied. I switch to 2777 then back to 0777 without changing anything. And my files were accessible/changeable again.
This is my crontab, and yes the RAILS_ENV is there :
This is my crontab, and yes the RAILS_ENV is there :
# Begin Whenever generated tasks for: my_app at: 2018-02-27 00:49:52 +0100
PATH=/home/deploy/my_app/shared/bundle/ruby/2.3.0/bin:/home/deploy/.rbenv/versions/2.3.3/bin:/home/deploy/.rbenv/libexec:/home/deploy/.rbenv/plugins/ruby-build/bin:/home/deploy/.rbenv/plugins/rbenv-vars/bin:/home/deploy/.rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games20 01 * * * /bin/bash -l -c 'cd /home/deploy/my_app/releases/20180227004925 && RAILS_ENV=production bundle exec rake bien --silent'
Got the same error message with : "PG::ConnectionBad: FATAL: role "deploy" does not exist"
And YES (finally), calling the rake task directly from console works with :
And YES (finally), calling the rake task directly from console works with :
bundle exec rake bien RAILS_ENV=production
Hmm, it's odd that the cron job is failing, yet manually running it is succeeding.
If you execute `env` from the linux console, do you have an entry for your database username in there? It would be whatever you have the name for the environment variable in your database.yml file.
If it is in there, then manually add it to your cron with:
crontab -e
If you execute `env` from the linux console, do you have an entry for your database username in there? It would be whatever you have the name for the environment variable in your database.yml file.
If it is in there, then manually add it to your cron with:
crontab -e
export ENV_VAR_DB_USERNAME='the_username_from_env' # replace ENV_VAR_DB_USERNAME with whatever the env var is from the database.yml file # rest of your cron stuff ...
There might be a better way of getting this working - I can't help but feel we're on the verge of using brute force to get this thing to work... it may be worth shooting Chris an email over @hatchbox.io to get your cron working right with a rolling db username. Unfortunately I haven't got a chance to mess with hatchbox yet so I'm just not sure the extent of what's going on there to be able to come up with a friendly solution with your complete setup.
There's nothing related to database when I execute "env" from the linux console.
I looked into database.yml, there's a "url" var :
I looked into database.yml, there's a "url" var :
production: adapter: postgresql url: <%= ENV['DATABASE_URL'] %> encoding: utf8
In hatch, the ENV['DATABASE_URL'] looks like : "postgres://my_randomized_database_username:my_database_password@127.0.0.1/my_app"
Do I have to export this full url to my crontab ?
Yes, yesterday, I did send an email to Chris from hatch support for that and the symlinked folder between releases !
Do I have to export this full url to my crontab ?
Yes, yesterday, I did send an email to Chris from hatch support for that and the symlinked folder between releases !
I got an answer from Chris on the cron/whenever gem :
Cron jobs with Whenever run in a special shell setup and actually load ~/.bash_profile if it exists. So you actually need that configured to load rbenv so your jobs run.
Apparently, Chris fixes that hatch config on my server modifying my deploy user bash_profile. My server was installed/provisioned before he got this fix for hatch, so he had to do it manually for my VPS specificaly.
I can't test right now, but i'll update when I can confirm that all works well.
I can't test right now, but i'll update when I can confirm that all works well.
Awesome! Good to know about the cron jobs, and great to hear that it seems to be a pretty simple fix!
I look forward to the "good news" update, fingers crossed! :)
I look forward to the "good news" update, fingers crossed! :)