Me and my team are using a shared hosting service with a limited linux container (without root and privileged user) and we need to develop a new feature that involves long running tasks (> 600ms).
We thought of two possible solutions:
Breaking apart the task, and via the frontend, make one separate http request to the server.
Use bash screen to run a bash script with a infinite loop calling php artisan schedule:run (mimicking cronjob)
I don't feel very confortable with the first solution, moving server logic to the browser seens wrong in my opinion.
The second solution is only a supposition (not tested), we are not sure if the bash screen would randomly stop at any time.
What would be the least unstable way to achive our goal? Thx
Assuming you already explored this because you mention that a CRON would not be an option, but even unprivileged users can setup a CRON, which is the simplest solution in combination with the Laravel scheduler.
If an actual CRON using the scheduler is really out of the question I do think making an HTTP endpoint you could call from the browser is the next best thing. Just don't think an endpoint you can call from a browser that you can only call it from a browser ;)
Take for example https://www.easycron.com/ (no affiliation but the first Google result). You can setup a CRON job there to call a URL to trigger those tasks on a CRON interval. Internally at my company called the "poor mans CRON" :)
I would agree that running a "screen" session is the most unreliable since on a server reboot those are not started again and if you "infinite loop" crashes it will not automatically restart.
If you go that route (or any CRON route) you can add some monitoring using for example https://healthchecks.io/ (again no affiliation, Google). This allows you to define a CRON schedule and gives you a URL to call after the CRON finishes, if your CRON does not call that URL according to the CRON schedule you will be notified. Good to have as insurance.
Related
i have to do a reptitive task in nodeJS and i've seen there is existing package like this one.
https://www.npmjs.com/package/node-cron
and the actual platform where i'm hosted propose inside cronjob.
https://www.netlify.com/docs/webhooks/
so my question is when it's more interessant to use the platform or a package.
thanks.
From the URL posted i didn't see any method of setting up a cron job using webhooks. Unless you were thinking of setting up a webhook that listens for a post which is sent using a linux cron job or the like?
Regardless, the actual question about using a platform or a package. They have pros and cons, but based purely on your question I would go with the platform.
If you choose to use a package you will have to write the code to call the package (which you need to test, maintain and run). You need to ensure that the node process is always up and running, if it dies or exits that it is re-spawned, that if the operating system reboots the node process gets kicked off again. All these problems are can be easily solved (PM2 for instance) but the fact is you need to think of the problems and solve them yourself or the cron job might not run when you want it to.
When using the platform you know that it is well tested, that it will work as documented, and that it will be resilient to failure modes that you might not be aware of.
We are using a dedicated Amazon Ubuntu ec2 instance as Cron server, which executed 16 cron jobs at different time intervals i.e, 10 cron jobs in morning 4:15 - 7:15 and the rest # 23:00 - 23:50. I get the results via email. I want to configure something, which shoots email message at the end of they day listing the cron jobs that are executed successfully and the one that failed.
I have a jenkins configured ubuntu instance for auto-building Dev, Beta, Staging & Live environments. Can i add these cron jobs(shell scripts) as external jobs in the jenkins and monitor them. Is it possible?
Definitely possible! You can monitor external cron jobs as described here:
https://wiki.jenkins-ci.org/display/JENKINS/Monitoring+external+jobs
You can also add cron job (-like behavior) to Jenkins by creating a freestyle software project and add "Execute shell" as build process.
It's a bit more convenient since you can also trigger the execution via Jenkins ("Build now").
You might be able to combine the Jenkins monitor external job project type with a matrix project. At the very least the former will enable you to monitor the cron jobs individually.
Alternatively you could have the last monitored cron job of the day trigger building a project that checks the status of all the cron jobs (for example by retrieving and comparing the build numbers of the last and the last successful builds) and sends an email accordingly. The email plugin might be useful for the latter.
Check the CPAN or do some web digging for shell or perl script for managing cron jobs and extend its behaviour to do some reporting which you can render using HTML. Alternatively write a servlet and a some function calls to do just that.
This becomes your own standalone monitor application, which can sit in jenkins or deployed independently. If you choose to add it to jenkins, then add the reporting HTML file and its scripts to the container holding deployed web files for jenkins, word of advice place your files and script in a separate container.
Add a hyperlink to jenkins index html which will load your reporter. Now reboot tomcat and go from there.
Another option could be to take a look at Cronitor (https://cronitor.io). It basically boils down to being a tracking beacon that uses http requests to ping when a cron job/scheduled task starts and ends.
You'll be notified if your job doesn't run on schedule, or if it runs for too long/too short, etc. You can also configure it to send alerts to you via email, sms, but also Slack, Hipchat, Pagerduty and others.
I want to create a website application, that will allow our members to get text message/email alerts every day 1 hour before their lesson (a reminder).
My server-side language background is strictly in PHP (although I've tampered some c++ back in the day). For this to work, obviously, I'll needs to somehow run a program constantly on my server.
Can this be accomplished in PHP?
If yes, is php efficient at this?
If not, how can I do this?
Or, maybe, this an entirely wrong approach, and there's a better way of creating such a service.
yes, u can consider make PHP as a daemon
or check this out php execute a background process
or simply use cron - http://en.wikipedia.org/wiki/Cron
but you should NOT create a web service/application just to run background PHP processes, it should cater for complex job
Sure, you can use PHP as a scripting language on the server. It's just as capable as any other.
Write a PHP script that checks your database for what members need to be alerted, then sends the message. Add a crontab to run this script every minute/hour/whatever. To run a php script from the command line, you run the php interpreter and give it the script name to run.
$ php /path/to/script.php
You would have to start a service on the server itself or create a CRON job to run at any given interval. If you don't have admin privileges you will have to do a CRON job, which can usually be setup in your host's cpanel.
For instance, you could create a small PHP script that
1) Searched for all lessons that start in the hour proceeding the current hour. So if the script is run at 5pm it would search for lessons that start between 6pm and 6:59.
2) Send an email to those members.
It wouldn't be exactly 1 hour though.
I have been using Quartz for over a year to schedule jobs in my app. I now have a new requirement to get rid of quarts, and instead use crontab. I have no experience in this area. How do I design something in my app that allows a job to be called by the operating system? Do I need to create a URL for each job that the OS can call?
you could define simple rest or soap services (using apache-cxf or axis2 for example) and call the from a ruby- or pythonscript
My company makes CloudQuartz (www.thecloudblocks.com) which allows you to schedule the jobs through an API and get callbacks when they are due to run.
It might be useful to what you're trying to do.
The situation is as follows:
A series of remote workstations collect field data and ftp the collected field data to a server through ftp. The data is sent as a CSV file which is stored in a unique directory for each workstation in the FTP server.
Each workstation sends a new update every 10 minutes, causing the previous data to be overwritten. We would like to somehow concatenate or store this data automatically. The workstation's processing is limited and cannot be extended as it's an embedded system.
One suggestion offered was to run a cronjob in the FTP server, however there is a Terms of service restriction to only allow cronjobs in 30 minute intervals as it's shared-hosting. Given the number of workstations uploading and the 10 minute interval between uploads it looks like the cronjob's 30 minute limit between calls might be a problem.
Is there any other approach that might be suggested? The available server-side scripting languages are perl, php and python.
Upgrading to a dedicated server might be necessary, but I'd still like to get input on how to solve this problem in the most elegant manner.
Most modern Linux's will support inotify to let your process know when the contents of a diretory has changed, so you don't even need to poll.
Edit: With regard to the comment below from Mark Baker :
"Be careful though, as you'll be notified as soon as the file is created, not when it's closed. So you'll need some way to make sure you don't pick up partial files."
That will happen with the inotify watch you set on the directory level - the way to make sure you then don't pick up the partial file is to set a further inotify watch on the new file and look for the IN_CLOSE event so that you know the file has been written to completely.
Once your process has seen this, you can delete the inotify watch on this new file, and process it at your leisure.
You might consider a persistent daemon that keeps polling the target directories:
grab_lockfile() or exit();
while (1) {
if (new_files()) {
process_new_files();
}
sleep(60);
}
Then your cron job can just try to start the daemon every 30 minutes. If the daemon can't grab the lockfile, it just dies, so there's no worry about multiple daemons running.
Another approach to consider would be to submit the files via HTTP POST and then process them via a CGI. This way, you guarantee that they've been dealt with properly at the time of submission.
The 30 minute limitation is pretty silly really. Starting processes in linux is not an expensive operation, so if all you're doing is checking for new files there's no good reason not to do it more often than that. We have cron jobs that run every minute and they don't have any noticeable effect on performance. However, I realise it's not your rule and if you're going to stick with that hosting provider you don't have a choice.
You'll need a long running daemon of some kind. The easy way is to just poll regularly, and probably that's what I'd do. Inotify, so you get notified as soon as a file is created, is a better option.
You can use inotify from perl with Linux::Inotify, or from python with pyinotify.
Be careful though, as you'll be notified as soon as the file is created, not when it's closed. So you'll need some way to make sure you don't pick up partial files.
With polling it's less likely you'll see partial files, but it will happen eventually and will be a nasty hard-to-reproduce bug when it does happen, so better to deal with the problem now.
If you're looking to stay with your existing FTP server setup then I'd advise using something like inotify or daemonized process to watch the upload directories. If you're OK with moving to a different FTP server, you might take a look at pyftpdlib which is a Python FTP server lib.
I've been a part of the dev team for pyftpdlib a while and one of more common requests was for a way to "process" files once they've finished uploading. Because of that we created an on_file_received() callback method that's triggered on completion of an upload (See issue #79 on our issue tracker for details).
If you're comfortable in Python then it might work out well for you to run pyftpdlib as your FTP server and run your processing code from the callback method. Note that pyftpdlib is asynchronous and not multi-threaded, so your callback method can't be blocking. If you need to run long-running tasks I would recommend a separate Python process or thread be used for the actual processing work.