I am using SailsJS for a web application which basically lets the user download and process any video from the internet(say youtube). The user enters a link to the video and my sails app downloads the video if available and then starts processing the downloaded video using a shell script(Come OpenCV processing to find different frames).
This process takes a very long time to complete, and the user can navigate away from the page and do whatever he wants. Now, to check on the progress by visiting this page later I need to be able to connect with the child process that was created earlier for this video file.
I have come up with two possible solutions:
1) Using gearman to implement a job server and connect to it every time the user navigates to the page and start getting the callback events and show the progress based on them. This is the first time I'll be using gearman.
2) Somehow storing the processID of the child process in the session/db and then using it to find the process using ps-node.
Which of these is the better approach(if you think they'll work fine)? Or is there any other solution I don't know about? Any pointers in the right direction will be appreciated.
Let's start with the second option. Don't use it. Simply because this way your site users will have sort of more control over the number of processes running on your server then you will.
Number one is way better, but using a separate job server seems like a bit of overkill to me (I have to admit though that I'm not fully informed the scale of your plans).
Bottom line, I would use a message/job queue (kue seems like a perfect fit to me) and store the progress in DB or (preferably) Redis (or whatever cache you are using).
Related
Hey guys after all these year I can finally get into node.js and react.js so i'm a total beginner.
My first project will be to migrate a time booking system for our production from the current Qt C++ application to a web application.
The biggest issues at this point is to decide how to build the backend.
The application needs to handle several concurrent users with full duplex communication between the client sides.
The closes example would be a system like Trello. So a collaborative system where multiple users are able to move tasks around.
Once a user moves a task from one spot to another i have to instantly reorganize the tasks for every currently logged in user. So i can guarantee that everyone is up to date at any point in time.
How would you build this and what libraries are best suited for this case?
I hope you understand what I'm trying to say.
greets
Use websocket to keep server and client side in touch.
To prevent operation conflict, you need to
lock the task/resource when user start doing edit/move, and inform other clients that it has been locked.
write result into db/nosql.
inform all other user client change result and get ACK response.
release the lock and inform all clients.
to smooth the process, release lock when getting the very first ACK.
A change log with timestamp (monotonic time) might be helpful to prevent conflict.
Besides, Electron (based on NodeJs) is suitable for building client side as an alternative to Qt.
I am new to mobile website development, and facing this issue where I want to refresh data on the website in every 30 sec which is invoked from the client side and server provides the data in response. Problem is when I close the browser or when the browser goes in background it stops working. Is there any thing we can do to make this thing possible?
Have a look at the Android Developers - Processes and Threads guide. You'll get a deeper introduction to how process life-cycles work and what the difference is between the states for background- and foreground processes.
You could embed your web app in a WebView. This way you could deal with the closing browser case: you could provide a means to "exit" the app that involves closing only your container activity. That way the timers you have registered in javascript will still be running in the 'WebViewCoreThread'. This is an undesirable behavior and a source of problems, but you can take advantage of it if you want (just make sure you don't run UI-related code there). I've never tested this in Kit Kat (which uses a different WebView based on Chrome) but works for previous versions, as I described here.
Now the user can always close any app. Even without user interaction, the OS can kill your app on low memory. So just give up on long-running apps that never end, because the OS is designed in such a way this is simply not possible.
You could go native and schedule Alarms using the AlarmManager.
Just checked this out on the Android KitKat WebView and as per Mister Smith's comments the javascript will continue executing in the background until the Activity is killed off:
Just tested with this running in a WebView:
http://jsbin.com/EwEjIyaY/3/edit
My gut instinct is that if the user has moved your application into the background, there seems little value in performing updates every 30 seconds, it makes more sense to just start updating again once the user opens the device up and cache what information you currently have available to you.
As far as Chrome for Android goes the same is happening, as Chrome falls into the background the javascript is still running.
If you are experiencing different behaviour then what exactly are you seeing and can you give us an example?
Team,
i'm developing an iOS application.
My requirement is to query for specific news service(REST API) in regular time interval.I wanted query the service twice for a day and update my sqllite db, even the applciation is in background state. My UI will be updated with data fetched from sqllite db, while the application is in foreground.
My question are,
Is it possible to run NSTimer in background continuously? if yes, is
there any maximum time limit for timer to run in background (say 10
mins or 60 mins)?
Is it possible to send request to download a file using
NSUrlConnection and save the file to documents directory, when the
application is in background ?
Your suggestions will be much helpful for my project design.
Thanks in advance.
What you are aiming for cannot be achieved on iOS:
Arbitrary apps cannot run in the background for an arbitrary amount of time.
You can try to mitigate some of this by using local notifications instead of NSTimer to schedule your updating. This will, however, only buy you a very limited amount of time to do your networking.
The question you should ask yourself at this point probably is:
If you are only updating twice a day, how bad can it be to initiate the download when your app becomes active?
Answering my own question, so that it will be helpful for others.
Ques 1: Is it possible to run NSTimer in background continuously?
Ans: Nstimer will not run while the application in background state. So there is no point of maximum allowed timer value in background. If the application enters into background while there is an ongoing process, [UIApplication beginBackgroundTaskWithExpirationHandler:] can be used to complete the ongoing process. The maximum time allowed by the OS with this handler is 10mins.
Ques 2: Is it possible to send request to download a file using NSUrlConnection and save the file to documents directory, when the application is in background ?
Ans:
Below given information is from Apple documentation. Detail info is found here
In iOS, only specific app types are allowed to run in the background:
Apps that play audible content to the user while in the background,such as a music player app
Apps that keep users informed of their location at all times, such as a navigation app
Apps that supportVoice over Internet Protocol (VoIP)
Newsstand apps that need to download and process new content
Apps that receive regular updates from external accessories
Info about running background process using VOiP type application can be found here
I'm writing a nodejs daemon for connecting in the background to Twitter, Facebook and other social networks. Every user can have multiple social networks connected to it's account. And say, when every user has somewhere around 5 networks connected and there are 1000 users. That would be 5000 connections. The connections have to be near realtime, like somewhere around 1 minute or something.
My idea was to start a new nodejs process for every new user that pulls in the content from those social networks instead of 1 process that pulls in every data from every user.
But my question is how to start. I allready wrote some of the script to do this, but since my knowledge of nodejs is nowhere near professional I don't know how this works.
Is there a way to automatically start a new nodejs process when a new user registers? Or is there a better way to do this?
Thanks in advance.
You want 1k processes? That's already up to 1gig overhead and a ton of work the OS.
Why not have all inside one Node.js process? Have you actually tested it? Are you sure you need multiple processes? If you want you can use the child_process module to start new child processes from within Node.js.
For keeping the main process running you should take a look at forever.
Still I would say test it with one process first, and then if you see that it doesn't perform/work as you expected it to, it should be trivial to put the stuff into multiple processes.
Even with a poor network connection?
Specifically, I've written code which launches a separate thread (from the UI) that attempts to upload a file via HTTP POST. I've found, however, that if the connection is bad, the processor gets stuck on outputstream.close() or httpconnection.getheaderfield() or any read/write which forces data over the network. This causes not only the thread to get stuck, but steals the entire processor, so even the user interface becomes unresponsive.
I've tried lowering the priority of the thread, to no avail.
My theory is that there is no easy way of avoiding this behavior, which is why all the j2me tutorial instruct developers to create a ‘sending data over the network…’ screen, instead of just sending everything in a background thread. If someone can prove me wrong, that would be fantastic.
Thanks!
One important aspect is you need to have a generic UI or screen that can be displayed when the network call in background fails. It is pretty much a must on any mobile app, J2ME or otherwise.
As Honza said, it depends on the design, there are so many things that can be done, like pre-fetching data on app startup, or pre-fetching data based on the screen that is loaded (i.e navigation path), or having a default data set built in into the app etc.
Another thing that you can try is a built-in timer mechanism that retries data download after certain amount of time, and aborting after say 5 tries or 1-2 minutes and displaying generic screen or error message.
Certain handsets in J2ME allow detection of airplane mode, if possible you can detect that and promptly display an appropriate screen.
Also one design that has worked for me is synchronizing UI and networking threads, so that they dont lock up each other (take this bit of advice with heavy dose of salt as I have had quite a few interesting bugs on some samsung and sanyo handsets because of this)
All in all no good answer for you, but different strategies.
It pretty much depends on how you write the code and where you run it. On CLDC the concept of threading is pretty limited and if any thread is doing some long lasting operation other threads might be (and usualy are) blocked by it as well. You should take that into account when designing your application.
You can divide your file data into chunks and then upload with multiple retries on failure. This depends on your application strategy . If your priority is to upload a bulk data with out failure. You need to have assemble the chunks on server to build back your data . This may have the overhead for making connections but the chance is high for your data will get uploaded . If you are not uploading files concurrently this will work with ease .