I am in the process of creating a site which enables users to upload audio. I just figured our how to use ffmpeg with PHP to convert audio files (from WAV to MP3) on the fly.
I don't have any real experience with ffmpeg and I wanted to know what's the best way to convert the files. I'm not going to convert them upon page load, I will put the conversions in a queue and process them separately.
I have queries about how best to process the queue. What is a suitable interval to convert these files without overloading the server? Should I process files simultaneously or one by one? How many files should I convert at each interval to allow the server to function efficiently?
Server specs
Core i3 2.93GHz
4GB RAM
CentOS 64-bit
I know these questions are very vague but if anyone has any experience with a similar concept, I would really love to hear what works for them and what common problems I could face in the road ahead.
Really appreciate all the help!
I suggest you use a work queue like beanstalkd. When there is a new file to convert simply place a message into the queue (the filename maybe). A daemon that works as beanstalkd client fetches the message and converts the audio file properly (the daemon can be written in any language that has a beanstalkd library).
Related
Unfortunately, I'm not really familiar with Linux and Docker. But maybe someone from you can help me.
I need a lightweight Docker image that can do multiple byte stream conversions (e.g. encoding, encryption, compression, ...) in parallel. The process should be called several times outside of the image (e.g. from another image). If I understand it correctly, I need some kind of server that receives and processes the requests.
How do you implement this and how would the program then be called?
Many Thanks
I am running a webservice to convert ODT documents to PDF using OpenOffice on an Ubuntu server.
Sadly, OpenOffice chokes occasionally when more then 1 request is made simultaneously (converting a PDF takes around 500-1000ms). This is a real threat since my webservice is multithreaded and jobs are mostly issued in batches.
What I am looking for is a way to hand off the conversion task from my webservice to a intermediate process that queues all requests and streamlines them 1 by 1 to OpenOffice.
However, sometimes I want to be able to issue a high priority conversion that gets processed immediately (after the current one, if busy) and have the webservice wait (block) for that. This seems a tricky addition that makes most simple scheduling techniques obsolete.
What you're after is some or other message/work queue system.
One of the simplest work queueing systems I've used, that also supports prioritisation, is beanstalkd.
You would have a single process running on your server, that will run your conversion process when it receives a work request from beanstalkd, and you will have your web application push a work request onto beanstalkd with relevant information.
The guys at DigitalOcean have written up a very nice intro to it here:
https://www.digitalocean.com/community/tutorials/how-to-install-and-use-beanstalkd-work-queue-on-a-vps
I want to create a tool, with which we can administer the server.There are two questions with in this question:
To administer access/hit rate of a server. That is to calculate how many times the server has been accessed from a particular time period and then may be generate some kind of graph to demonstrate the load at a particular time on a particular day.
However i don't have any idea, how i can gather these information.
A pretty vague idea is to
use a watch over access log(in case of apache) and then count the number of times the notification occurs and note down the time simultaneously
Parse access.log file every time and then generate the output(but access.log file can be very big, so not sure about this idea)
I am familiar with apache and hence the above idea is based on apache's access log and i don't have idea about other like nginx etc.
Hence i would like to know, if I can use the above procedure or is there any other way possible.
I would like to know when the server is reaching its limit. The idea of using top and then show the live result of cpu usage and ram usage via CPP
To monitor a web server the easiest way is probably to use some existing tool like webalizer.
http://www.webalizer.org/
To monitor other things like CPU and memory usage I would suggest snmpd together with some other tool like mrtg. http://oss.oetiker.ch/mrtg/
If you think that webalizer does not sample data often enough with its hourly statistics but the sample time of mrtg with 5 minutes would be better it is also possible to provide more data with snmpd by writing an snmpd extension. Such an extension could parse the apache log file with a rather small amount of code and give you all the graphing functionality for free from mrtg or some other tool processing snmp data.
I've been looking for a decent example to answer my question but, not sure if its possible at this point.
I'm curious if its possible to upload a image or any file and stream it so separate server? In my case I would like to stream it to imgur.
I ask this because I don't want to reduce the bandwidth hit for all the files to come to the actual nodejs server and upload it from there. Again, I'm not sure if this is possible or if I'm reaching but, some insight or an example would help a lot.
Took a look at Binary.js which may do what I'm looking for but, its IE10+ so no dice with that...
EDIT: based on comment on being vague
When the file is uploaded to the node server, it takes a bandwidth hit. When node takes the file and upload it to the remote server, it takes another hit. (if I'm not mistaken) I want to know it its possible to pipe it to the remote service(imgur in this case) and just used the node server as a liaison. Again, I'm not sure if this is possible which is why I'm attempting to articulate the question. I'm attempting to reduce the amount of bandwidth and storage space used.
Even with a poor network connection?
Specifically, I've written code which launches a separate thread (from the UI) that attempts to upload a file via HTTP POST. I've found, however, that if the connection is bad, the processor gets stuck on outputstream.close() or httpconnection.getheaderfield() or any read/write which forces data over the network. This causes not only the thread to get stuck, but steals the entire processor, so even the user interface becomes unresponsive.
I've tried lowering the priority of the thread, to no avail.
My theory is that there is no easy way of avoiding this behavior, which is why all the j2me tutorial instruct developers to create a ‘sending data over the network…’ screen, instead of just sending everything in a background thread. If someone can prove me wrong, that would be fantastic.
Thanks!
One important aspect is you need to have a generic UI or screen that can be displayed when the network call in background fails. It is pretty much a must on any mobile app, J2ME or otherwise.
As Honza said, it depends on the design, there are so many things that can be done, like pre-fetching data on app startup, or pre-fetching data based on the screen that is loaded (i.e navigation path), or having a default data set built in into the app etc.
Another thing that you can try is a built-in timer mechanism that retries data download after certain amount of time, and aborting after say 5 tries or 1-2 minutes and displaying generic screen or error message.
Certain handsets in J2ME allow detection of airplane mode, if possible you can detect that and promptly display an appropriate screen.
Also one design that has worked for me is synchronizing UI and networking threads, so that they dont lock up each other (take this bit of advice with heavy dose of salt as I have had quite a few interesting bugs on some samsung and sanyo handsets because of this)
All in all no good answer for you, but different strategies.
It pretty much depends on how you write the code and where you run it. On CLDC the concept of threading is pretty limited and if any thread is doing some long lasting operation other threads might be (and usualy are) blocked by it as well. You should take that into account when designing your application.
You can divide your file data into chunks and then upload with multiple retries on failure. This depends on your application strategy . If your priority is to upload a bulk data with out failure. You need to have assemble the chunks on server to build back your data . This may have the overhead for making connections but the chance is high for your data will get uploaded . If you are not uploading files concurrently this will work with ease .