I'm new to Node, and was hoping to use Node.js for a a small internal webapp for managing workflow for product photos.
The image files are RAW camera files, and stored on a local NAS device.
The webapp should:
Have a concept of workflow, and be able to jump back/forth between states, and handle error states.
Watch certain directories for image files, and react to new files being added, or existing files being moved/removed.
Send out emails in response to events.
Scan photos for QR barcodes, and generate events based on these.
Rename photos based on user-defined batch patterns in response to events.
Questions:
Is Node.js a suitable tool for something like this? Why or why not?
Any libraries to help manage the workflow? I could only find node-workflow (http://kusor.github.io/node-workflow/) - curious for anybody's experiences with this? Alternatives?
Likewise for file watching? I saw many wrappers for fs.watch (e.g. https://github.com/mikeal/watch), as well as some alternatives (e.g. https://github.com/paulmillr/chokidar) - any advice for somebody new to Node?
Apart from using a NAS with a network filesystem, are there any other alternatives stores I can use for the image files?
I'm open to other alternatives here. I'm worried that the system will get confused or lose track of files.
The Node.js docs also mention that watching files on network file systems might be unreliable (http://nodejs.org/api/fs.html#fs_fs_watch_filename_options_listener). Are there more robust solutions?
Any other tips/suggestions for this project?
Cheers,
Victor
Node.js is a fine platform for building an application like the one you describe. The key question is file storage. You might find this post very interesting:
Storing Images in DB - Yea or Nay?
This other post enumerates a few interesting options for writing workflows:
Workflow engine in Javascript
Shameless add: I have been working on an event coordination library that you might find useful and interesting.
http://durablejs.org
Related
This is unfamiliar ground for me and I've been poking around with resources trying to find a suitable solution. But essentially, I have a website setup that I want to allow users to upload MP3/FLAC files. Then I want to take those files and send them to a Google Cloud Bucket. The second part seems easy enough, plenty of NodeJS tutorials regarding that.
Since I'm pretty in the dark with how this is done, would I need to "upload" a file on my frontend and then hit my node-express api backend with some sort of fs. solution that looks up the file on my machine? If so, how could that be consistent between users, what if their directory structure is different on their machines?
Anyway, kinda shooting in the dark here. Would love to have some advice regarding this.
It's not really feasible for a backend to "reach into" a frontend machine to pull files from it. The client needs to provide the data directly to the backend.
Mostly commonly, Firebase client libraries are used to directly upload contents from a client machine to a storage bucket. If you don't do that, you'll need to create your own backend API that clients can invoke to send data.
I need to write own video hosting with player on client side.
My required approaches:
The user can upload video to hosting
The user can watch any video from hosting
I don't ask to write me solution, I am asking for help where should I start from to lean about it? Which technologies or frameworks should I learn for my task to realize it using python?
P.S. Each detail will be very useful, especially some links to articles because I couldn't find by myself not knowing accurately what do I need to search.
Added
Now, I think to store videos in the file system directly and use postgresql to store additional information about videos and users. Of course, large services use Hadoop, BigTable and etc but for my task so solution will be enough I think.
When the user uploaded a new video, my server saves it into a temporary directory and puts in the processing queue. Small programs takes new videos one by one, generates thumbnails and decrease a quality of videos and moves it to the base storage. Is it a good idea?
But I still can't get how make a video streaming
Ok So I dont want to encourage the behavior of people thinking SO is a codewriting service, But this is a truly legitimate answer. So first of all, you want to choose a language. Currently I'd recommend the use of javascript and Node.js (Java needs to die). However, IDK Node as well as I know Python. Python is an all purpose language yadda yadda yadda blah blah blah. Whats important in this case is your framework of choice (or library). The libraries that allow you to make websites in python (or make it easier to do so) are very interesting. There are several but my favorite is Python Flask. Python flask is actually very similar to Node.js + Express.js. Use this link to get started. Take a few days to learn the insides of this framework. VERY MODULAR, VERY POWERFUL. Using basing logic and database knowledge, one can easily accomplish a simple file upload and authentication-using, web service. However I Know of 2 really good guides that will help you with the streaming of the videos. I mean yes, You don't really need to know this. You could potentially load the requested video using a <video> tag, but streaming is a much MUCH
more favorable solution. Take some time to learn about video streaming and compression, and after you think about it, check out these links: AUDIO STREAMING GIST and MIGUEL GRINBERG FLASK VIDEO STREAMING BLOG POST
Good luck with flask and
Pro tip: learn about http(s) and the get and post methods
You would never imagine how many times I struggled with a bad request error or a method not allowed because I didn't do my research
I am developing an social app on iOS that have many-to-many relation, local persistency, and user interaction. I have tried using native Parse API in iOS and find it too cumbersome to do all the client-server logic. So my focus shifted to finding a syncing solution.
After some research I found AFIncrementalStore quite easy to use and it's highly integrated in CoreData. I just started to work on this and I have two questions to ask:
1) How to do the authentication process? Is it in AFRESTClient?
2) How to set up AFRESTClient to match Parse's REST API? (an example would be great!)
P.S. I also found FTASync, which seems to be another solution. Any thought on this framework?
Any general suggestion on client-server syncing solutions will be highly appreciated!
Thanks,
Lei Zhang
Back with iOS 5 Apple silently rolled out NSIncrementalStore to manage connection between APIs and persistent stores. Because I couldn't word it better myself:
NSIncrementalStore is an abstract subclass of NSPersistentStore designed to "create persistent stores which load and save data incrementally, allowing for the management of large and/or shared datasets". And while that may not sound like much, consider that nearly all of the database adapters we rely on load incrementally from large, shared data stores. What we have here is a goddamned miracle.
Source: http://nshipster.com/nsincrementalstore/
That being said, I've been working on my own NSIncrementalStore (built specifically for Parse and utilizing the Parse iOS/OS X SDK) and you're welcome to check out/use/contribute to the project at https://github.com/sbonami/PFIncrementalStore.
Take a look at this StackOverflow question and at Chris Wagner's article on raywenderlich.com.
The linked SO question has examples for how to include the authentication token with each request to Parse. So you'll just need to have the user log in first, and store their token to include it with each subsequent request.
Chris Wagner's tutorial has a sample AFHTTPClient named SDAFParseApiClient to communicate with the Parse REST API. You'd have to adapt it to be an AFRESTClient subclass, but it should give you a start.
Some other thoughts between the two solutions you're considering:
AFIncrementalStore does not allow the user to make any changes without a network connection, while FTASync keeps a full Core Data SQLite store locally and syncs changes to the server when you tell it to.
FTASync requires you to make all your synched managed objects subclasses of FTASyncParent, with extra properties for sync metadata. AFIncrementalStore keeps its metadata behind the scenes, not in your model.
FTASync appears not to be widely used and hasn't been updated in over a year; if you use it you will likely be maintaining it.
I have an app that I would like to create. But I am not sure how to go about it. I am using node.js and would like to use couchdb, but if something like mongodb or riak would be a better choice them im willing to hear ideas. But, i have a site, say
cool.com
and on there is a couchdb instance, as well as a site to manage a store. say a shopping cart. the db houses all the store's items and data. The app itself has an admin backend to manage that data and can change items. What i would like to be able to do, is have the ability to have the user be disconnected from the internet, and still have the admin backend work. I realize for this to work I need to use a client side framework with my models/routes/controllers/whatever. But what I am not sure of, is how to let the site function while offline. couchdb if installed locally can sync the data from local to remote when back online, and if the admin user is on the computer, i could have them install couch. but that could be messy.
Also, what if the admin user is on a tablet or a phone? Would I need to have an actual mobile app and a desktop app to do this? is there some way I can set this up so it is seamless the the end user. I would also like this to be offline for end users too, but the bigger audience is the admin.
Another use case, instore POS system. and the power goes out. But the POS system can be loaded from the web onto a tablet and they can still make card based sales if the wifi is out, because the app is available offline.
Im just not sure how to do this. lets assume i need a client framrwork that can handle the data as well as the backend. something like ember, or angular. theres also all in one stacks like meteor and derby js, but those arent fully offline,but are for the appearance of real time. though meteor does have mini mongo so it might be worth looking into.
I was hoping someone could help me figure out how I would get this setup to work, preferrably with couch, but other nosql's would work too if I can have a way to sync the data.
I'm not sure if it would work for you, but I have been thinking of such an application for quite a long time now and been doing some research on what's possible. The best solution I could come up with is using a server with a couchdb and writing the application clientside based. Then for the data storage use pouchdb and synchronize the pouchdb regularly with your serverside couchdb if the app is online. I know pouch is in an early stage and not production ready but if you are willing to put some work into it I'd say it's doable.
If you want clients that work seemless as they go offline and come online (like a POS with the power out) then I would recommend making the app primarily work off local storage with a background publishing or synchronization to the cloud.
Local storage options could be everything from something light like sqlite, sqlexpress, firebird to no sql options like mongo, couchdb etc...
But for the client or device, consider the ease of configuration and weight of the option. You also need to consider the type of clients - do you have many platforms varying from devices to PCs? You don't want something that has a heavy config and runtime footprint. That's fine on the service side.
On the service side, consider the nature of your data and whether it's fitted better for transactional/relational systems (banking etc...) or eventually consistent/non transactional (no-sql) documents. Don't forget hybrid as an option. Also consider the service platform - for example, node goes well with mongodb (json objects front to back) ...
The device and service storage options can be different (and likely should be) separate by service interfaces (soap, rest/http, sockets etc...).
It's hard to have a one size fits all solution but often something light weight like sqlite on the device or client makes for ease of installation/config while scalability on the service side with something like sqlserver/mysql or couchdb/mongodb makes sense.
Some links to read:
http://www.mongodb.org/display/DOCS/Comparing+Mongo+DB+and+Couch+DB
http://www.sqlite.org/
http://blogs.msdn.com/b/sqlexpress/archive/2011/07/12/introducing-localdb-a-better-sql-express.aspx
You're question is pretty wide open and there's no one size fits all solution. Hopefully I provided some options to think about.
There's an interesting project out there called AppJs (http://appjs.com/), which packages Node.JS and Chrominium as a desktop environment. It's currently very fresh (very little documentation), but it appears to be straight forward enough (you'll be using the same tools as you would for your online application).
As for synchronising the offline and online environments. I doubt you can rely on CouchDB in the way that you envisage. CouchDB mobile support is not as comprehensive as some of the documentation suggests. So in this sense, it would be no different to using SQL/Mongo/Punchcards.
You might have more luck with designing a suitable serialisation scheme based on XML or JSON (or just plain text), and passing files between the online and offline installations.
Edit - Since writing this, Node Webkit - http://nwjs.io/ - is clearly the most obvious replacement for App.js. It has a very simple API, and some great features.
We are working for a client to redesign an existing system which basicaly deals with a lot of files.
The files(more than 5 million) are currently stored on the servers filesystem.The client wants the new system to store the file in S3.
The files also have metadata associated(name,authors name,price ,description etc.).
The search functionality is also to be redesigned.The following are the basic requirements
Full text search should be available on file descriptions.
Filtering should be possible on other attributes of files.
Also , based on the file description, the system should also be able to give recommendation for similar files.
I do not have experience with creating such solution before,so asking for help and suggestion.
I was thinking on the lines of following solutions:
Store the file meta data in MongoDB ,and use the search functionality (http://www.mongodb.org/display/DOCS/Full+Text+Search+in+Mongo)
Use Amazon DynamoDB.It provides api to scan/query the dataset.
Use Lucene/Solr(I havent worked with these yet,I still need to look deeper)
There was this project that I found,that is very similar to what I require
http://www.thriftdb.com - On the home page it says its a datastore with search builtin.
Please let me know if this question should be a community wiki.
Thanks in advance.
You're in luck, announced today:
http://aws.amazon.com/about-aws/whats-new/2012/04/11/aws-announces-cloudsearch/
About searching files and filtering by attributes, the best would be Sphinx Search Engine which is used in filestube (google was using it also years ago).
I dont know if it will work on amazon servers.
Amazon has a custom AMI for Lucene/Solr and we have been happily using it in our projects. Lucene has a powerful indexing capability and executes at exceptional speeds. I would strongly recommend using Apache Lucene/Solr for all your search needs.