Users frequently open multiple tabs to my meteor app. Is there a way to get those tabs to share the same connection (state on the server) so that there isn't multiple redundant connections. I'm thinking about coming up with a package to do this myself, I'm wondering if anyone has given any thought to this. It should help with performance.
It is possible to share client-side data through localStorage (consider it a browser database). It is also possible to share server-side data, commonly through database (MongoDB in case of meteor). Network connection (instead of collection) is shared across tab automatically by the browser.
If you mean sharing collection (instead of connection), you don't need to do anything special to share them between tabs (clients). Clients observing the same collection will see same data.
However, the convenience offered by Meteor has its cost. One of it is that each client has its own partial collection copy, thus it can use/waste lots of memory.
This is implementation details, and just like how JavaScript use/waste more memory and cpu then native code in exchange for convenience, there is not much you can do about it, at least not easily.
Update: As Harry noted, for real DDP connection 'sharing', it is possible to detect and disconnect new tabs and use localStorage to sync data from first tab, so that there is only one active connection. However IMHO it would be quite a heroic feat.
You should be able to use HTML5 local storage for this. This library does just that:
https://github.com/diy/intercom.js
Related
SearchLight is a Julia package that builds the ORM (object-relational mapping) layer in Genie (a web development framework in Julia).
Right now, I am building a website backend and I decided to use SearchLight for data storing, because someone told me to do so with an ORM instead of "just" parsing SQL queries - sorry, I am so not a web developer and know basically nothing.
Unfortunately, SearchLight is (i) unbelievably badly documented (i.e. in most cases not at all) and (ii) missing important functionalities, like setting foreign keys in a DB table. I found several work-arounds, even if they are not pretty and kind of counter the ORM idea.
Actual Question
Does SearchLight use connection pools?
The web service I'm building, gets a JSON, queries the database, gets the result and sends it back to the client. This might take several seconds and multiple clients doing a request should not "stand in line" and wait for other requests to finish. Will SearchLight do that? I.e. open several connections and use them when they're needed? (Or does it something else that leads to what I want?)
I have been working on a Web App for visualizing live data. It is crucial that this data is kept up to date on the client side without such updates being invoked directly by the client (e.g. no button presses or refreshing the page). Currently, on page load, I grab the current data set from a database (DynamoDB) via Ajax, and subsequent updates are pushed to any listening clients every 5 minutes via a Websockets connection (using Socket.io).
I have overlooked the computational load of this update job. It has to mine some data, process it, update the database, and send the update out to all clients. As a result, the web server is left unresponsive for about 30 seconds with each update. Furthermore, my current architecture limits me from putting my server behind a load balancer, which is something I anticipate coming up in the future. For both these reasons, I really need to get this update job off my web server.
I am relatively inexperienced in web development, and I don't feel I am knowledgeable enough about these technologies to know the drawbacks of the solutions I have come up with. Currently, I am considering:
Break the update off into a separate process so it does not block the Node event loop. This would solve my issue in the short term, but if I ever want to load balance my application, I can't have the update running on multiple machines.
Drop Websockets entirely and just have the client query the database every 5 minutes, while a separate process (or separate server if I want load balancing) keeps the database up to date without interacting directly with the client. Will this kind of access pattern put too much load on my db?
Have a separate server run the update, and send the result via Websockets (or maybe some other protocol) to my load balanced application servers, which then push that update to all listening clients as usual. Is this even possible?
Perhaps there are other solutions. It seems like this would be a relatively common problem, so I was hoping I could find some guidance here. What are the potential issues with the solutions I have proposed, and are there other possible solutions that my suit my use case better?
It sounds like you want one process sitting somewhere which crunches the data and publishes it to a stream. Clients can then subscribe to the stream as and when they like. Redis handles streams nicely, you could process your data and push it into a redis stream. You could then create a small node service which subscribes to the redis stream and pushes the formatted data out over a websocket or via polling.
In this scenario you can then scale up either the publishing process (the one crunching the numbers) if your data load goes up, or scale up your subscribed process (which serves the data over a websocket to browsers) if you get an influx of clients watching the data.
You can also easily distribute the hosting of these services across other machines, and even write them in different languages if you decide the number crunching needs something like threading.
You're then left with the issue of clients (web browsers) consuming this data with a load balance in-between. This can be a hard problem if you use websockets and is bundled with pros and cons. But importantly you'll have separated your data crunching from your result publishing and that'll isolate out your issue to only the load balancing.
I have done pretty much the same to check ressources on some of our servers.
I have a C# service getting the information on each server that we manage, sending them to a queue (Amq).
From there, I have a stomp client fetching data from amq and emiting them to a websocket.
My main micro service is fetching the data to save them into a db.
My visualisation webapp is connected to the same ws and is fetching the data as they are sent to display them.
The Amq step isn't mandatory at all, it's just something I had to work with (historical).
I don't know what type of data your are working with, so I don't know if my solution can apply to you.
Don't hesitate if I'm not clear or you have any question.
This is a big question and I'm not going to try and give you a definitive answer.
For option 2
It really depends on how expensive your queries are. You can make DynamoDB fast if you pay for enough throughput. That said, on the face it, re-loading your whole dataset, when that sounds like its probably large, probably isn't good engineering.
For option 3
This option seems best to me if its achievable, although admittedly its hard to say with such a complex system - obviously you can't share your whole project.
Given your are already using AWS you might want to look into AWS Lambda. If you can move the update process into a stand alone job, you can host it on lambda and move the load off the web server. Lambda is essentially infinitely scalable and you only pay for the compute you use.
This really depends on you being able to split the update task off into a separate service. Its likely you would need a fair bit of refactoring to isolate it as a service. If you can break little bits off at a time, and make the move gradually, even better.
If you consider trying this, and you've not used Lambda before, I would definitely start small with some hello world examples. Then try a very simple service in your application, and build up to taking on the update service.
You might also consider looking in AWS Simple Message Queue Service to handle the comms between clients and server.
Database tuning
If a lot of your update time is spent waiting for database actions to complete, rather than server processing, you can consider tuning that side of things up. Things to consider are:
Buying more throughput
Using batch operations (as these move load to DynamoDB from your server)
Tuning keys, indexes and database access
I am using kue.js, which is a redis-backed priority queue for node, for pretty straightforward job-queue stuff (sending mails, tasks for database workers).
As part of the same application (albeit in a different service), I now want to use redis to manually store some mappings for a url-shortener. Does concurrent manual use of the same redis instance and database as kue.js interfere with kue, i.e., does kue require exclusive access to its redis instance?
Or can I use the same redis instance manually as long as I, e.g., avoid certain key prefixes?
I do understand that I could use multiple databases on the same instances but found a lot of chatter from various sources that discourage the use of the database feature as well as talk of it being deprecated in the future, which is why I would like to use the same database for now if safely possibly.
Any insight on this as well as considerations or advice why this might or might not be a bad idea are very welcome, thanks in advance!
I hope I am not too late with this answer, I just came across this post ...
It should be perfectly safe. See the README, especially the section on redis connections.
You will notice that each queue can have its own prefix (default is q), so as long as you are aware of how prefixes are used in your system, you should be fine. I am not sure why it would be a bad idea as long as you know about the prefixes and load usage by various apps hitting the redis server. Can you reference a post/page where this was described as a bad idea ?
I have an app that I would like to create. But I am not sure how to go about it. I am using node.js and would like to use couchdb, but if something like mongodb or riak would be a better choice them im willing to hear ideas. But, i have a site, say
cool.com
and on there is a couchdb instance, as well as a site to manage a store. say a shopping cart. the db houses all the store's items and data. The app itself has an admin backend to manage that data and can change items. What i would like to be able to do, is have the ability to have the user be disconnected from the internet, and still have the admin backend work. I realize for this to work I need to use a client side framework with my models/routes/controllers/whatever. But what I am not sure of, is how to let the site function while offline. couchdb if installed locally can sync the data from local to remote when back online, and if the admin user is on the computer, i could have them install couch. but that could be messy.
Also, what if the admin user is on a tablet or a phone? Would I need to have an actual mobile app and a desktop app to do this? is there some way I can set this up so it is seamless the the end user. I would also like this to be offline for end users too, but the bigger audience is the admin.
Another use case, instore POS system. and the power goes out. But the POS system can be loaded from the web onto a tablet and they can still make card based sales if the wifi is out, because the app is available offline.
Im just not sure how to do this. lets assume i need a client framrwork that can handle the data as well as the backend. something like ember, or angular. theres also all in one stacks like meteor and derby js, but those arent fully offline,but are for the appearance of real time. though meteor does have mini mongo so it might be worth looking into.
I was hoping someone could help me figure out how I would get this setup to work, preferrably with couch, but other nosql's would work too if I can have a way to sync the data.
I'm not sure if it would work for you, but I have been thinking of such an application for quite a long time now and been doing some research on what's possible. The best solution I could come up with is using a server with a couchdb and writing the application clientside based. Then for the data storage use pouchdb and synchronize the pouchdb regularly with your serverside couchdb if the app is online. I know pouch is in an early stage and not production ready but if you are willing to put some work into it I'd say it's doable.
If you want clients that work seemless as they go offline and come online (like a POS with the power out) then I would recommend making the app primarily work off local storage with a background publishing or synchronization to the cloud.
Local storage options could be everything from something light like sqlite, sqlexpress, firebird to no sql options like mongo, couchdb etc...
But for the client or device, consider the ease of configuration and weight of the option. You also need to consider the type of clients - do you have many platforms varying from devices to PCs? You don't want something that has a heavy config and runtime footprint. That's fine on the service side.
On the service side, consider the nature of your data and whether it's fitted better for transactional/relational systems (banking etc...) or eventually consistent/non transactional (no-sql) documents. Don't forget hybrid as an option. Also consider the service platform - for example, node goes well with mongodb (json objects front to back) ...
The device and service storage options can be different (and likely should be) separate by service interfaces (soap, rest/http, sockets etc...).
It's hard to have a one size fits all solution but often something light weight like sqlite on the device or client makes for ease of installation/config while scalability on the service side with something like sqlserver/mysql or couchdb/mongodb makes sense.
Some links to read:
http://www.mongodb.org/display/DOCS/Comparing+Mongo+DB+and+Couch+DB
http://www.sqlite.org/
http://blogs.msdn.com/b/sqlexpress/archive/2011/07/12/introducing-localdb-a-better-sql-express.aspx
You're question is pretty wide open and there's no one size fits all solution. Hopefully I provided some options to think about.
There's an interesting project out there called AppJs (http://appjs.com/), which packages Node.JS and Chrominium as a desktop environment. It's currently very fresh (very little documentation), but it appears to be straight forward enough (you'll be using the same tools as you would for your online application).
As for synchronising the offline and online environments. I doubt you can rely on CouchDB in the way that you envisage. CouchDB mobile support is not as comprehensive as some of the documentation suggests. So in this sense, it would be no different to using SQL/Mongo/Punchcards.
You might have more luck with designing a suitable serialisation scheme based on XML or JSON (or just plain text), and passing files between the online and offline installations.
Edit - Since writing this, Node Webkit - http://nwjs.io/ - is clearly the most obvious replacement for App.js. It has a very simple API, and some great features.
I have developed an application XPages that work very well in a Browser (Firefox ) and in every page the browser load max 150Kb of content (html, image, js, css...etc...)
When I have deploy the application to my remote user that directly access to server with XPiNC mode the speed are very very poor!
With a tool I sniffed the traffic and I see that for every GET there are 10Mbytes of data transfered (seem to transfer XML source and other code that is compiled on the fly...)
The application inside Notes Client is not useable so...and my customer has disappointed for this feature (is not possible use in local and replicate)
I have 8.5.3FP2 (client and server) with PRELOAD option setting.... without any change of this.
Have someone any suggest for me? Is this a BUG ?
It is true that remote applications (NSFs residing on a non-local server) are slower than local client replicas or remote apps run in a web browser. This is due to the fact that a lot more network transactions are generated when running in this mode. There are various things that can be done however to remedy the problem.
First however we need to identify the cause of the problem - you are seeing a 10MB transfer for each GET request, which is very large and will obviously negatively impact performance. One or more of the XPages in your application may be using the computeWithForm feature? If an XPages document data source "computes" a Notes form (typically to execute pre-existing application logic) then the form must be copied across the net to be computed in the local client. However all children of the form will also be hauled over - subforms, shared fields etc, and this can result in large net transactions like those you are seeing.
Often the computeWithForm feature is used as a development convenience and as long as the size of the form is small then the performance impact can be negligible. However, if the aggregate form is large, then it may be worth your while replacing the computeWithForm usage with separate XPages SSJS application logic.
Before going further we would need verify that this is in fact the issue - there could be other issues. Typically this manifests only on pages that open/edit documents - so you can maybe try turning computeWithForm off in a test environment and see if there is a difference.
XPiNC is a little special. When you open a server based NSF, all the program code needs to be downloaded to the client to be executed in the server container of the Notes client. The reasonable way to use an XPiNC with data in the server is to split the application. Have one NSF that contains all the program logic (all XPages and other code) and the other with forms, views and documents.
Replicate the application NSF locally and access only the data on the server. This should give you much better performance. You could have a configuration setting to compute the data NSF, so disconnected users could use a local replica of the data.
Let us know how it goes.
P.S.: There are some more tuning ideas...