how to serve a large database to the clients?
how to notify clients to update only the changes?
event better, how to provide this functionality with sync-mechanism?
scenario:
the scenario itself has some requirements I'll try to explain:
there is an offline database here that devices need to get for working offline independently.
clients only have to sync themself. some replication mechanism like master-slave .
the master can write data to DB and slaves only have to sync and read data .
I have two bottlenecks here:
database is about: 60 Mb but it can grow much more.
because of multi-platform use case : client's devices maybe be in macos , windows, andriod, ios
first I was using google-firestore for this purpose but our data is some kind of sensitive and we cannot use migration strategy in the future. so I created large sqllite db for clients and . the clients can download the database manually. this is not right. even with small updates, our clients have to download db again.
is it possible to create a self synced mechanism that backend notify clients for getting updates?
Related
Trying to decide between DynamoDB and CouchDB for my website. It's a static site (built with a static site generator) and I'm planning on using a JavaScript module to build a comment system.
I'm toying with using PouchDB and CouchDB so that synchronizing is easy. I'm also considering DynamoDB.
I have a performance question. From these databases, do any of them push data out to edge locations so that latency is reduced? Or is my Database essentially sitting on one virtual server somewhere?
From what I know, neither of these solutions utilise edge locations ootb.
Since you're mentioning PouchDB, I assume you want to use a client-side database in your app?
If that's the case you should keep in mind that, in order to sync, a client-side DB needs to have access to your cloud db. So it's not really suitable for a comment system since all client could just drop comments of others, edit them, etc.
I wish to use PouchDB - CouchDB for saving user data for my web application, but cannot find a way to control the access per user basis. My DB would simply consists of documents using user id as the key. I know there are some solutions:
One database per user - however it requires to monitor whenever a new user wants to save data in order to create a new DB, and may create a lot of DBs;
Proxy between client and CouchDB - however I don't want PouchDB to sync changes for the whole DB including documents of other users in those _all_docs, _revs_diff request.
Is there any suggestion for user access control for pouchDB for a user base of around 1 million (active users around 10 thousand only)?
The topic of a million or more databases has come up on the mailing list in the past. The conclusion was that it depends on how your operating system deals with that many files. CouchDB is just accessing parts of the .couch file when requested. Performance is related to how quickly it can find, open, access, and close that file.
There are tricks for some file systems like putting / delimiters in the database name--which will cause CouchDB to store them in matching directory structures such as groupA/userA.couch or using email-style database names com/bigbluehat/byoung.couch (or some similar).
If that's not sufficient, Apache CouchDB 2.0 brings in BigCouch code (which IBM Cloudant uses) to provide a fully auto-sharded CouchDB. It's not done yet, but it will provide scalability across multiple nodes using an Amazon Dynamo style sharding system.
Another option is to do your own username-based partitioning between multiple CouchDB servers or use IBM Cloudant (which is built for this level of scale).
All these options provide the same Apache CouchDB replication protocol and will work just fine with PouchDB sitting on the user's computer, phone, or tablet.
The user's device would then have their own database +/- any share databases. The apps on those million user devices would only have the scalability of their own content (aka hard drive space) to be concerned about. The app would replicate directly to the "cloud"-side user database for backup, web use, etc.
Hopefully something in there sounds promising. :)
I am working on inventory application (C# .net 4.0) that will simultaneously inventory dozens of workstations and write the results to a central database. To save me having to write a DAL I am thinking of using Fluent NHibernate which I have never used before.
It is safe and good practice to allow the inventory application which runs as a standalone application to talk directly to the database using Nhibernate? Or should I be using a client server model where all access to the database is via a server which then reads/writes to database. In other words if 50 workstations when currently being inventoried there would be 50 active DB sessions. I am thinking of using GUID-Comb for the PK ID's.
Depending on the environment in which your application will be deployed, you should also consider that direct database connections to a central server might not always be allowed for security reasons.
Creating a simple REST Service with WCF (using WebServiceHost) and simply POST'ing or PUT'ing your inventory data (using HttpClient) might provide a good alternative.
As a result, clients can get very simple and can be written for other systems easily (linux? android?) and the server has full control over how and where data is stored.
it depends ;)
NHibernate has optimistic concurrency control ootb which is good enough for many situations. So if you just create data on 50 different stations there should be no problem. If creating data on one station depends on data from all stations it gets tricky and a central server would help.
We have an application for iOS which has a chat feature. Currently it works with long poll. And now we are trying to modify it to work with sockets. When it comes to socket, we have started for a research and it seems that one of the best option is using nodejs with socket.io. Then we have used redis pub/sub to manage the message delivery and storage.
After a few researching on redis, the recommended usage suggests the stored data should fit on memory. But, we have a little big database. We would like to store the whole chat history. So we have started to plan to use redis as a cache database, which will store the online user's chat history (may be not whole of them) and write the actual conversation after getting offline from redis to mongodb/simpledb (or instantly both of them).
So as a summary, we are about to decide to use nodejs and redis pub/sub to deliver messages, redis as a cache database, and mongodb to store the whole conversation.
What do you think about the design? Is this acceptable? Or, if there is a better way you can suggest, can you please explain a little more?
Thanks in advance.
For a chat system, you're thinking big. If you think you're going to reach a million users then go for it. Consider also availability - how will your system deal with failure of a machine?
Iam developing a VOIP platform which would allow users to make 100s of calls concurrently using my service.
Asterisk stores all call detail records in the CDR table. I would like to know where is best place to keep this table for the best possible architecture of my system.
Should I keep it on the Asterisk Box and run a cron to sync it with the database server OR should I directly call the DB server by the Asterisk Box and log all data directly on the database remotely though Asterisk.
All feel that both the architectures have their own pros and cons. I would like the help of experts to suggest me which would be best possible path for long term scalability and sustainability.
The best architecture would be to use distributed nodes(Server) i.e. PBX,web server & DB server in different nodes. PBX will populate your CDR table(this must be in a DB server) after every call, you can fetch these records from your web server for your reporting & billing purpose.
Using Cron to Sync DB table is not recommended as it will eat up the system resources & Bandwidth too (as this cron will run each time eating up the system resource & syncing with Db will cause bandwidth usage)
So using above defined architecture you can save system resources that will be used in running cron
Secondly if you place CDR in same node as PBX it will save system resource due to cron but for reporting & billing you have to fetch data from this node so you cant save Bandwidth, this schema has a major drawback, as currently you are talking about 100 calls concurrently what about if you had 1000 or more ??
In this case you have to definitely use PBX clustering in that case you will need a centralized DB server that will be synced by your PBX clusters.
So in all aspects my suggested architecture would perfectly suite your need.
As it is stated in the question that you need only 100s of concurrent calls you can use a single node for DB & Web server while PBx in other node
Using a separate database server to store your CDR's is the correct option for anything but a hobby Asterisk implementation. Asterisk makes it easy to select a destination database for your CDR's and has a myriad of different database options: MySQL, Postgresql. MSSQL etc. The Asterisk CDR implementation only uses a single table so it's actually a very simple integration between it and your database server.
One thing to be VERY aware of is that if your database server or the connection between it and your Asterisk server has problems it WILL impact your call processing. If there's a problem Asterisk will block while it keeps trying to connect to the database to write the CDR's. While it's doing that it won't process any other calls. Arguably this is desired behaviour as CDR's are critical for billing and not being able to log them means any calls would potentially end up being free. As a backup you can also set up CDR logging to a .csv file on the Asterisk server as a belt and braces approach.
I think that if you can connect directly from Asterisk to database than you should use it. I have seen it on some Asterisk installations (including one quite big call center) and it worked well.
The other option I use where there is no direct connection from Asterisk to database, but there is HTTPS connection to other service, or where billing table structure is not compatible to Asterisk tables it to use CSV CDR files. Such file is send every few minutes to CRM system. I use cron and little Python script. This way I can easily adapt to CSV format used by CRM billing system.