I want to build a hybrid using Ionic/Electron to run it on Desktop,Browser and Android, after doing some research i found about RxDB which by it's definition seems to fit this kind of DB usage (https://rxdb.info/)
I want to be able to store data locally in an SQLite-like storage capacity without being limited by the web browser's storage capacity and use the same api (rxdb ?) to manipulate data on Desktop (Electron), Mobile and web browser (Ionic).
Is this possible using RxDB ? if it's not i would like to know if there are tools out there capable of allowing such functionality ?
Yes this is surely possible with RxDB, you can use IDB which stands for indexed data storage, which in terms of browser means that your data will not be limited by the browser capacity, since it is stored offline in your computer,
It is limited only to your available disk space Not to the browser capacity
That is what idb adapter helps in browsers
Situation:
I have a website where users can enter information, save their information, refresh the page and see the information they just entered.
Currently I am using sql database, but planning to moving to some cloud storage service.
I store user information in a json doc. This is usually 30 KB or less but worst case could be 900 KB.
I want strongly read-after-write consistency.
I am expecting json to doc to grow in the future, but not above 1 MB.
I want save/load to be as fast as possible (this is least important).
Investigations:
AWS DynamoDB restricts string to have a maximum size of 400 KB. I can try compress the json doc to meet this requirement but I am afraid in the future document grow to a size that cannot be compressed to 400 KB.
AWS S3 can store file up to 5 TB. However, it does not support consistent read after write.
Question:
I am not familiar with Windows Azure or Google Cloud. So my questions are:
Does Windows Azure / Google Cloud has strongly read-after-write consistency?
Does Windows Azure / Google Cloud has any restriction on single string size?
I am currently using EC2 hosts as server. What are the upload speed difference among the three? (I guess S3 is faster since I am using EC2?)
Are there any other data storage services that supports strong read-after-write consistency, 1 MB single string size, save/load less than 5 sec for 1 MB string?
I will try to answer your questions from Azure's perspective. I am sure someone will answer from Google's perspective as well.
Does Windows Azure / Google Cloud has strongly read-after-write
consistency?
Yes, Azure storage (Blobs, Tables, Queues, Files) follow strong consistency model. If the write is successful, you can immediately read the contents.
Does Windows Azure / Google Cloud has any restriction on
single string size?
Restrictions are there. It depends on the kind of storage you're using. If you're using Azure Blob Storage, for your purpose you would be using Block Blob and the maximum size of a block blob can be 200 GB. If you're using Azure Tables (which is Key/Value Pair NoSQL storage), maximum size of an entity can be 1MB. If you're using DocumentDB, maximum size of a document can be 512KB.
I am currently using EC2 hosts as server. What are the upload speed
difference among the three? (I guess S3 is faster since I am using
EC2?)
Quite honestly, this is a very broad question as upload speed depends on a number of factors. But yes, having application and data store in the same region has definite advantage from speed perspective as latency would be comparatively lower than if they were in a different region (or different cloud providers). I would not recommend splitting your application and storage in different cloud providers. If possible, put both of them with same cloud provider.
Are there any other data storage services that supports strong
read-after-write consistency, 1 MB single string size, save/load less
than 5 sec for 1 MB string?
Again, this is an opinion soliciting question and is quite broad thus I would refrain from answering this question.
I decided to use indexeddb to store data for a chrome extension I wrote. I've noticed that if I remove the extension and readd it, the data is lost.
Is there a way to persist the database across re-installations?
The apps docs (which also applies to extensions) recommend using storage.sync to handle persistence across re-installations, but the quota is very limited: 102,400 bytes. And it's not the same thing as indexeddb so some data processing would need to be done.
If your data is bigger than that I think the only solution is to sync your data manually with a server-side component, either your own or some other cloud-based solution like Firebase:
https://www.firebase.com/blog/2013-03-06-power-your-chrome-extension-with-firebase.html
Indexeddb is temporary storage. https://developers.google.com/chrome/whitepapers/storage
Browser may clear data anytime as necessary.
Until persistent data storage is available, possibly in combine with quota management api, treat indexeddb data as temporary.
I'm working on Integration project where third party will call our web service in Azure. For performance reason I would like to store 2 table data (more than 1000 records) on to the app fabric cache.
Could anyone please suggest if this is the right design pattern?
Depending on how much data this is (you don't mention how wide the tables are) you have a couple of options
You could certainly store it in the azure cache, this will cost though.
You might also want to consider storing the data in the http runtime cache which is free but not distributed.
You choice would largely depend on the size of the data, how often it changes and what effect is caused if someone receives slightly out of date data.
I am designing a system that's going to have about 10 millions+ users, each has a photo, which is about 1~2 MB.
We are going to deploy both database and web app using Microsoft Azure
I am wondering the way I should store the photos, there are currently two options,
1, Store all photos use Sql Server FileStream
2, Use File Server
I haven't experienced such large scale BLOB data using FileStream.
Can anybody give my any suggestion? The Cons and Pros?
And anyone with Microsoft Azure experiences concerning the large photos store is really appreciated!
Thx
Ryan.
I vote for neither. Use Windows Azure Blob storage. Simple REST API, $0.15/GB/month. You can even serve the images directly from there, if you make them public (like <img src="http://myaccount.blob.core.windows.net/container/image.jpg" />), meaning you don't have to funnel them through your web app.
Database is almost always a horrible choice for any large-scale binary storage needs. Database is best for relational-only systems, and instead, provide references in your database to the actual storage location. There's a few factors you should consider:
Cost - SQL Azure costs quite a lot per GB of storage, and has small storage limitations (50GB per database), both of which make it a poor choice for binary data. Windows Azure Blob storage is vastly cheaper for serving up binary objects (though has a bit more complicated pricing system, still vastly cheaper per GB).
Throughput - SQL Azure has pretty good throughput, as it can scale well, however, Windows Azure Blog storage has even greater throughput as it can scale to any number of nodes.
Content Delivery Network - A feature not available to SQL Azure (though a complex, custom wrapper could be created), but can easily be setup within minutes to piggy-back off your Windows Azure Blob storage to provide limitless bandwidth to your end-users, so you never have to worry about your binary objects being a bottleneck in your system. CDN costs are similar to that of Blob storage, but you can find all that stuff here: http://www.microsoft.com/windowsazure/pricing/#windows
In other words, no reason not to go with Blob storage. It is simple to use, cost effective, and will scale to any needs.
I can't speak on anything Azure related but for my money the biggest advantage of using FILESTREAM is that that data can get backed up inside the normal SQL Server backup process. The size of the data that you are talking about also suggests that FILESTREAM may be a good choice as well.
I've worked on a SCM system with a RDBMS back end and one of our big decisions was whether to store the file deltas on the file system or inside the DB itself. Because it was cross-RDBMS we had to cook up a generic non-FILESTREAM way of doing it but the ability to do a single shot backup sold us.
FILESTREAM is a horrible option for storing images. I'm surprised MS ever promoted it.
We're currently using it for our images on our website. Mainly the user generated images and any CMS related stuff that admins create. The decision to use FILESTREAM was made before I started. The biggest issue is related to serving the images up. You better have a CDN sitting in front. If not, plan on your system coming to a screeching halt. Of course, most sites have a CDN, but you don't want to be at the mercy of that service going down meaning your system will get overloaded. The amount of stress put on your sql server is the main problem here.
In terms of ease of backup. Your tradeoff there is that your db is MUCH MUCH LARGER and, therefore, the backup takes longer. Potentially, much longer and the system runs slower during the backup. Not to mention, moving backups around takes longer (i.e., restoring prod data in a dev environment or on local machines for dev purposes). Don't use this as a deciding factor.
Most cloud services have automatic redundancy of any files that you store on their system (i.e., aws's S3 and azure's blob). If you're on premise, just make sure you use a shared location for the images and make sure that location is backed up. I think the best option is to set it up so each image (other UGC file types too) has an entry in your db with a path to that file. Going one step further, separate the root path into a config setting and only store the remaining path with the entry. For example, root path in config might be a base url, a shared drive or virtual dir, or a blank entry. Then your entry might have "/files/images/image.jpg". This way, if you move your filestore, you can just update the root config. I would also suggest creating a FileStoreProvider interface (Singleton) that can be used for managing (saving, deleting, updating) these files. This way, if you switch between AWS, Azure, or on premise, you can just create a new Provider.
I have a client server DB, i manage many files (doc, txt, pdf, ...) and all of them go in a filestream BLOB. Customers has 50+ MB dbs. If in azure you can do the same go for it. Having all in the db is a wonderful thing. It is considered good policy also for Postgres and MySQL