I'm developing a .NET app, which needs to run both on Azure and on regular Windows Servers(2003). It needs to store a few GB of data and SQL Azure is too expensive for me, so I'll use Azure tables in the cloud version. Can you recommend a storage solution, which will run on standalone servers and have an API and behavior similar to Azure tables? From what I've seen Server AppFabric does not include Tables.
If you think what Windows Azure Table Storage is, it is a Key-Value pair based non-relational databse which is accessible through REST API. Please download this document about Windows Azure and NoSQL database details.
If I were in your situation, my approach would have been to find something similar to Azure Table Storage which I can access over REST and have similar accessibility API. So if you try to find the similar database to run on a machine you really need to look for:
Key Value Pair DB
Support for basic operations i.e add, delete, insert, modify an entity
Partition Key and Row Key based Accessibility
RESTful Interface to connect
If you would want to try something you sure can look at:
DBreeze (C# based Key Value Pair NoSQL DB) I just saw it and looks exciting
Googles LevelDB (Key Value Pair DB, open source and available on Windows) I have no idea about API
Redis (Great Key-Value Pair DB but not sure for Windows compatibility and API)
Here is a list of key/value databases without additional indexing facilities are:
Berkeley DB
HBase
MemcacheDB
Redis
SimpleDB
Tokyo Cabinet/Tyrant
Voldemort
Riak
If none works, you sure can get any of open source DB and modify to work for your requirement and then make that available to others as your contribution to community.
ADDED
Now you can use Windows Azure Virtual Machine to run any kind of Key-Value pair DB on Linux or Windows Machine and connection with your application.
I'm not sure which storage solution to recommend, but just about any database solution would work provided that you write an Interface to abstract all your data storage code. Then write implementations of that interface for Azure Table storage and whatever other database you want to use on the non-cloud server
You should be doing that anyway so that your code isn't tightly coupled with Azure Table Storage APIs.
If you combine coding against that Interface with an IoC container, then a single line of code or a single configuration setting would enable you to switch between data implementations based on which platform the code is running on.
Related
Microsoft Cosmos DB includes DocumentDB API, Table API and others. I have about ~ 10 TB of data and would like to have a fast key-value lookup (very little updating and writing, mostly are reading). Add a link for Microsoft Cosmos DB:
https://learn.microsoft.com/en-us/azure/cosmos-db/
So how should I choose between DocumentDB API and Table API?
Or when should I choose DocumentDB API? When should I choose Table API?
Is it a good practice to use DcoumentDB API to store 10 TB of data?
The Azure Cosmos DB Table API was introduced to make Cosmos DB and its advanced indexing, geo-distribution, etc. features available to the Azure Table storage community. The idea is that someone using Azure Table storage who needs more advanced features only offered by Cosmos DB can literally just change their connection string and their existing code will work with Cosmos DB.
But if you are a greenfield customer then I would recommend using SQL API (formerly called Document DB API) which is a super set of Table API. We are constantly investing in providing more advanced features and capabilities to SQL API where as for Table API we are just looking to maintain compatibility with Azure Table storage's API which hasn't changed in many years.
How much data you have doesn't have any affect on what API you choose. They both have the same multi-model infrastructure and can handle the same sizes of data, query loads, distribution, etc.
So how should I choose between DocumentDB API and Table API?
Choosing between DocumentDB API and Table API will primarily depend on the kind of data that you're going to store. DocumentDB API provides a schema-less JSON database engine with SQL querying capabilities whereas Table API provides a key-value storage database service. Since you mentioned that your data is key-value based, recommended is that you use Table API.
Or when should I choose DocumentDB API? When should I choose Table API?
Same as above.
Is it a good practice to use DcoumentDB API to store 10 TB of data?
Both Document DB API and Table API are designed to store huge amounts of data.
However you may want to look into Azure Table Storage as well. Cosmos DB lets you fine tune the throughput that you need and robust indexing/querying support and that comes at a price. Azure Tables on the other hand comes with fixed throughput and limited indexing/querying support and is extremely cheap compared to Cosmos DB.
You may find this link helpful to explore more about Cosmos DB: https://learn.microsoft.com/en-us/azure/cosmos-db/introduction.
Please don't flag this as off-topic.
It might help for you to know in advance: if you are considering the document interface, then in fact there is a case-insensitivity that can affect how DataContract classes (and I believe all others) are transformed to and from Cosmos.
In the linked discussion below, you will see that there is a case insensitivity in Newtonsoft.Json that can have effects on your handling of objects that you pass or get directly from the API. Not that Cosmos has ANY flaws, and in fact it is totally excellent. But with a document API, you might (like me) start to simply pass DataContract objects into Cosmos (which is obviously not wrong, and in fact very much expected from the object API), but there are some serializer and naming strategy handler options that you are probably better of at least being aware of up front.
So just to add a note for you to be aware of this behavior with an object interface. The discussion is here on GitHub:
https://github.com/JamesNK/Newtonsoft.Json/issues/815
First time using Azure. I have a basic node js bot built with Microsofts Bot Framework, and deployed on Azure. What are my options for storage?
I will most likely just be needing simple key:value storage. Mongodb was my first though but I dont think Azure supports it nativeley.
That said, what are my options for storage on Azure? I usual shy away from MySQL just from preference, but theres no actual reason that wouldnt work either.
Take a look at Azure Table Storage for a NoSql solution
Table storage is a key/attribute store with a schemaless design. Because Table storage is schemaless, it's easy to adapt your data as the needs of your application evolve. Access to data is fast and cost-effective for all kinds of applications. Table storage is typically significantly lower in cost than traditional SQL for similar volumes of data.
I am trying to get to grips with Azure Mobile Services and am a little lost about which Azure services I should be using for what. When you set up a new Azure Mobile Service you only get the option to create it with Microsoft SQL Server as the database. Can you configure it to use the documentDB service instead?
Also a lot of what I read about Azure Mobile Services seem to be setup around storing data for a particular user. Is this the correct place to store data and query it for all the systems users rather than an individual or should that be handled elsewhere?
thanks
Andy
using the .NET backend, you have a choice of using Azure SQL Database,MongoDB or Table Storage. There's no out of the box support yet for DocumentDB, but you can make it work as shown here
what you store in the storage listed above is up to you, there is no specific reason/restriction/limitation that's its only for user specific data. the samples just happen to show that.
You may use any backend database service. It's true you have to set up SQL when you create a new mobile service space. And the SDK has a Table provider that maps to SQL (or MongoDB). However, you can ignore all that if you want. As long as you can access the appropriate driver for your database-of-choice, you can make calls to the database from the API backend methods. This is especially true when building custom API methods.
Tricky question - Does anyone know where the ORM SQL code lives that the Azure Dev storage emulator uses for Table Storage?
I'm wanting to implement a SQL version of Azure Table storage for use on-premises (outside of Azure).
And before explaining to me the differences between a relational db like SQL Server and a NoSQL key/value store like Azure Table storage - I know already, thanks :)
Go to the SQL server you installed the Storage Emulator. You will see a database called "DevelopmentStorageDb20110816" if you have installed the Azure SDK v1.5+ (It was a different date previous to that - something like DevelopmentStorageDb20090919 I think).
In that database you will find all the tables, stored procedures and functions needed to emulate Windows Azure storage.
Still searching for an answer to this question:
It's my understanding that there is no requirement that there be any structural or type similarity between different entities in a table in Azure storage. But what about development storage? I recently read that this is not true as development storage requires a schema because it stores the data using SQL Server Express.
Can someone confirm this. Also if this is the case then how could I use Dev storage to model different entities within a table in production?
Thanks in advance
Gordon,
Development Storage uses SQL server under the hood to simulate the live azure storage. (It must store the data somewhere). Think of it as an emulator. You use the same API to access live storage or development storage. You are not required to use a schema for development storage.
Think of it as changing a connection string from a staging server to a production server.
Check out this link for more detailed information:
You are not required to use a schema in production, but all objects in a development storage table must have the same set of columns (see here). I'd suggest using multiple tables in development, and switching to one table in production if you need to (maybe use a configuration value to indicate what table tables to use?)