I have read here and on other post and forums that the best place to save session state in Azure is AppFabric Cache, but I find that very expensive and would like to give a go to either table storage or a SQL database.
I read that a SQL database will be faster but I can't understand why it would be. Surely the SQL database will cache hot data in memory, but I would expect Table Storage to also do that (does it?). Otherwise I don't see why a SQL database would be faster at retrieving a row than Table Storage, in the end both are just retrieving data from a local disk based on a key. I would even expect that because Table Storage scales up well and automatically (vs a SQL databases that needs to be partitioned manually), it would be preferable as session state isn't a good candidate for local caching.
Does anyone have any experience or opinion on this?
thanks
Charles
You mentioned AppFabric Cache, which is a retired service. Regarding SQL vs Table: There isn't really a right answer to this. If you want to spin up a SQL Database instance (running about $2.50 monthly for a Basic-tier database), you'll have 2GB to work with. With Table storage, you'll pay about $0.15 for the same storage. Then there is Redis cache, your own cache (such as memcached), Azure Managed Cache service, etc. Performance-wise, you'd need to do some benchmarking to see how each performs. Any of these would work with Virtual Machines, Cloud Services (web/worker roles), and Web Sites, as they all have very well-defined APIs and, if using ASP.NET MVC, good provider support. Each has different capacity limits and different pricing.
One thing with Table storage: each entity (row) is limited to 1MB, so if you're attempting to cache > 1MB per cache entry, you'll need to consider another option.
#Gaurav mentioned in-role cache. This is a great way to use extra memory in your web/worker role instances. However: It's limited to web/worker Cloud Services; it doesn't help with Web Sites or Virtual Machines. For those, you really need some type of independent cache provider.
Related
I am now extending some features of our corporate sharepoint, and one of the wishes of my customer is following:
Sharepoint farm has little space allocation, about 2 Gb per department. The department I am working with has hundreds of midsized documents (4-10 Mb in average, pptx, doc, pdf...) per month to be stored permanently. Currently they do it in a NAS file share. They want however to store them in Sharepoint for accessibility, but avoiding the 2Gb limit. Is there a way to integrate the external file storage to the Sharepoint?
Alternatively, is it possible to store the file data in a database (i.E. Access or MS SQL) but avoiding the farm-wide installations of frameworks like RBS?
You can using Remote BLOB Storage (RBS) as part of your data storage solution.
Check the article: Manage RBS in SharePoint Server
Before you migrate your BLOBs out of the database, you’ll need to choose an option. Here are several typical options:
1.File System: You can use a normal file system (perhaps a large disk partition on a file server) to store your BLOBs.
2.SAN/NAS Storage: Storage area network (SAN) and network attached storage (NAS) are usually high-end storage options. They’re expensive, but well-suited if the business value of your documents and their size can justify the cost. Both SAN and NAS provide data replication and mirroring and seamless growth into terabytes of data.
3.Cloud Storage: This is useful in at least two situations. First, when you’re running SharePoint in the cloud, but still want to externalize the BLOBs, your natural choice is to store them in a nearby, vendor-provided cloud storage. The second is when you’re running SharePoint within your own datacenter but want to store or archive all BLOBs in the cloud due to space limitations or reliability issues in your datcenter. Archiving is the most common reason in this situation. Make sure that if a user is creating or modifying document for cloud storage that your EBS or RBS provider does this in the background, as it could degrade performance. You also want to make sure your third-party EBS or RBS provider supports storing BLOBs in the cloud storage.
More information is here: Optimize SharePoint with External Storage
RBS is the way to leverage non-SQL Server storage for your BLOBs. Any other approach you take will be a hack/work around. Only your customer's requirements can tell us if the limitations of these workarounds are acceptable. For example, you could store the files elsewhere (network share or cloud share) and have SharePoint Search index them. In this case you loose out on a consistent UI for managing content and you'd still need the SP hosting team's help to setup the search crawling.
The real answer is to work with your customer to document their business needs and why IT's offering doesn't meet that need so that they can give you more space.
First time using Azure. I have a basic node js bot built with Microsofts Bot Framework, and deployed on Azure. What are my options for storage?
I will most likely just be needing simple key:value storage. Mongodb was my first though but I dont think Azure supports it nativeley.
That said, what are my options for storage on Azure? I usual shy away from MySQL just from preference, but theres no actual reason that wouldnt work either.
Take a look at Azure Table Storage for a NoSql solution
Table storage is a key/attribute store with a schemaless design. Because Table storage is schemaless, it's easy to adapt your data as the needs of your application evolve. Access to data is fast and cost-effective for all kinds of applications. Table storage is typically significantly lower in cost than traditional SQL for similar volumes of data.
I need to consider a database to store large volumes of data. Though my initial requirement is to simply retrieve chunks of data and save them in excel file, I am expecting more complex use cases for this data in future where the data will be consumed by different applications especially for analytics - hence need to use aggregated queries.
I am open to use either cloud based storage or on-premises storage. I am considering azure storage table (when there is a need to use aggregated data, I can have a wrapper service + cache around azure table storage but eventually will end up with nosql type storage) and on-premises MongoDB. Can someone suggest pros and cons of saving large data in azure table storage Vs on-premises MongoDB/couchbase/ravendb? Cost factor can be ignored.
I suspect this question may end up getting closed due to its broad nature and potential for gathering more opinions than fact. That said:
This is really going to be an app-specific architecture issue, dealing with latency and bandwidth, as well as the need to maintain on-premises servers and other resources. On-prem, you'll have full control of your hardware resources, but if you're doing high-volume querying against your database, from the cloud, your performance will be hampered by latency and bandwidth. Cloud-based storage (whether in MongoDB or any other database) will have the advantage of being neighbors with your app if set up in the same data center.
Keep in mind: Any persistent database store will need to back its data in Azure Storage, meaning a mounted disk backed by Blob storage. You'll need to deal with the 1TB-per-disk size limit (expanding to 16TB on an 8-core box via stripe), and you'll need to compare this to your storage needs. If you need to go beyond 16TB, you'll need to either shard, go with 200TB Table storage, or go with on-prem MongoDB. But... MongoDB and Table Storage are two different beasts, one being document-based with a focus on query strength, the other a key/value store with very high speed discrete lookups. Comparing the two on the notion of on-prem vs cloud is secondary (in my opinion) to comparing functionality as it relates to your app.
Microsoft Enterprise Library 5.0 Integration Pack for Windows Azure
As far as I understand, cache will be stored in database, and all web roles are sharing same cache which is same as Azure's in-memory distributed caching.
I'm wondering anyone has successfully used Caching Application Block on Windows Azure instead of Azure Caching.
If so, how is the speed? Any noticeably different from Azure Caching.
Thank you for comments!
The caching application block is deprecated (see here). It is replaced by the System.RunTime.Caching namespace. However, SqlDependency is not supported by SQL Azure, so you cannot use the caching namespace either for a database-driven cache.
So your choices boil down to: implement your own local cache by reading from SQL Azure yourself, use Windows Azure cache, or simply read from SQL Azure on demand.
The performance will be much better with the local cache because it is... local :). A background thread (that you need to write) will refresh your cache on a specific frequency. So accessing the local cache will be going at memory access speed. But... it will not be synchronized perfectly across your Windows Azure nodes, because each node will have its own cache. To work around this issue, you could code your refresh logic in a way that it is done at the same time across all your nodes.
Reading from the Azure Cache will be slower than accessing the local cache, but all your Windows Azure nodes will have access to the same data at the same time. You still need to build a thread that will update the Windows Azure cache on a specific frequency; it won't refresh itself automatically.
Warning: keep in mind that Azure Cache works well if you know in advanced the key of your key/value pair cache entry. In other words, you have no way of getting the list of keys from the Azure Cache. For example, if you store a list of Zip Codes / Temperature, the zip code would be your key (which is a known set). So you could use the Azure Cache for this purpose. The Azure Cache was designed to help store ASP.NET session state, which also has a known key (the Session ID).
Finally, keeping your data in SQL Azure (or another store, like an Azure Table) may work very well too. However that depends on how much traffic you get and how many times you will read from the data source. Remember that in some cases building a caching mechanism is be more work than necessary.
I am not very familiar with Enterprise Library. But Windows Azure Caching stores cache data in a remote server’s memory. When we request a cached data, we make a web request, and the server will return the data from its memory. If the cache server and our Windows Azure server are in the same data center (which is a recommended practice), the web transfer is extremely fast. If we use SQL Azure to store cached data, it takes about the same time for the data to transfer from SQL Azure server to our Windows Azure server. But it will take longer to query the data from the database compared to query the data from the cache server’s memory. SQL Azure may do some optimizations to increase data access, for example, cache the data in memory. But it is recommended you to use Windows Azure Cache over SQL Azure in this case.
Best Regards,
Ming Xu.
The Caching Application Block does not provide the features required for distributed caching. Instead, you should consider using one of the following technologies:
Windows Azure Caching. This enables you to provision a distributed
cache in the cloud. For more information, see Windows Azure Caching
Service.
Maintaining state and value in Windows Azure table storage.
Examples of how to implement this, and other useful resources, are
included in the Windows Azure Training Kit.
The caching mechanism of
the System.Runtime.Caching namespace of .NET Framework 4.0. For more
information, see System.Runtime.Caching Namespace.
The SQL Scripts which are compatible with the SQL Azure and are required for the Caching Database provider are available as a separate download. To get the scripts, download the EnterpriseLibraryIntegrationPack-WindowsAzure-sqlscripts.exe from the Microsoft Enterprise Library 5.0 Integration Pack for Windows Azure.
For information on using other classic Enterprise Library blocks in Windows Azure applications, see this white paper.
I am at the planning stage of a web application that will be hosted in Azure with ASP.NET for the web site and Silverlight within the site for a rich user experience. Should I use Azure Tables or SQL Azure for storing my application data?
Azure Table Storage appears to be less expensive than SQL Azure. It is also more highly scalable than SQL Azure.
SQL Azure is easier to work with if you've been doing a lot of relational database work. If you were porting an application that was already using a SQL database, then moving it to SQL Azure would be the obvious choice, but that's the only situation where I would recommend it.
The main limitation on Azure Tables is the lack of secondary indexes. This was announced at PDC '09 and is currently listed as coming soon, but there hasn't been any time-frame announcement. (See http://windowsazure.uservoice.com/forums/34192-windows-azure-feature-voting/suggestions/396314-support-secondary-indexes?ref=title)
I've seen the proposed use of a hybrid system where you use table and blob storage for the bulk of your data, but use SQL Azure for indexes, searching and filtering. However, I haven't had a chance to try that solution yet myself.
Once the secondary indexes are added to table storage, it will essentially be a cloud based NoSQL system and will be much more useful than it is now.
Despite similar names SQL Azure Tables and Table Storage have very little in common.
Here are a two links that might help you:
Table Storage, a 100x cost factor
Fat Entities on Table Storage
Basically, the first question should wonder about is Does my app really need to scale? If not, then go for SQL Azure.
For those trying to decide between the two options, be sure to factor reporting requirements into the equation. SQL Azure Reporting and other reporting products support SQL Azure out of the box. If you need to generate complex or flexible reports, you'll probably want to avoid Table Storage.
Azure tables are cheaper, simpler and scale better than SQL Azure. SQL Azure is a managed SQL environment, multi-tenant in nature, so you should analyze if your performance requirements are fit for SQL Azure. A premium version of SQL Azure has been announced and is in preview as of this writing (see HERE).
I think the decisive factors to decide between SQL Azure and Azure tables are the following:
Do you need to do complex joins and use secondary indexes? If yes, SQL Azure is the best option.
Do you need stored procedures? If yes, SQL Azure.
Do you need auto-scaling capabilities? Azure tables is the best option.
Rows within an Azure table cannot exceed 4MB in size. If you need to store large data within a row, it is better to store it in blob storage and reference the blob's URI in the table row.
Do you need to store massive amounts of semi-structured data? If yes, Azure tables are advantageous.
Although Azure tables are tremendously beneficial in terms of simplicity and cost, there are some limitations that need to be taken into account. Please see HERE for some initial guidance.
One other consideration is latency. There used to be a site that Microsoft ran with microbenchmarks on throughput and latency of various object sizes with table store and SQL Azure. Since that site's no longer available, I'll just give you a rough approximation from what I recall. Table store tends to have much higher throughput than SQL Azure. SQL Azure tends to have lower latency (by as much as 1/5th).
It's already been mentioned that table store is easy to scale. However, SQL Azure can scale as well with Federations. Note that Federations (effectively sharding) adds a lot of complexity to your application. I'm also not sure how much Federations affects performance, but I imagine there's some overhead.
If business continuity is a priority, consider that with Azure Storage you get cheap geo-replication by default. With SQL Azure, you can accomplish something similar but with more effort with SQL Data Sync. Note that SQL Data Sync also incurs performance overhead since it requires triggers on all of your tables to watch for data changes.
I realize this is an old question, but still a very valid one, so I'm adding my reply to it.
CoderDennis and others have pointed out some of the facts - Azure Tables is cheaper, and Azure Tables can be much larger, more efficient etc. If you are 100% sure you will stick with Azure, go with Tables.
However this assumes you have already decided on Azure. By using Azure Tables, you are locking yourself into the Azure platform. It means writing code very specific to Azure Tables that is not just going to port over to Amazon, you will have to rewrite those areas of your code. On the other hand programming for a SQL database with LINQ will port over much more easily to another cloud service.
This may not be an issue if you've already decided on your cloud platform.
I suggest looking at Azure Cache in combination with Azure Table. Table alone has 200-300ms latencies, with occasional spikes higher, which can significantly slow down response times / UI interactivity. Cache + Table seems to be a winning combination, for me.
For your question, I want to talk about how to decide with logic choose SQL Table and which need to use Azure Table.
As we know SQL Table is a relational database engine. but if you have a big data in one table the SQL Table is not applicable, because SQL query get big data is slow.
At this time you can choose Azure Table, the Azure Table query is so fast then SQL Table for big data, for example, in our website, someone subscribed many articles, we make the article as feed to user, every user have a copy of article title and description, so in the article table there are lots of data, if we use SQL Table, each query execution maybe take more than 30 seconds. But in Azure Table get users article feed by PartitionKey and RowKey is so fast.
From this example you may know how to choose between in SQL Table and Azure Table.
I wonder whether we are going to end up with some "vendor independent" cloud api libraries in due course?
I think that you have first to define what your application usage funnels are. Will your data model be subjected to frequent changes or it is a stable one? You have to be able to perform ultra fast inserts and reads are not so complicated? Do you need advance google like search? Storing BLOBS?
Those are the questions (and not only) that you have to ask and answer yourself in order to decide if you are more likely going to use NoSql or SQL approach in storing your data.
Please consider that both approaches can easily coexist and can be extended with BLOB storage as well.
Both Azure Tables and SQL Azure are two different beasts.Both are meant for different scenarios, one con to azure table is that you cannot move from azure to any other platform, unless you write providers in your code that can handle such shifts.