I have used Data Factory wizard to copy Azure tables from one storage account to another storage account. Tables are huge with millions of entities and hundreds of partitions.
Now i want to make sure the copied tables are correct. Is there anyway i can compare integrity of tables between 2 storage accounts ? Does azure has any feature to do this ?
There's no built-in "compare tables" feature. This will be up to you to figure out how to do. I'm guessing you'll need to go partition-by-partition, comparing content. Assuming the content is the same, the order should be the same as well, but it would be an entity-by-entity comparison.
Maybe also consider, on the "write" end of the process, ensuring that you're doing one-for-one item copies?
Related
I have an Azure SQL database with many tables that I want to update frequently with any change made, be it an update or an insert, using Azure Data Factory v2.
There is a section in the documentation that explains how to do this.
However, the example is about two tables, and for each table a TYPE needs to be defined, and for each table a Stored Procedure is built.
I don't know how to generalize this for a large number of tables.
Any suggestion would be welcome.
You can follow my answer https://stackoverflow.com/a/69896947/7392069 but I don't know how to generalise creation of table types and stored procedures, but at least the metadata table of the metadata driven copy task provides a lot of comfort to achieve what you need.
I have a worker role running that creates tables in table storage, and I would like to be able to group these tables into categories like you would under a folder.
I cannot see any way to do this with the table classes in .Net, but when I look in my table storage 'Tables', I see a 'Metrics Table' entry which looks like a 'folder' and expands to show multiple metrics tables below it.
How can I create/add one of these myself programmatically?
Any ideas gratefully received?
I'm afraid this is not possible. Metric tables are handled differently by Visual Studio. They are not even returned when using Query Tables storage REST API (you can only use them directly by name). Tools like Azure Storage Explorer do not show them at all.
Back to your question. Best practice is to use common prefix for tables in same 'category'.
ex. WAD* for all azure diagnostics tables, NLog*for nlog tables.
Simple answer is that you can't. Table Storage Service contains tables and then each table contains entities. The functionality about Metrics Table you're talking about is a UI feature where the UI combines all these tables together.
http://news.ycombinator.com/item?id=5292591
This kind of f*** up can easily be done in Azure tables - whenever I use Cerebrata, I have an unnerving feeling that I might screw up things with one click!
What are the best-practices of taking backups in azure storage (both table storage and the blob-storage) so that it can be repatched if something unfortunate happens.
Kapil
As #Sandrino stated, lock down access to production data. Don't give out the storage account keys to anyone except the one or two people managing production rollouts. And if you must provide table access (say, for some type of emergency debugging), you can generate a Shared Access Signature for a table with just Query permissions.
Try to avoid accessing production databases, storage accounts, ... outside of your application. And set up backups, also for Table Storage.
You can use a commercial service like RedGate Cloud Services.
It's quite a topic, blobs vs tables vs SQL, and despite all I read so far I still can't find some proper reasoning on what to use when.
We have a multi-tenant SaaS web-application which we are about to move to Azure. We use an SQL Server 2008 database. We store documents and log information that belongs to the documents. Kinda like dropbox does.
The forums state that you better use Azure Tables when you are considering "large" objects. We typically store hundreds of documents per user where the size of the documents vary from 5kb to 30mb where the vast majority will be around 1MB?
Are there some ground rules when to go for Blobs, Tables, Sql? I already learned that I shouldn't store my documents in SQL since it is too expensive. But when does it get "beneficial" to store the documents in Blobs and when would I be better of with tables? Is there some kind of formula like :
if (objects * MB/object * objectrequested > y) then blobs, else tables
I think Igorek has addressed your SQL Azure concerns. You seem to still have questions about Tables vs Blobs, though.
In your case using Table storage would be annoying. Each property/column in ATS can be at most 64KB, so you would have to split the documents across multiple properties and then reassemble them. There is also a limit of 4MB per entity, which would be a problem. Blob storage has neither of these limitations.
I would tend to use Azure Table Storage when you have smallish entities with many properties that need to be stored and queried separately. So it works wells for stored objects, or small documents with lots of metadata.
Blob storage works better for things without a ton of metadata. It's good for things that might work well as files on a filesystem.
I would store documents themselves in the Azure Blob storage (not table storage). Outside of the fact that it is pretty expensive to store documents in a SQL Azure database that charges a penny per meg (or less depending on volume), SQL database is generally not a good place for documents. SQL is a relational database that provides benefits of ability to do queries, joins, etc. There is usually no benefit to storing large documents or images in a SQL database, especially when there is a highly scalable central storage system that is pretty cheap to store/access.
Now, if you need to search thru the documents themselves, I'd use something like Lucene.NET to provide a search capability for document-based repository.
HTH
I have an application that looks up data for a page. The data is looked up by primary key and row key in table storage.
I am considering SQL Azure storage. Is there some advantage in my going to this kind of storage being that the look up will always be very direct. Note that I do NOT need any reporting. ALL I want is single row look up
I am considering SQL Azure storage. Is there some advantage in my going to this kind of storage being that the look up will always be very direct. Note that I do NOT need any reporting. ALL I want is single row look up
Assuming that your requirements are fully stated as will only ever need single row access, and assuming that you only want to know about advantages and not disadvantages, then the only advantages I can think of are that SQL azure offers:
time-based subscription pricing instead of pricing per transaction
options for backup (in CTP)
options for replication/synchronisation
more client library options (e.g. Entity Framework, Linq2SQL, etc)
more data types supported
more options for moving your app outside of Azure if you ever want to
Use Table Storage if you don't need relational database functionality.