http://news.ycombinator.com/item?id=5292591
This kind of f*** up can easily be done in Azure tables - whenever I use Cerebrata, I have an unnerving feeling that I might screw up things with one click!
What are the best-practices of taking backups in azure storage (both table storage and the blob-storage) so that it can be repatched if something unfortunate happens.
Kapil
As #Sandrino stated, lock down access to production data. Don't give out the storage account keys to anyone except the one or two people managing production rollouts. And if you must provide table access (say, for some type of emergency debugging), you can generate a Shared Access Signature for a table with just Query permissions.
Try to avoid accessing production databases, storage accounts, ... outside of your application. And set up backups, also for Table Storage.
You can use a commercial service like RedGate Cloud Services.
Related
My Azure WebApp stores data in Azure Storage Tables and Blob storage.
There is a backup functionnality, but as I understood it just does not support azure tables/blobs... however I would like to automatically backup my tables to protect against accidental data corruption by users or by a software issue...
I would like to backup MyProdTables in MyBackupBlob container. Is there away to do it actually?
I read something about AZCopy, but it seems working with virtual machine's hard drives, but we have the WebApplication as a Service, so I am not sure that it will work in our case...
Edit: There is a partial (negative) MS feedback on the question, as mentioned in this answer, but it focused rather on migration and entire account snapshots. I am rather focused on the table storage, and maybe even the possibility of backuping individual tables... is strange that is nothing possible in this field, cause MS Azure Storage Explorer can easily backup the tables as CSV files.
There is no built-in backup feature for blobs and tables, as you've surmised. However: blobs do offer snapshots (a point-in-time snapshot may be taken at any time).
There are also Shared Access Signatures (and Policies) to limit exposure to your storage. And you can even protect the storage account itself from deletion.
As for AzCopy: that has nothing to do with VM disks. That's specifically for moving content in and out of blobs and tables.
I am beginning to use Azure Storage (blob specifically) in my application but wanted to know what the norm was in the case of testing versus production storage.
So is it routine to create one storage account? ie:
http:// <storage-account-name>.blob.core.windows.net/
and then have different containers for each environment? ie:
http://<storage-account-name>.blob.core.windows.net/testContainer
http://<storage-account-name>.blob.core.windows.net/productionContainer
so then it would end up looking like with populated data:
http://<storage-account-name>.blob.core.windows.net/testContainer/<whateverkey>
http://<storage-account-name>.blob.core.windows.net/productionContainer/<whateverkey>
or is should I be creating two different storage accounts? I had assumed that the connectionString generated was for just the storage account name and then later in my logic I would be specifying the containers and keys when adding data.
Thanks
There is no standard way, but... keep in mind: Azure storage isn't multi-level regarding subfolders (though the paths can be simulated). So, using containers to organize test vs production will hinder your ability to take advantage of conainers properly within your app (e.g. if you want /images/foo.png ... now you must have /productioncontainer/images/foo.png).
Remember that storage accounts are free: You pay only for storage used. So it costs nothing extra to have both a test and a production storage account. And then, the only thing that changes is the base address (storage account name).
You're correct regarding connection string: You just have accountname.blob.core.windows.net/container/object .
You should use different Storage Accounts - that way in addition to having storage isolation you can also ensure you have different security protection for accessing your development environment vs your production environment.
I have a Azure website that allows customers to upload documents. There are a lot of documenet (~200Gb so far).
I need a way to backup the documents to another location (Azure or other location), or have live replication to a server. Is there anything I can use that will do this?
Have you looked at Azure storage redundancy options? The geo redundant option might solve your replication need.
http://blogs.msdn.com/b/windowsazurestorage/archive/2013/12/04/introducing-read-access-geo-replicated-storage-ra-grs-for-windows-azure-storage.aspx
You can use Cloudberry. Here is the link to their website. The tool will not give you cloud storage, but will assist in backup process. It can back up either to the cloud or to the local storages.
Our team has Windows Azure MSDN - Visual Studio Premium subscriptions for all our devs. I have been taking advantage of the $100 per month allowance and am building more infrastructure in the cloud.
However, I would like other members of our team to access certain of the assets. I am quite new to the Azure infrastructure, so this might be a dumb question. But can they access my blobs? and can I control exactly who can access my blobs?
They can obviously RDP into my VMs, that's not an issue. I assume they can hit my VMs too, via the IP address, inside Azure, etc. However, I am more interested in the Blobs. Mostly because I am starting to upload a lot of utility data (large sample datasets, common software we all install, etc.) and I would like to avoid all of us having to upload all of it again for each subscriptions.
As of today (11/8/2013), you cannot "pool" MSDN resources meaning..have 4 subscriptions add up to $400/month and do ala carte cloud services
You can have one admin/or several for multiple subscriptions, this will allow you to view the different subscriptions in the portal and manage them in a single spot
You can also have different deployment profiles, so one Visual Studio instance can deploy to different Azure accounts.
Specific to your question, you have blob access keys and if you share the name of the storage account and key...yes they can access your data located there.
Yes, it is possible to control access to your blobs by using SAS (Shared Access Signatures)
SAS grants granular access to container, blob, table, & queue
This should be a good resource to start with :
Manage Access to Windows Azure Storage Resources
Create and Use a Shared Access Signature
However, I would like other members of our team to access certain of
the assets. I am quite new to the Azure infrastructure, so this might
be a dumb question. But can they access my blobs? and can I control
exactly who can access my blobs?
To answer specifically this question, Yes your team members can access the data stored in any blob storage account in any of your subscription. There are two ways by which you can provide them access to blob storage:
By giving them account name/account key: Using this, they get full access to storage account and essentially become owners of that storage account.
By using Shared Access Signature: If you want to give them restricted access to blob storage, you would need to use SAS as described by Dan Dinu. SAS basically gives you a URL using which users in possession of that URL can explore storage (by writing some code), however it is not possible to identify which user accessed which storage. For that you would need to write something on your own.
I think we have gone slightly wrong on the way we have used Azure storage in a SAAS system. We created a storage account per client (Securtiy was prime consideration) and containers per system area e.g. Vehicle, Work etc
Having done further reading it seems a suggestion would be that we should have used one account for all clients. Each client should have a container (so we can programmatically create it) which we then secure. Then files should just be structured using "virtual" folder structure e.g. Container called "Client A". Then Files for the Jobs (in Work area of system) stored like Work/Jobs/{entity id}/blah.pdf. Does this sound sensible?
If so we now have about 10 accounts that we need to restructure. Are there any tools that will let us easily copy one accounts contents to another containers account? I appreciate we probably can't move the files between accounts (as we set them up ages ago so can't use native copy function) so I guess some sort of copy. There are GB of files across all the accounts.
It may not be such a bad idea to keep different storage accounts per client. The benefits of doing that (to me) are:
Better security as mentioned by you.
You'll be able to achieve better throughput / client as each client will have their own storage account. If you keep one storage account for all clients, and if one client starts hitting that account badly other clients will be impacted.
Better scalability. Each storage account can hold up to 200 TB of data. So if you keep just one storage account and assuming each client consumes 100 GB of data, you'll be able to accommodate only 2000 clients (I hope my math is right :)). With individual storage accounts, you won't be restricted in that sense.
There're some downsides as well. Some of them are:
Management would be a nightmare. Imagining you have 2000 customers then you would end up managing 2000 storage accounts.
You may be limited by Windows Azure. Currently by default you get about 10 or 20 storage accounts per subscription and you would need to contact support to manually up that limit. They can do that for you but I would imagine you would want this to be a self-service model where you would be able to create as many storage accounts as you want without contacting support.
Now coming to your question about tooling, you could possibly write something on your own which makes use of Copy Blob functionality. This functionality allows you to copy blob data across storage accounts asynchronously. Basically this is what you would do is:
First create a blob container for each client in the target storage account.
Enumerate all blob containers in source storage account.
For each blob container in source storage account, enumerate the blobs.
Copy each blob asynchronously to target storage account in the client's blob container.
If you're a PowerShell fan, you can look into Cerebrata's Azure Management Cmdlets (http://www.cerebrata.com/Products/AzureManagementCmdlets) as well which wraps this functionality. I could have recommended Cerebrata's Azure Management Studio as well but I haven't tried this functionality just yet there [Disclosure: I'm one of the devs on Cerebrata team].
Hope this helps.
Adding to Gaurav Mantri answer...
You can have shared storage account for customers and use Shared Access Signature(SAS) to limiting access to particular container or blobs(as well as for tables and queues)...
http://msdn.microsoft.com/en-us/library/windowsazure/hh508996.aspx