I am planning for AWS QLDB for audit data.
Does QLDB support GDPR? Is there any performance impact to this?
There are some fields encrypted using custom encryption key before storing into QLDB. I might change the key down the line when the key gets compromised or for the key rotation policy. So, I may need to read all the old records, decrypt using old key and encrypt using new record and update again. Is this possible with QLDB?
How to do multi tenancy with QLDB? Like, I have multiple apps writing to audit and would like to have a virtual separation for each app in the same cluster.
thank you for the question; it touches some of the concepts that is at the heart of QLDB.
Does QLDB support GDPR? Is there any performance impact to this?
The QLDB developer guide page on data protection may help provide more information about the AWS shared responsibility model. It may also be helpful to read this AWS blog post about the shared responsibility model and GDPR.
We are currently working on a feature that will allow customers to remove the customer data payload from QLDB revisions. Many customers have asked for this feature in order to accommodate GDPR ”Right to forget“ requirements. Please do be aware that this is not a claim of ”compliance“ - as this is something you would need to evaluate independently. We do not anticipate this impacting any read/write performance. If you’re interested to know more about this, please reach out to AWS support and they’ll connect you with our team to tell you more about it.
There are some fields encrypted using custom encryption key before storing into QLDB. I might change the key down the line when the key gets compromised or for the key rotation policy. So, I may need to read all the old records, decrypt using old key and encrypt using new record and update again. Is this possible with QLDB?
Reading all the old records is possible in QLDB through a few different methods — querying revision history, exporting journal data, or streaming journal data.
However, it is worth noting that QLDB does provide encryption at rest via KMS. You can leverage KMS for key rotations or key expiry as well, and you’ll be able to access the old data with the new key via KMS’s key hierarchy. KMS will allow you to rotate keys without the need to reencrypt all your data.
How to do multi tenancy with QLDB? Like, I have multiple apps writing to audit and would like to have a virtual separation for each app in the same cluster.
There are a few potential ways to go about this, that ultimately may depend on your use-case(s). Within a single ledger you could leverage attributes in each document to differentiate between tenants. You could leverage multiple ledgers in QLDB in a single account with the default quota. It may also be the case that you want even more separation and may consider creating multiple accounts and leveraging something like AWS Control Tower.
All that said, the best approach could depend very heavily on your use-case(s), as well as other AWS products that you’re using. You may want to reach out to AWS support on this as well to potentially connect with the relevant Solutions Architect who could consult on approaches, given your specific use-case(s).
Related
-- I am exploring Azure functionality and am wondering if Azure Table Storage can be an easy way for holding application configuration for an entire environment. It would be easy to see and change (adding list values etc.). Can someone please guide me on whether this is a good idea? I would expect this table to hold no more than 2000 rows if all our applications were moved over to Azure.
Partition Key --> Project Name + Component Name (Azure Function/Logic App)
Row Key --> Parameter Key
Value column --> Parameter Value
-- For securing password/keys, I can use the Azure Key Vault.
There are different ways of storing application configurations:
Key Vault (as you stated) for sensitive information. Ex. tokens, keys, connection strings. It can be standardized and extended to any type of resources for ease of storing and retrieving these.
Application Settings, found under each App Service. This approach assumes you have an App Service for each of your app.
Release Pipeline, such as Azure DevOps Services (AzDo). AzDo has variables that can be global to the release pipeline or some that can be specific to each stages
I am exploring Azure functionality and am wondering if Azure Table
Storage can be an easy way for holding application configuration for
an entire environment. It would be easy to see and change (adding list
values etc.). Can someone please guide me on whether this is a good
idea?
Considering Azure Tables is a key/value pair store, it is certainly a good idea to store application configuration values there. Only thing I would recommend is that you incorporate some kind of caching layer between your application and table storage so that you don't end up making calls to table storage every time you need to fetch a setting.
I would expect this table to hold no more than 2000 rows if all our
applications were moved over to Azure.
Considering the number of entities is going to be less than 2000, I think your design would have no impact in querying the entities however I think your design is good. For best performance, please ensure that you're including both PartitionKey and RowKey while querying. At the very least, include PartitionKey in your query.
Please see this for more details: https://learn.microsoft.com/en-us/azure/cosmos-db/table-storage-design-guide.
For securing password/keys, I can use the Azure Key Vault.
That's the way to go for storing sensitive data in Azure.
Have you looked at the App Configuration service?
There are client libraries in .NET, Java, TypeScript and Python to interact with the service that you can leverage in your application.
We are going to have a new business system and I'm trying to convince my boss to host it on cloud in China cause business is there, ie: Azure, AWS, etc. He has a concern about data confidentiality and he doesn't want the company's financial info to leak out. The software vendor also suggested we build our own data center if we are so concern about data confidentiality. This makes me even more difficult to convince him. He has the impression that anything can be done in China.
I understand that Azure SQL is not an option for me cause host admin still have control even though I implement TDE (cannot use Always Encrypt). Now I'm looking at VM where I have full control over - at VM level up. I can also use disk encryption. Couple that with other security measures like SSL I'm hoping that this will improve the security of the data is it in transit or at rest. Is my understanding correct?
With that said, can the Azure admin still overwrite anything set on VM and take over the VM fully?
Even though it's technically possible but if this takes a lot of effort (benefit < effort) it still worth trying.
Any advice will be much appreciated.
Azure level Admin can just login to your VM, doesnt matter if its encrypted or not (or decrypt it, for that matter). You cannot really protect yourself from somebody inside your organization doing what he is not supposed to do (you can with to some extent with things like Privileged Identity Management, proper RBAC, etc).
If you are talking about Azure Fabric admin (so the person working for Microsoft or the chinese company in this particular case). He can, obviously pull the hard drive and get access to your data, but its encrypted at rest. Chances are he cannot decrypt it. If you encrypt the VM on top of that with Azure Disk Encryption (or Transparent Data Encryption) using your own set of keys he wouldn't be able to decrypt the data even if he can, somehow, get past the Azure side encryption
If you want to more control better to have IaaS services than PaaS services. You have more control on IaaS. You can use Bit loker to encrypt your disks if you are using Windows OS. China data center also under industry specific standards. Access to your customer data is controlled by an independent company in China, 21Vianet. Not even Microsoft can access your data without approval and oversight by 21Vianet. I think there is no big risk but you have to implement more security mechanism than Azure provide for better security.
I have REST Service hosted as AzureWeb App & Another Cloud-Service WorkerRole, both need to share few common info like DB Connection string / Storage Connection string Etc.,
What is the right way to do this?
Since your question is rather broad I will try to answer in a similar way - A good practice in distributed application and micro service architectures is to have services query a single store for their configuration by so allowing your configuration to be consistent and easily changed.
In these cases you would probably want to set up some kind of database known to all services as they initialize. Depending on how complex your config data is, you can decide between several options on Azure:
Easy, quick store for simple key value pairs such as strings: consider Azure Table Storage
For more complex document like configurations (e.g. JSON): consider DocumentDB
In some rare cases where latency and throughput is a concern and you might even want to consider an in-memory store such as Azure Redis cache, though mostly for configuration data this is an overkill.
Note that all of the suggested services above are Azure managed services meaning you get availability, redundancy and robustness out of the box. This is important since the configuration store you use can be a single point of failure in your system.
We want to enable Geo-Replication in Azure SQL Database. However for compliance reasons, we want to be sure that replication to secondary region happens over a secure encrypted channel.
Is there any documentation available to confirm that data in-transit during geo-replication goes over a secure encrypted channel?
I have looked into Microsoft Azure Trust center and there is a brief mention about using standard protocols for in-transit data. However I could not find information related to which protocols are used and how security of in-transit data is ensured.
Thank you for this question. Yes, the geo-replication uses a secure channel. If you are using V11 servers the SSL certificates are global and regularly rotated. If you are using V12 servers the certificates are scoped to the individual logical servers. This provides secure channel isolation not only between different customers but also between different applications. Based on this post I have filed a work time to reflect this in the documentation as well.
I have an ASP.NET MVC 2 Azure application that I am trying to switch from being single tenant to multi-tenant. I have been reviewing many blogs and posts and questions here on Stack Overflow, but am still trying to wrap my head around the specifics of what's right for this particular app.
Currently the application stores some information in a SQL Azure database, as well as some other info in an Azure Storage Account. I'm considering writing the tenant provisioning code to simply create a new database for a new tenant, along with a new azure storage account. This brings me to the following question:
How will I go about testing this approach locally? As far as I can tell, the local Azure Storage Emulator only has 1 storage account. I'm not sure if I'm able to create others locally. How will I be able to test this locally? Or will it be possible?
There are many aspects to consider with multitenancy, one of which is data architecture. You also have billing, performance, security and so forth.
Regarding data architecture, let's first explore SQL storage. You have the following options available to you: add a CustomerID (or other identifyer) that your code will use to filter records, use different schema containers for different customers (each customer has its own copy of all the database objects owned by a dedicated schema in a database), linear sharding (in which each customer has its own database) and Federation (a feature of SQL Azure that offers progressive sharding based on performance and scalability needs). All these options are valid, but have different implications on performance, scalability, security, maintenance (such as backups), cost and of course database design. I couldn't tell you which one to choose based on the information you provided; some models are easier to implement than others if you already have a code base. Generally speaking a linear shard is the simplest model and provides strong customer isolation, but perhaps the most expensive of all. A schema-based separation is not too hard, but requires a good handle on security requirements and can introduce cross-customer performance issues because this approach is not shared-nothing (for customers on the same database). Finally Federations requires the use of a customer identifyer and has a few limitations; however this technology gives you more control over performance distribution and long-term scalability (because like a linear shard, Federation uses a shared-nothing architecture).
Regarding storage accounts, using different storage accounts per customer is definitively the way to go. The primary issue you will face if you don't use separate storage accounts is performance limitations, such as the maximum number of transactions per second that can be executed using a single storage account. As you are pointing out however, testing locally may be a problem; however consider this: the local emulator does not offer 100% parity with an Azure Storage Account (some functions are not supported in the emulator). So I would only use the local emulator for initial development and troubleshooting. Any serious testing, including multitenant testing, should be done using real storage accounts. This is the only way you can fully test an application.
You should consider not creating separate databases, but instead creating different object namespaces within a single SQL database. Each tenant can have their own set of tables.
Depending on how you are using storage, you can create separate storage containers or message queues per client.
Given these constraints you should be able to test locally with the storage emulator and local SQL instance.
Please let me know if you need further explanation.