This question may sound a little odd, but here it goes: A customer of ours would like to get access to certain metrics of his environment of our product which we host on Azure for the customer. It's a pretty complicated deployment, but in the end it consists of an Application Gateway, some virtual machines and a dedicated Azure SQL database.
The customer now would want to get select metrics from this deployment forward to their own DataDog subscription, e.g. VM CPU metrics, database statistics and those things. DataDog obviously supports all this information (which is good), but as a default would slurp in information from all resources within our subscription (which is not OK).
Is there a way to fine-granularly define which data is forwarded to DataDog, e.g. the resources and also which type of metrics to forward for each resource? What are my options here? Is it enough to create a service principal with a limited reading right, or can I configure this somewhere else? I am unfortunately not familiar with DataDog.
The main thing which must be prevented is that the customer due to the metrics forwarding could get access to other metrics in our subscription - we need to control the exact scope of the metrics.
The pretty straightforward solution to this issue is to create a service principal via command line, and then to assign the monitoring role to this service principal only exactly for the resources you need. This even works down to a level of specific databases for example.
Kicker: This is not possible to do in such a granularity from the UI, but the az command line accepts assigning the monitoring reader permission on a deep resource ID level, even if the UI for this is not there. By finding the resource ID from the UI, and then using the resource ID from the command line, it's possible to achieve exactly this behaviour.
Related
Kind of a simple question, but puzzling...
Is there a stat in Azure services to monitor how many times data factory is / was accessed ?
So, as an example if an automated system is set up to make persistent API calls to ADF with the malicious intent exhaust it is there a way to monitor for that and gather some kind of stats?
The monitoring built into the Azure Data Factory PaaS itself only monitors legitimate, authenticated usage. You can see this on the https://adf.azure.com/en/monitoring/pipelineruns?factory=%2Fsubscriptions%... dashboard.
Notice how the root domain is adf.azure.com - this is the same for all tenants using data factory around the world. Your specific subscription / instance are mere query parameters in the URL. Microsoft Azure is fully managing the actual hosting of this PaaS, which means they are entirely responsible for subverting any DDOS or similar bad-actor attempts on this service. It's not something you have to worry about, and therefore not something you have much visibility into.
If you ever needed or wanted to check in on how microsoft is doing with this, head on over to https://status.azure.com/status and search for the "Azure Data Factory" row:
This is really one of the biggest selling points of using a fully-hosted cloud PaaS such as Data Factory. You are no longer responsible for the hardware, or even range of ip addresses that back this service. No more than you have to worry about someone DDOS'ing outlook.office.com which probably services your entire organisation's email. I could happen, but if it did, it affects all of Microsoft's customers around the world, not just you personally, so there should be no expectation that you personally are doing anything special to mitigate against it.
Note that more generically if you want to monitor network traffic within your NSGs, iterfaces, VNETs etc in general on Azure, the thing to use is the Application Insights' Network Monitoring at https://portal.azure.com/#view/Microsoft_Azure_Monitoring/AzureMonitoringBrowseBlade/~/networkInsights
This is more generically applicable to all provisioned resources and services on Azure though, not something specific to Azure Data Factory.
We have a table in Azure Log Analytics that keeps the logs from many different systems.
For example, our CommonSecurityLog table has the logs from different Firewalls. I have created a custom RBAC role that allows access to this specific table only but would like to go further and limit the access to specific rows only.
I did some research but can't find a way to do this, is it possible?
There's no way to do this natively in Azure - RBAC only supports controlling access at the Table level.
EDIT:
So, as #FidelCasto mentioned, there's also the option of using Custom Logs. This will be helpful in many cases when you need to collect Custom Windows-related, Application-related. This could be a more user-friendly option but obviously there will be other cases where it will not apply, specially when you have devices sending non-standard logs.
If your requirements are not met by the option above, the only other catch-all option is to put a Log Collector between the firewalls and Azure, and use a script to filter the logs before sending them over via the Log Analytics (OpInsights) REST API. You could use a PowerShell script to handle this.
Each Firewall would send their logs to a local/remote Log Collector.
Have a script query/filter through the logs with If/Else based on the Firewall name.
For each Firewall, you would create a new Log-Type based on the Firewall name. Log-Type corresponds to the table name in Log Analytics.
Assign permission based on the newly created custom tables.
It's not as straight-forward but gets the job done!
I am working on a script to collect all resources and put them into one resource group - however, the command in Powershell used for moving resources works on a resource group by resource group basis. This means that if there are dependent/related resources in different resource groups, the command fails.
The alternative, then, is to group resources by their relation to one another and move them using some other method (probably manually through the portal/REST API.)
How can one then go through a list of resources in a subscription and group them by their dependency/relations?
IMHO one of the reason to introduce resource group is to try avoid such situation / scenario i.e., as per this Azure document, a resource group is a container that holds related resources for an Azure solution.
I believe a straight forward way or feature (to get list of dependent / related resources that are sitting in different resource groups in a subscription and group them by their dependency/relations) is currently not available or not supported. But I see this feature request raised in UserVoice / feedback forum as a related one so if interested, you may upvote it or create a new feedback in there. In general, responsible Azure product / feature team would triage / start checking feasibility and prioritizing a received feedback based on various factors like number of votes a feedback receives, feasibility, open prioritized backlog items, etc.
On the other hand, as a workaround to accomplish this requirement for now, I suggest to try coming up with an automation (to get list of dependent / related resources that are sitting in different resource groups in a subscription and group them by their dependency/relations) by leveraging Azure Platform logs (and fetching those logs at a particular time window of operation) as those logs provide detailed diagnostic and auditing information for Azure resources and the Azure platform they depend on.
Other related references:
Move resources to a new resource group or subscription
Move operation support for resources
Troubleshoot moving Azure resources to new resource group or subscription
We have Azure environment with 3 different subscription and around 5 project resources are deployed in this environment.
Each project team has rights to create resources under specific Resource Group (RG) within Azure.
Now from Azure Admin perspective, i would like to know Who, When
This is basic requirements for any organization to track their cost, resource information. When i looked in Azure, this information is not available directly at resource level.
Few posts are mentioning to use Tagging for this or use logs (2 years back, really?)? Is it? I am surprised.
Can i use Application Insight for this? or only available for App Service kind of services?
Please help me to get this information in efficient way
Your only option is to implement some sort of logging (like poll Azure Subscription events) and save it somewhere. You can use Azure Monitor to achieve that rather easily. But by itself Azure doesnt offer anything like that out of the box.
you can use tagging, but with obvious challenges. logs only go 3 months back.
I just read a small post of #pierreroman with the title Azure Cloud Service VS Azure Resource Manager. Unfortunately I was not able to comment his post (There was an error saving your comment. (Cannot create comment - access denied.)). Therefore and because I think this topic is interesting for many folks who are using azure cloud services right now, I write my "comment" here and send him a tweet. Maybe he or someone else can answer my questions.
I think that more interesting than the "what is the difference between cloud services and resource groups" is the question "should we plan to move from cloud services to azure resource groups?". Is it even possible? Or are we comparing apples to oranges?
What about scaling (for example adding new instances of a worker role with a simple slider or with auto scaling)?
The comparison is kind of apples(Cloud Service) and grocery(ARM) where can manage fruits, meats, and fishes.
But, very first concept of Cloud Service was similar to the ARM. That's why sometimes confusing.
Below quote is from free ebook Azure Web Apps for Developers (download), page 12 and 13.
An Azure Resource Group is a logical container for grouping Azure resources.
Grouping resources this way helps simplify the implementation, deployment, management, and monitoring of resources in the resource group. From a billing perspective, it gives you a way to view costs for the resource group rather than for individual resources, eliminating the need to figure out which resources are related. You can think of an Azure Resource Group as a unit of management.
The last line helps to understand clearly, Azure Resource Group is a unit of management.