I'm looking for some method to get report(*.csv) of CloudFront distributions.
Unfortunately, CloudFrontClient of AWS SDK doesn't support that function.(Or I could not find it.)
How can I get that informations by AWS SDK?
There are several reports that can be accessed in the CloudFront console.
The reports are derived from the CloudFront access logs, and access logging can be configured so that the raw logs are stored automatically in S3 where you can access them.
From the access logs, you can derive the same information is in the console reports, but the reports available in the CloudFront console can only be accessed from the CloudFront console. There does not appear to be a reporting API.
This is an unusual design for an AWS service, since most things you can do in the console are simply mapped to a documented API method, which the console is invoking in the background on your behalf. Here, that isn't the case.
I would prefer to have found a more convincing citation than the one that follows, in order to justify the assertions above, but I have not found what I was looking for. Note the omission of any mention of API access to the reports.
You can access all our reporting options by visiting the Amazon CloudFront Reporting & Analytics dashboard in the AWS Management Console.
You can also choose to receive even more information about the traffic delivered by your Amazon CloudFront distribution by enabling CloudFront access logs.
https://aws.amazon.com/cloudfront/reporting/
Related
I want to get cloud metrics of a service deployed in GCP for my python project but I can't find anything like boto3 library for AWS. Is there any google API to fetch metrics like CPU utilization, memory used etc.
There is Google Cloud SDK for Python. That is the equivalent of boto3. The difference is, as you can see from the link, instead of 1 single library, Google splits it into multiple libraries based on service. For example, for logging, the library is google-cloud-logging.
And of course they have APIs. This article from their docs explained the cloud logging API and even mentioned which Cloud Client Library to use.
We are using AWS for our infra requirement, and for billing and costing purpose we need to know the exact amount of data transferred to our EC2 instances for a particular client. Is there any such utility available in AWS or how should I approach this problem.
Our Architecture is simple we have a api server which is a Node.js® server on one of the ec2 instance, this talks to the db server which is a MongoDB® on another ec2, apart from this we also have a web application server which runs angular web application in Node.js® again.
Currently we don't use ELB and we Identified the client by there login informations i.e the organisation id in the JWT Token.
Given your current architecture, you will need to create some form of Node middleware that extracts the client ID and content-length from the request (and/or response) and writes them to persistent storage. Within the AWS ecosystem, you could write to DynamoDB, or Kinesis, or even SQS. Outside the AWS ecosystem you could write to a relational DB, or perhaps the console log with some form of log agent to move the information to persistent store.
However, capturing the data here has a few issues:
Other than logging to the console, it adds time to each request.
If logging to the console, there will be a time delay between the actual request and the time that the log is shipped to persistent storage. If the machine crashes in that interval you've lost data.
When using AWS services you must be prepared for rate limiting (this is one area where SQS is better than Kinesis or DynamoDB).
Regardless of the approach you use, you will have to write additional code to process the logs.
A better approach, IMO, would be to add the client ID to the URL and an ELB for front-end load distribution. Then turn on request logging and do after-the-fact analysis of the logs using AWS Athena or some other tool.
If you run these EC2 instances in VPC, you can use VPC Flow Logs to get insight into how much data each of the instances transfers.
Is Azure diagnostics only implemented through code? Windows has the Event Viewer where various types of information can be accessed. ASP.Net websites have a Trace.axd file at the root that can viewed for trace information.
I was thinking that something similar might exist in Azure. However, based on the following url, Azure Diagnostics appears to require a custom code implementation:
https://azure.microsoft.com/en-us/documentation/articles/cloud-services-dotnet-diagnostics/#overview
Is there an easier, more built-in way to access Azure diagnostics like I described for other systems above? Or does a custom Worker role need to be created to capture and process this information?
Azure Worker Roles have extensive diagnostics that you can configure up.
You get to them via the Role configuration:
Then, through the various tabs, you can configure up specific types of diagnostics and have them periodically transferred to a Table Storage account for later analysis.
You can also enable a transfer of application specific logs, which is handy and something that I use to avoid having to remote into the service to view logs:
(here, I transfer all files under the AppRoot\logs folder to a blob container named wad-processor-logs, and do so every minute.)
If you go through the tabs, you will find that you have the ability to extensively monitor quite a bit of detail, including custom Performance Counters.
Finally, you can also connect to your cloud service via the Server Explorer, and dig into the same information:
Right-click on the instance, and select View Diagnostics Data.
(a recent deployment, so not much to see)
So, yes, you can get access to Event Logs, IIS Logs and custom application logs without writing custom code. Additionally, you can implement custom code to capture additional Performance Counters and other trace logging if you wish.
"Azure diagnostics" is a bit vague since there are a variety of services in Azure, each with potentially different diagnostic experiences. The article you linked to talks about Cloud Services, but are you restricted to using Cloud Services?
Another popular option is Azure App Service, which allows you many more options for capturing logs, including streaming them, etc. Here is an article which goes into more details: https://azure.microsoft.com/en-us/documentation/articles/web-sites-enable-diagnostic-log/
We're using Role Based IAM credentials in our AWS VPC. This means that you never pass in keys to the client of the AWS SDK.
Previously we've used the PHP SDK. Amazon specifically recommends to cache the credentials when using role based authentication with the PHP SDK:
https://docs.aws.amazon.com/aws-sdk-php/guide/latest/credentials.html#caching-iam-role-credentials
I'm now writing a Node.JS application using the S3 client. I'm wondering if I need to cache the credentials (as per PHP SDK) or is this something that the Node.JS SDK automatically does for us?
The docs for the Node.JS SDK do not specifically mention anything about caching role based credentials:
http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/node-configuring.html#Credentials_from_IAM_Roles_for_EC2_Instances
Thanks
No, you do not need to cache IAM role credentials when using the AWS Node.js SDK.
I believe that the recommendation to cache credentials when using PHP is related to the request/CGI model that PHP uses. Without caching, your per-request PHP process would have to call out to the EC2 instance metadata service to retrieve credentials. Not ideal if you're handling high load.
With Node.js, you have a node process constantly running and it can persist credentials so will only need to call out to the EC2 instance metadata service once to retrieve initial credentials and then periodically to renew credentials when they are auto-rotated (every few hours, I believe).
As far as I can work out, unless you keep the client object around, the SDK will go back to the instance metadata service when it's instantiated again (except in the case where you instantiate a bunch of clients at the same time, in which case they all use the same instance metadata request event - odd).
i.e. cache your Amazon client objects (but this is not PHP, so you don't need an on-disk cache :) ).
I have a website hosted in Azure as a cloud service (not as a website), and I need to get the hit count for every web page of the site.
I enabled Azure Diagnostics, and I see the IIS logs copied to my blob storage, however this logs contain very few data (only one hit to a javascript file).
Furthermore, putting "Verbose" or "All" in the diagnostics configuration of the web role doesn't seem to affect the results, I get only one line (an access to a css file, or an image file, etc).
I'm using Azure SDK 2.0.
Is it possible to use the included IIS logs generated by azure to get a hit count? What should I need to change in the diagnostics configuration?
Or should I need a different approach to achieve this?
The IIS logs it produces are the same ones you'd find on a Windows Server anywhere. Note that depending on the settings you provided to the diagnostics it might take a little while before the data is moved to the storage account. Setting the level of verbosity for the configuration determines what is moved from the instances over to the storage account. Did you give it plenty of time to move the data over before looking at the file in storage again? Sometimes it just brings over what it has, and of course, there could be buffering which means when the file was brought over not everything was in it, etc.
You should be able to get this information from the logs, and yes, you should be able to do it from the IIS logs. That being said, if what you are after is a hits per page I would suggest actually a different approach. Look at an analytics provider like Google Analytics or one of the competitors to that. You'll get a massive amount of information beyond just page hits and no need to worry about parsing log files, etc.