I was reading some on the Prometheus time-series database, on what their core model was. I've never understood what exactly a time-series database was until then. Excited enough, I started wondering about Azure Application Insights. Sure enough, Application Insights is a very powerful tool and easy to use for .NET Core devs, especially ASP.NET Core. I've used it for years and like it a lot.
I haven't found any official docs (nor un-official) that states whether Application Insights is a time-series database. From looking at the portal, it's atleast presented as a "normal" relational storage.
This brings us to my question - anyone know what kind of database Application Insights actually is?
Thanks for your help!
I think the reality is more complicated (disclaimer: I'm a part of Azure Monitor team) =) There are many underlying databases to power various experiences. Metrics and Logs are powered by different solutions. Recently we released Azure Data Explorer (https://learn.microsoft.com/en-us/azure/data-explorer/). Essentially this is a backend which powers both Application Insights and Log Analytics.
Re: time-series. Azure Data Explorer stores all ingested data. Timestamp is one of columns.
Related
I am working on modernizing a reporting solution where the data sources are on prem on the customers' sql servers (2014) and the reports are displayed as Power BI reports on the customer's Power BI Service portal. Today I use SSIS to build a data warehouse, as well as an on premise data gateway to ensure the transport of data up to an Azure analysis services which in turn are used by the Power BI reports.
I have been wondering if I could use Azure Synapse to connect to customer data and in a most cost effective way transport data to Azure and link them to the Power BI workspace as a shared dataset. There are many possibilities, but it is important that the customer experiences that the reports are fast and stable, and if possible can cope with near real time.
I experience SSIS being cumbersome and expensive in azure. Are there mechanisms that make it cheap and fast to get data in azure? Do I need a data warehouse (Azure SQL database) or is it better to use data lake as storage for data? Needs to do incremental load too. And what if I need to do some transformations? Should I use Power BI dataflow or do I need to create Azure Data flows to achieve this?
Does anyone have good experience to use synapse (also with DevOps in mind) and get a good DEV, TEST and Prod environment for this? Or is using Synapse a cost driver and a simpler implementation will do? Give me your opinions and if you have links to good articles, please do so. Looking forward to hear from you
regards Geir
The honest answer is it depends on a lot of different things and I don't know that I can give you a solid answer. What I can do is try to focus down which services might be the best option.
It is worth noting that a Power BI dataset is essentially an Analysis Services database behind the scenes, so unless you are using a feature that is specifically only available in AAS and using a live connection, you may be able to eliminate that step. Refresh options are one of the things that are more limited in Power BI though, so the separate AAS DB might be necessary for your scenario.
There is a good chance that Power BI dataflows will work just fine for you if you can eliminate the AAS instance, and they have the added advantage of have incremental refresh as a core feature. In that case, Power BI will store the data in a data lake for you.
Synapse is an option, but probably not the best one for your scenario unless your dataset is large, SQL pools can get quite expensive, especially if you aren't making use of any of the compute options to do transformations.
Data Factory (also available as Synapse pipelines) without the SSIS integration is generally the least expansive option for moving large amounts of data. It allows you to use data flows to do some transformations and has things like incremental load. Outputting to a Data Lake is probably fine and the most cost effective, though in some scenarios something like an Azure SQL instance could be required if you specifically need some of those features.
If they want true real time, it can be done, but none of those tools really are built for it. In most cases the 48 refreshes per day (aka every 30 minutes) available on a Premium capacity are close enough to real time once you dig into the underlying purpose of a given report.
For true real time reporting, you would look at push and/or streaming datasets in Power BI and feed them with something like a Logic App or possibly Stream Analytics. There are a lot of limitations with push datasets though- more than likely you would want to set up a regular Power BI report and dataset and then add the real time dataset as a separate entity in addition to that.
As far as devops goes, pretty much any Azure service can be integrated with a pipeline. In addition to any code, any service or service settings can be deployed via an ARM template or CLI script.
Power BI has improved in the past couple years to have much better support for devops and dev/test/prod environments. Current best practices can be found in the Power BI documentation: https://learn.microsoft.com/en-us/power-bi/create-reports/deployment-pipelines-best-practices
I'm working on an Azure Application Insights dashboard for a system involving an InterSystems Cache database.
Is it possible to make Application Insights "aware" of a Cache database, in the way it's possible with our .NET code and other MS products on the server?
Lacking any quick-and-easy way, are there any Cache performance counters that might appear in the standard Windows Performance Monitor tool? (Because it's possible to gather these into AppInsights.)
Lacking both of these...are there any best practices otherwise?
I am using Azure Table storage to store the data of my application. I need to build an API to retrieve the data from Azure table storage. There are like a million records in the table. In the initial stage, my API would be getting approximately 100-1000 hits per day. What would be the best choice to develop that API on, API apps or Service Fabric?
It depends on a lot of aspects:
How do you want to maintain it?
What performance requirements do you have? Do all those 100-1000 hits per day happen at the same time or are they distributed across the day?
How do you want to handle scaling?
As described by #feranto the main difference is the architectural pattern. The decision will have a lot of implications for you, not only when it comes to development, but you will have to consider how to maintain your application.
Only based on your short description, it sounds like a relatively straightforward solution and not very complex nor very high performance/scale requirements. Given that, it might be easier for you to go for a more basic API App. Service Fabric is an excellent framework and architecture that allows us to build very flexible, highly scalable solutions, but it should be noted that that also comes with a lot more complexity when it comes to handling, monitoring and maintaining both the underlying service cluster and it's applications.
Setting up an API App, connecting to an Azure Storage table and control that with for instance Application Insights is relatively straight forward and you can find a lot of documentation and samples for that. As a contrast, Service Fabric is fairly nascent in documentation and samples, and you would have to read up on a lot to get a basic stable solution running if you are new to it.
The main difference between Azure API and Azure Service Fabric is the architecture pattern each one follows:
Azure API APPs supports a monolithic architecture. You have all your code running as one single package. You can develop this using Java, PHP, nodejs, python or .net.
Service Fabric follows a microservice architecture. Every service is deployed on a separate node, which can scale independently, and you can monitor the health of every service in every node. You can develop this using Java or C# mainly.
Here you can see advantages and disadvantages between microservices and a monolithic architecture.
I'm quite new in Development for Azure, I have a asp.net mvc 4 application in a Azure Cloud Service.
I have a application that has a considering quantity of transaction providing by API and I need to implement some applications loggings for improve the daily diagnostic, I'm looking for a tutorial that store those into a Blob Storage instead of SQL Database, but without relevant success.
Blob Storage sounds good because I don't need to increase substantially my database that also has all the business data and could compromise a business resource (Database) becoming slower because of log transactions.
If I decide to go to storage in SQL database I'm thinking in use Log4Net.
What you guys suggest and send me tutorial that I can follow, please.
Thank you.
Sorry our logging guidance is a little hard to find - something that we are currently working on resolving - but for now please take a look at the following resources:
Client logging overview - Essentially all client library operations are output using System.Diagnostics, so you intercept and write to text / xml file just using a standard TraceListener.
Analytics and Server logs - We have extensive service side logging capabilities as well - which troubleshooting distributed apps much simpler.
Let me know if you have any questions.
Jason
I have got two database (SQL azure) in North America, I'm getting error that the applications can't Access to the server.
I didn't update the application or database, so I supose that there are a problem with sql azure service. How can I notify Microsoft of this problem?
To 'notify' Microsoft, you have very limited options.
They have their own support forums.
They have their own support ticketing system, that costs a pretty penny but is the fastest way to get their attention.
They have their dashboard, which in my own experience is terrible. It is not a true representation, ever. The updates are very very late.
You also have StackOverflow - but there will be little that we can advise on if there is a problem on the Azure infrastructure side of life.
To help aid in your Azure support woes, I would suggest you get an account with Pingdom and get MetricHubs for your subscription. These will help in showing what goes down, when, how often, and for how long. It can help show if the problem really is in your application or not.
I would also ensure you have diagnostics set up, and log everything you can.
Many many people forget or don't know about the transient error problems. Microsoft have a huge article on it, but it does trip people up a lot.
The Azure Management Portal should also be able to give you a quick summary of if your roles & instances are actually up, healthy and stable.