I am having a below API
http://example.com/?mdule=API&method=Live.getLastVisitsDetails&filter_limit=2000&filter_offset=0
Have to extract more data by increasing the offset into 2000
http://example.com/?mdule=API&method=Live.getLastVisitsDetails&filter_limit=2000&filter_offset=2000
http://example.com/?mdule=API&method=Live.getLastVisitsDetails&filter_limit=2000&filter_offset=4000
http://example.com/?mdule=API&method=Live.getLastVisitsDetails&filter_limit=2000&filter_offset=6000
I have a maximum of 6000 records. I don't know how to pass the offset value of every 2000 in the data factory
I can able to extract individually with the above links but wanted to do it automatically for all 6000 records.
Can anyone show me some documentation or advise how to execute this in the datafactory?
I saw the pagination documentation, but no success
Extracting data using copy activity from rest API
Step1: Create a new pipeline and add a Copy Data activity.
Step2: Configure the source of copy activity, adding a pagination rule configured as below
Follow this sample URL and also make sure ={offset} at the end of URL.
Pagination rule either select option1 or option2 .In my case I selected Range:0:8:2
In your scenario you can follow the range as below :
Option1: QueryParameter.{offset}: Range:0:6000:2000
Option2: AbsoluteUrl.{offset}:Range:0:6000:2000
Range option are using 0 as the start value,6000 as the max value, and increasing by 2000 each time.
Storage account
For more detail information refer this official article.
Related
I have created a azure storage table and I need to get the last inserted row. I have tried using a filter with something like #equals(int(formatDateTime(item().date,'yyMMddHHmmss')),max(int(formatDateTime(item().date,'yyMMddHHmmss'))))
I am open to any ideas
If you are trying with ADF, kindly note that max() is not supported for azure table storage as specified in this MS Doc of Table Service Query Operators.
Two of the methods for getting the latest entry/row is using the date-time. For example, DateTime.MaxValue.Ticks - DateTime.UtcNow.Ticks
This is not good practice if the order is random based on the rowkey.
Another approach is using the Timestamp:
results.OrderByDescending(r => r.Timestamp).FirstOrDefault();
Refer to the SO Thread 1 & 2 for more information.
Is it possible to have data driven layout in Azure Workbooks?
For each row of a Kusto query, I'd like to add a tile to the dashboard.
There's no direct way to have a single query generate many visualizations in workbooks (yet!)
There are some options:
you could use a visualization like "tiles" which creates an item for each row, and you can configure the various parts of the tile to get info from different columns.
presuming you literally mean "Azure Dashboard" here when you say dashboard:
(more manual) you could create a parameter that returns all those values, and then configure a subsequent query to reference the selected value to generate the visualization you want, and manually pick a value, pin the chart to the dashboard, repeat for each value you want pinned.
(more technical) you could create the visualization you want, and pin it to a dashboard. then download the dashboard as json, and copy+paste the pin, but modify the query/parameter values in the dashboard, then upload it as a new dashboard?
I have added a schedule export for my Azure Resource Group Billing invoice on a monthly basis.
The invoice generated will consist of the billing details of the Last Month and will store the
.CSV file in my storage account as a Blob every month.
I'm using an Azure Logic app to retrieve the Invoice file and send it via mail to a group of
recipients.
The invoice is a .CSV file which consists of a number of columns like "InstanceID, MeterID,
UsageQuantity, ResourceLocation". But I need to get the TOTAL COST for the billing period.
Any idea how I can achieve this? Is there a specific column that I need to include in my CSV file. Or do I need to do some sort of data processing of the CSV file to get the total amount of resources consumed?
Any advise on this?
Thanks!
1. I created a csv file(named billing.csv) as below and upload it to blob storage.
InstanceID, MeterID, UsageQuantity, ResourceLocation, Pre tax cost
1,1,2,aa,10
2,2,3,bb,20
3,3,5,cc,30
2. In logic app, use "Get blob content" to get the csv file.
3. Search the action "Parse CSV" in you logic app.
4. The "Parse CSV" action will ask you to input "API Key", you need to go to this page first --> click "Start free trial", register an account and create a new API Key.
Copy the secret and paste it to your logic app as "API key", it will allow you to connect Plumsail.
5. Then choose the blob content into the "Parse CSV" and input the headers InstanceID, MeterID, UsageQuantity, ResourceLocation, Pre tax cost. Add a new parameter "Skip first line" and set its value as Yes.
6. Initialize a variable sum and set its value as 0 in integer type. Initialize another variable tempItem and also set its value as 0.
7. Use a "For each" loop.
The Body comes from "Parse CSV" action and the expression of "value" is: add(variables('tempItem'), int(items('For_each')?['Pre tax cost']))
8. After running the logic app, we can see the sum in last loop is:
9. Here is the whole logic app for your reference:
Import:
This solution uses the third party connector "Plumsail Documents", I'm not sure if it is free. I registered account in the past, it worked without any cost. But today the api key can't continue use, I need to register another account and create another api key. So I think this third party connector need extra cost if you want to use it for a long time.
Is it possible to update a column using Query in azure table storage ? The table has already 100 records and now I have added a new column. For the previous 100 records , the newly added column value is "null" as shown in the image. I just want to set a default value for all the 100 records. I need to know how to update the table.
There are several ways to edit columns in Azure storage, you can use the Update Entity functionality to do via Web requests, I'd recommend checking the documentation, Then you can also use Storage Explorer to modify single columns if you have enough permissions.
I have a requirement where i need to pass the column mapping dynamically from a stored procedure to the copy activity. This copy activity will perform update operation in Dynamics CRM. The source is SQL server (2014) and sink is Dynamics CRM.
I am fetching the column mapping from a stored procedure using look up activity and passing this parameter to copy activity.'
When i directly provide the below mentioned json value as default value to the parameter, the copy activity is updating the mapped fields correctly.
{"type":"TabularTranslator","columnMappings":{"leadid":"leadid","StateCode":"statecode"}}
But when the json value is fetched from the SP , it is not working . I am getting the error ColumnName is read only.
Please suggest if any conversion is required on the output of the loopup activity before passing the parameter to copy activity. Below is the output of the lookup activity.
{\"type\":\"TabularTranslator\",\"columnMappings\":{\"leadid\":\"leadid\",\"StateCode\":\"statecode\"}}
Appreciate a quick turnaround.
Using parameter directly and Using lookup output are different. can you share how did you write the parameter from the output of lookup actvitiy.
you can refer to this doc https://learn.microsoft.com/en-us/azure/data-factory/control-flow-lookup-activity