I have created a azure storage table and I need to get the last inserted row. I have tried using a filter with something like #equals(int(formatDateTime(item().date,'yyMMddHHmmss')),max(int(formatDateTime(item().date,'yyMMddHHmmss'))))
I am open to any ideas
If you are trying with ADF, kindly note that max() is not supported for azure table storage as specified in this MS Doc of Table Service Query Operators.
Two of the methods for getting the latest entry/row is using the date-time. For example, DateTime.MaxValue.Ticks - DateTime.UtcNow.Ticks
This is not good practice if the order is random based on the rowkey.
Another approach is using the Timestamp:
results.OrderByDescending(r => r.Timestamp).FirstOrDefault();
Refer to the SO Thread 1 & 2 for more information.
Related
I am having a below API
http://example.com/?mdule=API&method=Live.getLastVisitsDetails&filter_limit=2000&filter_offset=0
Have to extract more data by increasing the offset into 2000
http://example.com/?mdule=API&method=Live.getLastVisitsDetails&filter_limit=2000&filter_offset=2000
http://example.com/?mdule=API&method=Live.getLastVisitsDetails&filter_limit=2000&filter_offset=4000
http://example.com/?mdule=API&method=Live.getLastVisitsDetails&filter_limit=2000&filter_offset=6000
I have a maximum of 6000 records. I don't know how to pass the offset value of every 2000 in the data factory
I can able to extract individually with the above links but wanted to do it automatically for all 6000 records.
Can anyone show me some documentation or advise how to execute this in the datafactory?
I saw the pagination documentation, but no success
Extracting data using copy activity from rest API
Step1: Create a new pipeline and add a Copy Data activity.
Step2: Configure the source of copy activity, adding a pagination rule configured as below
Follow this sample URL and also make sure ={offset} at the end of URL.
Pagination rule either select option1 or option2 .In my case I selected Range:0:8:2
In your scenario you can follow the range as below :
Option1: QueryParameter.{offset}: Range:0:6000:2000
Option2: AbsoluteUrl.{offset}:Range:0:6000:2000
Range option are using 0 as the start value,6000 as the max value, and increasing by 2000 each time.
Storage account
For more detail information refer this official article.
I am using Azure Data Factory (ADF) to load data from Dynamics 365 to an Azure Database. However, for some columns I get a GUID value like 8234CCBA-7B01-E211-B551-D48564518CCA instead of the actual value.
Is there a way to retrieve the real data either by using SQL or changing something in Dynamics?
Normally we will use web api formatted values or CRM SDK methods to get the display name field of related table (Foreign key) in C# or javascript. In ADF, I don't think you can fetch the formatted values for picklist (Integer key values) or lookup (GUID) attributes.
Instead you can use ADF data flow to fetch the different datasets and join them for these kind of data transformations. Read more
Otherwise, dump all the datasets you need into Azure DB - for transformations later in DB views.
I'm trying to use ADF to create azure table storage tables from one source SQL table.
Within my pipeline..
I can query a distinct list of customers pass this into a for-each
task.
Inside the for each select the data for each customer
But when I try to to create an Azure table for each customers data with the table name based on the customer ID I hit errors.
The customer ID is a GUID so I'm trying to format this to remove the dashes which are invalid in a table name...
Something along the lines of
#replace('#{item().orgid}','-','')
So 7fb6d90f-2cc0-40f5-b4d0-00c82b9935c4 becomes 7fb6d90f2cc040f5b4d000c82b9935c4
I can't seem to get the syntax right
Any ideas?
try this: #replace(item().orgid,'-','').
Is it possible to update a column using Query in azure table storage ? The table has already 100 records and now I have added a new column. For the previous 100 records , the newly added column value is "null" as shown in the image. I just want to set a default value for all the 100 records. I need to know how to update the table.
There are several ways to edit columns in Azure storage, you can use the Update Entity functionality to do via Web requests, I'd recommend checking the documentation, Then you can also use Storage Explorer to modify single columns if you have enough permissions.
I was wondering if there is any way to programatically know if a table in Azure Table Storage has changed and then what has changed in the table?
No, not really. Your only option would be to go through all records in the table and see if the record changed based on the Timestamp property (even though the official documentation advises you not to read the Timestamp property).
Or have your code do the tracking each time an entity is added/updated/removed by writing this to an other table (an audit table).