I'm trying to using pagination rules in the source options for a DataFlow source that is using a DataSet which is using a REST linked service. Pagination works in this particular API where a header w/ a cursor value is returned, and that value is used in the next call's query parameter list: cursor=xxxx. It looks like the API is not having any of this because the cursors that they return aren't URI encoded, and I can't figure out how to set that up encoding in the pagination rule or anywhere else. I would assume that encoding would be automatic, but based on the number of results that I get back using both the ADF DataFlow and Postman, it appears that when I put a non-encoded cursor value in my parameter list, I get an empty result as well as an empty next-cursor header, and that seems to be what is ending the pagination.
So how do I encode URI values, either in the pagination rules of the DataFlow source, the REST DataSet or the REST Linked Service? There doesn't seem to be any available functions for that in the dynamic content option.
Related
While trying to build an ADF pipeline that generates datasets within Data Factory, I ran into an interesting issue. Or maybe I misunderstand some components completely, in which case I'd happily be educated.
I basically read some meta data from a SQL Database table which determines which source system, schema and tables I should pull new data from. The meta data is stored within a bunch of variables, which then feed a Web Request that attempts to generate a new Data Source as per the MS documentation. Yes, I'm trying to use Azure Data Factory to generate Azure Data Factory components.
The URL to create the DataSet and the JSON Body for the request are both generated using #Concat and a number of the variables. The resulting DataSet is a very straightforward file that does not contain references to the columns, but just the table schema and table name. I generated these manually before, and that all seems to work brilliantly. I basically have a dataset connected to the source system, referincing the table from the meta data.
The code runs, but the resulting dataset is directly published, as opposed to being added in my working branch. While this should not be a big issue once I manage to properly test everything, ideally the object would be created in my working branch (using Azure DevOps, thus a local file).
My next thought was to set up a linked service to my local PC, and simply write the same contents as above there. My challenge seems to be that I essentially am creating a file out of nothing. I am trying to use a Copy Data component, and added an empty placeholder file to act as a source.
I configure the sink with Dynamic Content for Copy Behavior, and attempted to add the JSON contents there. This gets the file created, but it's unfortunately empty. I also attempted to add a new column to the source with the data being the same contents.
However, seeing the file to be used as a sink doesn't exist, a mapping error will occur. Apart from this, I'd not want a column header to be written; just the dynamically created contents.
I'm not sure how to continue with this. I feel I'm very close to achieving my goal, but cannot seem to take this final hurdle.
Any hints or suggestions would be very welcome.
I have a Copy Data task which is obtaining data from an API
The API is a GET call to a method and requires 2 parameters
_token
Symbols
I have defined these as parameters
What is the syntax that allows me to use the values of my parameters as the values that are in the query string? So in the screenshot above Symbols is hard coded, but I want the value to be the value of the parameters
I need a screen solution rather than code please as I am not comfortable with ADF yet and I dont know how to get to the code/ARM views
Paul
Using a feature called string interpolation where expressions are wrapped in #{ ... }
Click on the Base URL field. Add Parameters. Using Concat expression function,
Example:
#{Concat('https://stackoverflow.com/questions/GetRealTimeRates?',linkedService().Symbols,'=',linkedService()._token)}
Add first parameter:
Add second parameter:
Test connection. If you see any error, it would provide a description as to debug further.
If I may have missed this in some other area of SO please redirect me but I don't think this is a duped question.
I am using Azure Search with an index field called Title which is searchable and filterable using a Standard Lucerne Analyzer.
When using the built-in explorer, if I want to return all Job Titles that are explicitly named Full Stack Developer I can achieve it this way:
$filter=Title eq 'Full Stack Developer'&$count=true
But if I want to retrieve all the Job Titles using a wildcard to return all records having Full Stack in the name this way:
$filter=Title eq 'Full Stack*'&$count=true
The first 20 or so records returned are spot on, but after that point I get a mix of records that have absolutely nothing in common with the Title I specified in the query. My initial assumption was that perhaps Azure was including my specified Title performing an inclusive keyword search on the text as well.
Though I found a few instances where that hypothesis seemed to prove out, so many more of the records returned invalidated that altogether.
Maybe I don't understand fully the mechanics under the hood of Azure Search and so though my query appears to be valid; my expectation of the result is way off.
So how should my query look to perform a wildcard resulting to guarantee the words specified in the search to be included in the Titles returned, if this should be possible? And what would be the correct syntax to condition the return to accommodate for OR operators to be inclusive?
Azure Cognitive Search allows you to perform wildcard searches limited to specific fields only. To do so, you will need to specify the name of the fields in which you want to perform the search in searchFields parameter.
Your search URL in this case would be something like:
https://accountname.search.windows.net/indexes/indexname/docs?api-version=2020-06-30&searchFields=Title&search=Full Stack*
From the link here:
I am using the Preview of the Azure Search Blob Indexer. All of the data that is indexed is contained in the metadata of the blobs. I have therefore set AzureSearch_SkipContent to 'true' to prevent content indexing.
I am having trouble getting DateTimeOffset fields to work, as I cannot figure out the correct format to store the DateTimeOffset value as.
When I create the indexer, I get an error response (400 Bad Request) with the response body holding the following content:
{"error":{"code":"","message":"Column 'ProcessDT' is of type String that is not compatible with the field of type Edm.DateTimeOffset in the index"}}
I have tried various string formats (the value on the blob I tested is
2015-01-03T11:13:00.000+01:00
so it should correspond to a format that is compatible to OData 4.0 and therefore azure search as stated here: https://msdn.microsoft.com/en-us/library/azure/dn798938.aspx
converting strings into DateTimeOffset field values is not currently supported. We recognize this is a painful limitation, and will address this in the near future. As a partial workaround, you can index those values as strings (so you can format them as you wish and still use them for search and filtering).
Thanks for bringing this up!
I'm trying to create a Projection Query that has a Title Part filter based on a property of the current content item - let's say the Display Text ({Content.DisplayText}) for example.
The query is not working and when I debug the solution and set a break point on the GetFilterPredicate method of the StringFilterForm class the token value is always coming through as an empty string. It's as if it's not being resolved. I am seeing the same behavior for all the tokens in the Content Items group.
Other tokens are working just fine, like the QueryString ({Request.QueryString:*}) token for example.
Why are the Content Item tokens always coming through empty in my query filter? I'm using Orchard v.1.7.1.0
This is not how it works. Tokens are not meant in this context to generate query constraints on the content item properties. Tokens can only be used there to provide values for parameters of the query. If you want to filter on the title of a content item, use a title part record filter.
2 years later, but in Orchard 1.9.2 you can query based on the current content item using the token:
{Request.Content.*}
I was missing the Request part of the token. Not sure if this worked at the time I originally asked the question.