Difference among the Azure CosmosDB Query Explorer's results and the Azure Functions results - azure

I executed a following query on the CosmosDB Query Explorer.
SELECT c.id, c.created_at FROM c
WHERE c.created_at_epoch <= 1499871600 - 86400*31
AND (CEILING(1499871600/86400) - CEILING(c.created_at_epoch / 86400)) % 31 = 0
That's result is followings.
[
{
"id": "70251cbf-44b3-4cd9-991f-81127ad78bca",
"created_at": "2017-05-11 18:46:16"
},
{
"id": "0fa31de2-4832-49ea-a0c6-b517d64ede85",
"created_at": "2017-05-11 18:48:22"
},
{
"id": "b9959d15-92e7-41c3-8eff-718c4ab2be6e",
"created_at": "2017-05-11 19:01:43"
}
]
It looks that problem does not exist.
Next, I replaced epoch values that defined statically to placeholders for using it as sqlQuery of the Azure Functions DocumentDB input bindings. Then, I replaced a modulations symbol to %modulationsymbol% for avoiding this issue
SELECT c.id, c.created_at FROM c
WHERE c.created_at_epoch <= {epoch} - 86400*31
AND (CEILING({epoch}/86400) - CEILING(c.created_at_epoch / 86400)) %modulationsymbol% 31 = 0
And I defined modulationsymbol = % as an application setting.
Then, I specified the function as follows.
// index.js
module.exports = function (context, myQueueItem) {
context.log(context.bindings.members, myQueueItem);
context.done();
};
// function.json
{
"bindings": [
{
"name": "myQueueItem",
"type": "queueTrigger",
"direction": "in",
"queueName": "myqueue",
"connection": "MYSTORAGE"
},
{
"type": "documentDB",
"name": "members",
"databaseName": "myproject-db",
"collectionName": "member",
"sqlQuery": "SELECT c.id, c.created_at FROM c WHERE {epoch} - c.created_at_epoch >= 86400*31 AND (CEILING({epoch}/86400) - CEILING(c.created_at_epoch / 86400)) %modulationsymbol% 31 = 0",
"connection": "MYCOSMOSDB",
"direction": "in"
}
],
"disabled": true
}
Afterward, I triggered the function and this result is followings.
2017-07-05T03:57:29.640 Function started (Id=d980521e-d23a-4bda-a730-57a236bcd011)
2017-07-05T03:57:30.594 [] { epoch: 1499871600 }
2017-07-05T03:57:30.594 Function completed (Success, Id=d980521e-d23a-4bda-a730-57a236bcd011, Duration=951ms)
It looks context.bindings.members is a empty list. It differenced from the CosmosDB Query Explorer's result.
Why appeared this differences?

After print out the {epoch}, I found its type is string and the expected type is number instead of string. That is the reason of getting empty list when use the same query.
To fix this issue, you could convert the type to number before using it to filter your query result. Steps below for your reference.
Step 1, Create a UDF which could convert string to number. Script Explorer->Create User Defined Function
function toNumber(ts) {
return parseInt(ts);
}
Step 2, After created the ConvertToNumber function, you could use it to convert the type of {epoch} to number.
SELECT c.id, c.created_at FROM c
WHERE c.created_at_epoch <= udf.ConvertToNumber({epoch}) - 86400*31
AND (CEILING(udf.ConvertToNumber({epoch})/86400) - CEILING(c.created_at_epoch / 86400)) %modulationsymbol% 31 = 0
If you are familiar with C#, you could create the function with C#. Since C# is a strongly typed language. We can define a class which is used to deserialize the message from queue. It will convert the type in language layer.
public class EpochMessage
{
public int epoch { get; set; }
}
The whole function could be like this.
using System;
public static void Run(EpochMessage myQueueItem, TraceWriter log, IEnumerable<dynamic> members)
{
log.Info(context.bindings.members, myQueueItem);
}
public class EpochMessage
{
public int epoch { get; set; }
}

Related

How can I return all nested objects using python?

I wrote an Elastic query which will check the condition (status="APPROVED") and Gets all approved_by objects.
This is my index (portfolio):
{
"settings": {},
"mappings": {
"portfolio": {
"properties": {
"status": {
"type": "keyword",
"normalizer": "lcase_ascii_normalizer"
},
"archived_at": {
"type": "date"
},
"approved_by": {
"id": "text",
"name":"text"
}
}
}
}
}
Currently I have 60 objects whose status are approved , so when i run the query it will show 60 objects,but in my case i am getting only one object(I debugged the code, total 60 objects are coming as expected, but still returning only single object), please help guys.
My query:
profiles = client.search(index='portfolio', doc_type='portfolio',
scroll='10m', size=1000,
body={
"query": {"match": {"status": "APPROVED"}}
})
sid = profiles['_scroll_id']
scroll_size = len(profiles['hits']['hits'])
while scroll_size > 0:
for info in profiles['hits']['hits']:
item = info['_source']
approved_by_obj = item.get('approved_by')
if approved_by_obj:
return (jsonify({"approved_by": approved_by_obj}))
Expected o/p format:
{
"approved_by": {
"id": "system",
"name": "system"
}
}
You're getting only one result because you're returning from your loop, effectively breaking out of it completely.
So, instead of returning from it, append the found approved_by_object to a list of your choice and then return that 60-member list:
profiles = client.search(index='portfolio', doc_type='portfolio',
scroll='10m', size=1000,
body={
"query": {"match": {"status": "APPROVED"}}
})
sid = profiles['_scroll_id']
scroll_size = len(profiles['hits']['hits'])
approved_hits_sources = [] # <-- add this
while scroll_size > 0:
for info in profiles['hits']['hits']:
item = info['_source']
approved_by_obj = item.get('approved_by')
if approved_by_obj:
approved_hits_sources.append({"approved_by": approved_by_obj}) # <--- append and not return
return jsonify({"approved_hits_sources": approved_hits_sources})

Defining a return for a very simple manual Azure function

I have an Azure function that does this:
public static int Run(int myvalue, TraceWriter log)
{
log.Info($"C# manually triggered function called with input: {myvalue}");
if (myvalue == 1) return 1;
return (myvalue + 1);
}
I created it directly in the portal; however, my attempt at the bindings are probably wrong:
{
"bindings": [
{
"type": "manualTrigger",
"direction": "in",
"name": "myvalue"
},
{
"type": "int",
"direction": "out",
"name": "$return"
}
],
"disabled": false
}
When I run it inside the portal, it gives me a 202 (accepted), but doesn't output the return value from the function.
My question is, basically, why am I not getting any output; however, I suppose my first question should be (is), should I be getting output; and if so, what's wrong with my binding?
This is a simplified version of a slightly more complex function, that I intend to use as a condition inside a logic app (hence why I need a return value).
int is not a supported type for manual trigger. You should be seeing an error like this in the logs:
[Error] Microsoft.Azure.WebJobs.Host: Error indexing method 'Functions.ManualTriggerCSharp1'. Microsoft.Azure.WebJobs.Script: Can't bind ManualTriggerAttribute to type 'System.Int32'.
If you change the parameter type to string, logging should work:
public static void Run(string myvalue, TraceWriter log)
{
log.Info($"C# manually triggered function called with input: {myvalue}");
}
Now, there is no point in returning anything from manual trigger. You can keep returning an int but it will be ignored. But remove the second binding:
{
"bindings": [
{
"type": "manualTrigger",
"direction": "in",
"name": "myvalue"
}
],
"disabled": false
}
To return a value, you probably need to switch to HTTP trigger, read the data from HTTP request and return HTTP response.

How to call azure function with params

I am trying to use Azure function with to invoke the same function with different time and different param(url).
I didn't find any good example that shows how to pass some data. I want to pass link to function.
My code is:
var rp = require('request-promise');
var request = require('request');
module.exports = function (context //need to get the url) {
and the function
{
"bindings": [
{
"name": "myTimer",
"type": "timerTrigger",
"direction": "in",
"schedule": "0 0 */1 * * *"
}
],
"disabled": false
}
If your settings are relatively static (but you still don't want to hard code them), you may use app settings to store them, and then read e.g.
let url = process.env['MyUrl'];
If URL should be determined per request, you may use HTTP trigger and read the URL from query parameters:
let url = req.query.myurl;
I'm not sure what exactly you are trying to achieve with parameterized timer-triggered function.
Another possibility is if your parameters are stored somewhere in e.g. Azure Document DB (Cosmos).
You could still use a TimerTrigger to invoke the function, and include a DocumentDB input binding allowing you to query for the specific parameter values needed to execute the function.
Here's a C# example triggered by Timer, with a DocumentDB input binding. Note: I'm using the latest VS2017 Preview tooling for Azure functions.
[FunctionName("TimerTriggerCSharp")]
public static void Run(
[TimerTrigger("0 */1 * * * *")]TimerInfo myTimer, TraceWriter log,
[DocumentDB("test-db-dev", "TestingCollection", SqlQuery = "select * from c where c.doc = \"Test\"")] IEnumerable<dynamic> incomingDocuments)
{..}
With the following binding json:
{
"bindings": [
{
"name": "myTimer",
"type": "timerTrigger",
"direction": "in",
"schedule": "0 */5 * * * *"
},
{
"type": "documentDB",
"name": "incomingDocuments",
"databaseName": "test-db-dev",
"collectionName": "TestingCollection",
"sqlQuery": "select * from c where c.docType = \"Test\"",
"connection": "my-testing_DOCUMENTDB",
"direction": "in"
}
],
"disabled": false
}

How can I merge the outputs from a For_Each loop in an Azure Logic App to a single flat array?

I have a For_Each loop in an Azure Logic App that calls another, nested, Logic App. The result from each iteration of the nested Logic Apps is a JSON object that contains an array of strings, like this:
{
"Results": ["string a", "string b"]
}
So the output from my For_Each loop in the parent Logic App looks like this:
[
{"Results": ["string a", "string b"]},
{"Results": ["string c", "string d"]}
]
I want to put all these strings into a single flat list that I can pass to another action.
How can I do this? Is it possible using the workflow definition language and built-in functions, or do I need to use an external function (in a service, or an Azure Function)?
There's a simpler solution, working with Array Variables.
At the top level, outside the For Each loop, declare a variable with an InitializeVariable action:
"Initialize_Items_variable": {
"inputs": {
"variables": [
{
"name": "Items",
"type": "Array",
"value": []
}
]
},
"runAfter": {},
"type": "InitializeVariable"
}
Inside the For Each, use a AppendToArrayVariable action. You can append the Response object of the Nested Logic App you just called.
"Append_to_Items_variable": {
"inputs": {
"name": "Items",
"value": "#body('Nested_Logic_App_Response')"
},
"runAfter": {
},
"type": "AppendToArrayVariable"
}
Hope it helps.
Picking up on #DerekLi's useful comment above, it seems this is not possible at the time of writing with Logic Apps schema version 2016-06-01.
One of the great strengths of Logic Apps is the ability to leverage the power of Azure Functions to solve problems like this that can't (yet) be solved in the schema language.
Re-writing the array is trivial in c# within a function:
using System.Net;
public class Result
{
public List<string> Results {get; set;}
}
public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, TraceWriter log)
{
log.Info("C# HTTP trigger function processed a request.");
var inputs = await req.Content.ReadAsAsync<List<Result>>();
var outputs = new List<string>();
foreach(var item in inputs)
{
log.Info(item.Results.ToString());
outputs.AddRange(item.Results.Where(x => !string.IsNullOrEmpty(x)));
}
return req.CreateResponse(HttpStatusCode.OK, outputs);
}
And this function can then be passed the result of the For_Each loop:
"MyFunction": {
"inputs": {
"body": "#body('Parse_JSON')",
"function": {
"id": "/subscriptions/{subscription-id}/resourceGroups/{resource-group-name}/providers/Microsoft.Web/sites/{function-app-name}/functions/{function-name}"
},
"method": "POST"
},
"runAfter": {
"For_each": [
"Succeeded"
]
},
"type": "Function"
}
There is also a way to do it using the workflow definition language. (https://learn.microsoft.com/en-us/azure/logic-apps/logic-apps-workflow-definition-language).
Using the fonctions string and replace you can work on your json as a string rather than on objects.
Here is a Flat_List action that follows a Parse_JSON action with your data:
Your data:
[
{"Results": ["string a", "string b"]},
{"Results": ["string c", "string d"]}
]
Flat_List component:
"Flat_List": {
"inputs": "#replace(replace(replace(string(body('Parse_JSON')),']},{\"Results\":[',','),'}]','}'),'[{','{')",
"runAfter": {
"Parse_JSON": [
"Succeeded"
]
},
"type": "Compose"
},
What happens here? First we use string that takes your json data and gives:
[{"Results":["string a", "string b"]},{"Results":["string c", "string d"]}]
We replace all the ]},{"Results":[ by ,.
We replace all the }] by }.
We replace all the [{ by {.
We get the string {"Results":["string a","string b","string c","string d"]}
Then you are free to parse it back to json with:
"Parse_JSON_2": {
"inputs": {
"content": "#outputs('Flat_List')",
"schema": {
"properties": {
"Results": {
"items": {
"type": "string"
},
"type": "array"
}
},
"type": "object"
}
},
"runAfter": {
"Flat_List": [
"Succeeded"
]
},
"type": "ParseJson"
}
You can see it as a proof of concept as the Azure Function may be easier to re-read later but there may be many reason not to want to instantiate a new Azure Function while you can do the job in Logic App.
Feel free to ask for more details if needed :)
This technique works pretty well, and only uses run-of-the-mill Logic App actions:
1. start with declaring an empty array variable (action Variable: Initialise variable)
2. iterate through your items (action Control: For each), e.g. the resultset from a previous action
in each iteration, first compose the JSON fragment you need (action Data Operations: Compose)
then append the output of your Compose action to the array (action: Variable: Append to array variable)
3. then, outside the loop, join the elements of the array (action Data Operations: Join)
4. do what you need with the output of the Join action, e.g. send as response payload (action Request: Response)
This is what it looks like in the end:
You can use #body(nestedLogicApp) outside of the for-each loop to access all the nested Logic Apps' response in an array.

Not Getting the Shape Right in DocumentDb Select

I'm trying to get only the person's membership info i.e. ID, name and committee memberships in a SELECT query. This is my object:
{
"id": 123,
"name": "John Smith",
"memberships": [
{
"id": 789,
"name": "U.S. Congress",
"yearElected": 2012,
"state": "California",
"committees": [
{
"id": 444,
"name": "Appropriations Comittee",
"position": "Member"
},
{
"id": 555,
"name": "Armed Services Comittee",
"position": "Chairman"
},
{
"id": 678,
"name": "Veterans' Affairs Comittee",
"position": "Member"
}
]
}
]
}
In this example, John Smith is a member of the U.S. Congress and three committees in it.
The result that I'm trying to get should look like this. Again, this is the "DESIRED RESULT":
{
"id": 789,
"name": "U.S. Congress",
"committees": [
{
"id": 444,
"name": "Appropriations Committee",
"position": "Member"
},
{
"id": 555,
"name": "Armed Services Committee",
"position": "Chairman"
},
{
"id": 678,
"name": "Veterans' Affairs Committee",
"position": "Member"
}
]
}
Here's my SQL query:
SELECT m.id, m.name,
[
{
"id": c.id,
"name": c.name,
"position": c.position
}
] AS committees
FROM a
JOIN m IN a.memberships
JOIN c IN m.committees
WHERE a.id = "123"
I'm getting the following results which is correct but the shape is not right. I'm getting the same membership 3 times. Here's what I'm getting which is NOT the desired result:
[
{
"id": 789,
"name": "U.S. Congress",
"committees":[
{
"id": 444,
"name": "Appropriations Committee",
"position": "Member"
}
]
},
{
"id": 789,
"name": "U.S. Congress",
"committees":[
{
"id": 555,
"name": "Armed Services Committee",
"position": "Chairman"
}
]
},
{
"id": 789,
"name": "U.S. Congress",
"committees":[
{
"id": 678,
"name": "Veterans' Affairs Committee",
"position": "Member"
}
]
}
]
As you can see here, the "U.S. Congress" membership is repeated 3 times.
The following SQL query gets me exactly what I want in Azure Query Explorer but when I pass it as the query in my code -- using DocumentDb SDK -- I don't get any of the details for the committees. I simply get blank results for committee ID, name and position. I do, however, get the membership data i.e. "U.S. Congress", etc. Here's that SQL query:
SELECT m.id, m.name, m.committees AS committees
FROM c
JOIN m IN c.memberhips
WHERE c.id = 123
I'm including the code that makes the DocumentDb call. I'm including the code with our internal comments to help clarify their purpose:
First the ReadQuery function that we call whenever we need to read something from DocumentDb:
public async Task<IEnumerable<T>> ReadQuery<T>(string collectionId, string sql, Dictionary<string, object> parameterNameValueCollection)
{
// Prepare collection self link
var collectionLink = UriFactory.CreateDocumentCollectionUri(_dbName, collectionId);
// Prepare query
var query = getQuery(sql, parameterNameValueCollection);
// Creates the query and returns IQueryable object that will be executed by the calling function
var result = _client.CreateDocumentQuery<T>(collectionLink, query, null);
return await result.QueryAsync();
}
The following function prepares the query -- with any parameters:
protected SqlQuerySpec getQuery(string sql, Dictionary<string, object> parameterNameValueCollection)
{
// Declare query object
SqlQuerySpec query = new SqlQuerySpec();
// Set query text
query.QueryText = sql;
// Convert parameters received in a collection to DocumentDb paramters
if (parameterNameValueCollection != null && parameterNameValueCollection.Count > 0)
{
// Go through each item in the parameters collection and process it
foreach (var item in parameterNameValueCollection)
{
query.Parameters.Add(new SqlParameter($"#{item.Key}", item.Value));
}
}
return query;
}
This function makes async call to DocumentDb:
public async static Task<IEnumerable<T>> QueryAsync<T>(this IQueryable<T> query)
{
var docQuery = query.AsDocumentQuery();
// Batches gives us the ability to read data in chunks in an asyc fashion.
// If we use the ToList<T>() LINQ method to read ALL the data, the call will synchronous which is why we prefer the batches approach.
var batches = new List<IEnumerable<T>>();
do
{
// Actual call is made to the backend DocumentDb database
var batch = await docQuery.ExecuteNextAsync<T>();
batches.Add(batch);
}
while (docQuery.HasMoreResults);
// Because batches are collections of collections, we use the following line to merge all into a single collection.
var docs = batches.SelectMany(b => b);
// Return data
return docs;
}
I just write a demo to test with your query and I can get the expected result, check the snapshot below. So I think that query is correct, you've mentioned that you don't seem to get any data when you make the call in my code, would you mind share your code? Perhaps there are some mistakes in you code. Anyway, here is my test just for your reference and hope it helps.
Query used:
SELECT m.id AS membershipId, m.name AS membershipNameName, m.committees AS committees
FROM c
JOIN m IN c.memberships
WHERE c.id = "123"
Code here is very simple, sp_db.innerText represents a span which I used to show the result in my test page:
var docs = client.CreateDocumentQuery("dbs/" + databaseId + "/colls/" + collectionId,
"SELECT m.id AS membershipId, m.name AS membershipName, m.committees AS committees " +
"FROM c " +
"JOIN m IN c.memberships " +
"WHERE c.id = \"123\"");
foreach (var doc in docs)
{
sp_db.InnerText += doc;
}
I think maybe there are some typos in the query you specified in client.CreateDocumentQuery() which makes the result to be none, it's better to provide the code for us, then we can help check it.
Updates:
Just tried your code and still I can get the expected result. One thing I found is that when I specified the where clause like "where c.id = \"123\"", it gets the result:
However, if you didn't make the escape and just use "where c.id = 123", this time you get nothing. I think this could be a reason. You can verify whether you have ran into this scenario.
Just updated my original post. All the code provided in the question is correct and works. I was having a problem because I was using aliases in the SELECT query and as a result some properties were not binding to my domain object.
The code provided in the question is correct.

Resources