Very specific question, if I have the following input in my Streaming Analytics component:
//some input
"outputs": [
{
"name": "output1",
"unit": "unit1",
"value": "95813"
},
{
"name": "output2",
"unit": "unit2",
"value": "303883"
}, // and more array values
How can I get a JSON result that would look as follows:
"outputs":[ {
"output1":95813,
"output2":303883
//etc
}]
So, I don't need the unit value, and to save space, I'd like to use the 'name' as the key, and the 'value' as the value of the key-value array.
This is my current query:
SELECT
input.outputs as outputs
INTO "to-mongodb"
FROM "from-iothub" input
but this of course creates seperate JSON arrays, with the same structure as I do get as my input.
Anyone any idea on how to do this?
In worst case, just filtering out the 'unit' would also already be a great help.
Thanks in advance
You could use user-defined functions in Azure Stream Analytics. Please refer to the sample function I tested for you.
UDF:
function main(arg) {
var array = arg.outputs;
var returnJson = {};
var outputArray = [];
var map = {};
for(var i=0;i<array.length;i++){
var key=array[i].name;
map[key] = array[i].value;
}
outputArray.push(map);
returnJson = {"outputs" : outputArray};
return returnJson;
}
Query:
WITH
c AS
(
SELECT
udf.processArray(jsoninput) as result
from jsoninput
)
SELECT
c.result
INTO
jaycosmostest
FROM
c
Test Output:
Hope it helps you.
Related
I'm new to Node.js/AWS lambda. Ive successfully created several documentClient QUERY functions that return a single or multiple item JSON Document in this format:
[
{
"name": "andy",
"color": "purple",
"snack": "oreos"
}
]
When I use documentClient GET and get back my single record its in THIS format, which is not playing well with the client code (apple / ios swift)
{
"name": "andy",
"color": "purple",
"snack": "oreos"
}
I'm hopign i can change the format returned from documentClient.get() to include the fill JSON document format including leading and trailing brackets .. []
I am a node.js & aws.lambda and documentClient novice, so apologies if this is a very basic question....
provided in above text
If I understood well, you're receiving an object instead of a array.
You can use the scan function to retrieve an array of results:
var params = {
TableName : 'Table',
FilterExpression : 'Year = :this_year',
ExpressionAttributeValues : {':this_year' : 2015}
};
var documentClient = new AWS.DynamoDB.DocumentClient();
documentClient.scan(params, function(err, data) {
if (err) console.log(err);
else console.log(data);
});
Or you can transform the result to array:
const document = await documentClient.get({
TableName: "table-of-example",
Key: {
id: "id-of-example"
}
});
return [data]
Please read the document to understand more how the document dynamodb sdk works: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB/DocumentClient.html#scan-property
I have the below snippet from a JSON Object that has 3,500 records in it.
[
{
"use:firstName": "Bob",
"use:lastName": "Smith",
"use:categoryId": 36,
"use:company": "BobSmith",
"use:webExId": "Bob.Smith#email.com",
"use:address": {
"com:addressType": "PERSONAL",
"com:city": "US-TX",
"com:country": 1
}
},
{
"use:firstName": "Jane",
"use:lastName": "Doe",
"use:categoryId": 36,
"use:webExId": "Jane.Doe#email.com",
"use:address": {
"com:addressType": "PERSONAL",
"com:city": "US-CA",
"com:country": "1_1"
}
}
{
"use:firstName": "Sam",
"use:lastName": "Sneed",
"use:categoryId": 36,
"use:webExId": "Sam.Sneed#email.com",
"use:address": {
"com:addressType": "PERSONAL",
"com:city": "US-CA",
"com:country": "1_1"
}
}
]
I am using NodeJS and I have been stuck on figuring out the best way to:
1. Iterate through ['use:address']['com:city' to map out and identify all of the Cities. (In the example above, I have two: US-TX and US-CA in the three records provided)
2. Then identify how many records match each City (In the example above, I would have US-TX: 1 and US-CA: 2)
The only code I have is the easy part which is doing a forEach loop through the JSON data, defining userCity variable (to make it easier for me) and then logging to console the results (which is really unnecessary but I did it to confirm I was looping through JSON properly).
function test() {
const webexSiteUserListJson = fs.readFileSync('./src/db/webexSiteUserDetail.json');
const webexSiteUsers = JSON.parse(webexSiteUserListJson);
webexSiteUsers.forEach((userDetails) => {
let userCity = userDetails['use:address']['com:city'];
console.log(userCity);
})
};
I've been searching endlessly for help on the topic and probably not formulating my question properly. Any suggestions are appreciated on how to:
1. Iterate through ['use:address']['com:city' to map out and identify all of the Cities.
2. Then identify how many records match each City (In the example above, I would have US-TX: 1 and US-CA: 2)
Thank you!
You could reduce the webexSiteUsers array into an object that is keyed by city, where each value is the number of times the city occurs. Something like the below should work.
const counts = webexSiteUsers.reduce((countMemo, userDetails) => {
let userCity = userDetails['use:address']['com:city'];
if (countMemo[userCity]) {
countMemo[userCity] = countMemo[userCity] + 1;
} else {
countMemo[userCity] = 1;
}
return countMemo;
}, {});
counts will then be an object that looks like this.
{
"US-TX": 1,
"US-CA": 2
}
How can CosmosDB Query the values of the properties within a dynamic JSON?
The app allows storing a JSON as a set of custom properties for an object. They are serialized and stored in CosmosDb. For example, here are two entries:
{
"id": "ade9f2d6-fff6-4993-8473-a2af40f071f4",
...
"Properties": {
"fn": "Ernest",
"ln": "Hemingway",
"a_book": "The Old Man and the Sea"
},
...
}
and
{
"id": "23cb9d4c-da56-40ec-9fbe-7f5178a92a4f",
...
"Properties": {
"First Name": "Salvador",
"Last Name": "Dali",
"Period": "Surrealism"
},
...
}
How can the query be structured so that it searches in the values of Properties?
I’m looking for something that doesn’t involve the name of the
sub-propety, like SELECT * FROM c WHERE
some_function_here(c.Properties, ‘Ernest’)
Maybe I get your idea that you want to filter the documents by the value of the Properties, not the name. If so , you could use UDF in cosmos db.
sample udf:
function query(Properties,filedValue){
for(var k in Properties){
if(Properties[k] == filedValue)
return true;
}
return false;
}
sample query:
SELECT c.id FROM c where udf.query(c.Properties,'Ernest')
output:
Just summary here, Ovi's udf function like:
function QueryProperties (Properties, filedValue) {
for (var k in Properties) {
if (Properties[k] && Properties[k].toString().toUpperCase().includes(filedValue.toString().toUpperCase()))
return true;
return false;
}
Both of the following syntax's will work.
SELECT * FROM c where c.Properties["First Name"] = 'Salvador'
SELECT * FROM c where c.Properties.fn = 'Ernest'
so i am trying to query some data from a cosmos documentdb database as follows
here is my data :
{
"id": "**********",
"triggers": [
{
"type": "enter_an_area",
"trigger_identity": "***********",
},
},
{
"type": "enter_an_area",
"trigger_identity": "********",
},
},
{
"type": "exit_an_area",
"trigger_identity": "*******",
},
this is one document of my collection, where i have a document for every user with a unique ID, now what i want to do is count the number of users that use a specific trigger, a user may have the same trigger multiple times, as you can see in the example "enter_an_area" has more than one entery, but i would still want to count it as one entery.
i use this query to get the count for a specific trigger :
SELECT VALUE COUNT(1) FROM u JOIN t in u.triggers WHERE CONTAINS(t.type, "enter_an_area")
but for the example above this would return: 2 where i want it to return 1
is there a query to do this in documentdb? if there is no such a query, is there a way to return results without duplicates? because as a solution i thought i can return the IDs that use this specific trigger, but then i would get duplicate IDs when a user have more than one entery for a trigger.
It seems that your issue is about function like distinct the results of the query joins an array. I suggest you using stored procedure to implement your solution as a workaround.
Please refer to my sample stored procedure:
function sample() {
var collection = getContext().getCollection();
var isAccepted = collection.queryDocuments(
collection.getSelfLink(),
'SELECT u.id FROM u JOIN t in u.triggers WHERE CONTAINS(t.type, "enter_an_area")',
function (err, feed, options) {
if (err) throw err;
if (!feed || !feed.length) getContext().getResponse().setBody('no docs found');
else {
var returnResult = [];
var temp =''
for(var i = 0;i<feed.length;i++){
var valueStr = feed[i].id;
if(valueStr != temp){
temp = valueStr;
returnResult.push(feed[i])
}
}
getContext().getResponse().setBody(returnResult.length);
}
});
if (!isAccepted) throw new Error('The query was not accepted by the server.');
}
Hope it helps you.
I'm trying to retrieve all items from a DynamoDB table that match a FilterExpression, and although all of the items are scanned and half do match, the expected items aren't returned.
I have the following in an AWS Lambda function running on Node.js 6.10:
var AWS = require("aws-sdk"),
documentClient = new AWS.DynamoDB.DocumentClient();
function fetchQuotes(category) {
let params = {
"TableName": "quotient-quotes",
"FilterExpression": "category = :cat",
"ExpressionAttributeValues": {":cat": {"S": category}}
};
console.log(`params=${JSON.stringify(params)}`);
documentClient.scan(params, function(err, data) {
if (err) {
console.error(JSON.stringify(err));
} else {
console.log(JSON.stringify(data));
}
});
}
There are 10 items in the table, one of which is:
{
"category": "ChuckNorris",
"quote": "Chuck Norris does not sleep. He waits.",
"uuid": "844a0af7-71e9-41b0-9ca7-d090bb71fdb8"
}
When testing with category "ChuckNorris", the log shows:
params={"TableName":"quotient-quotes","FilterExpression":"category = :cat","ExpressionAttributeValues":{":cat":{"S":"ChuckNorris"}}}
{"Items":[],"Count":0,"ScannedCount":10}
The scan call returns all 10 items when I only specify TableName:
params={"TableName":"quotient-quotes"}
{"Items":[<snip>,{"category":"ChuckNorris","uuid":"844a0af7-71e9-41b0-9ca7-d090bb71fdb8","CamelCase":"thevalue","quote":"Chuck Norris does not sleep. He waits."},<snip>],"Count":10,"ScannedCount":10}
You do not need to specify the type ("S") in your ExpressionAttributeValues because you are using the DynamoDB DocumentClient. Per the documentation:
The document client simplifies working with items in Amazon DynamoDB by abstracting away the notion of attribute values. This abstraction annotates native JavaScript types supplied as input parameters, as well as converts annotated response data to native JavaScript types.
It's only when you're using the raw DynamoDB object via new AWS.DynamoDB() that you need to specify the attribute types (i.e., the simple objects keyed on "S", "N", and so on).
With DocumentClient, you should be able to use params like this:
const params = {
TableName: 'quotient-quotes',
FilterExpression: '#cat = :cat',
ExpressionAttributeNames: {
'#cat': 'category',
},
ExpressionAttributeValues: {
':cat': category,
},
};
Note that I also moved the field name into an ExpressionAttributeNames value just for consistency and safety. It's a good practice because certain field names may break your requests if you do not.
I was looking for a solution that combined KeyConditionExpression with FilterExpression and eventually I worked this out.
Where aws is the uuid. Id is an assigned unique number preceded with the text 'form' so I can tell I have form data, optinSite is so I can find enquiries from a particular site. Other data is stored, this is all I need to get the packet.
Maybe this can be of help to you:
let optinSite = 'https://theDomainIWantedTFilterFor.com/';
let aws = 'eu-west-4:EXAMPLE-aaa1-4bd8-9ean-1768882l1f90';
let item = {
TableName: 'Table',
KeyConditionExpression: "aws = :Aw and begins_with(Id, :form)",
FilterExpression: "optinSite = :Os",
ExpressionAttributeValues: {
":Aw" : { S: aws },
":form" : { S: 'form' },
":Os" : { S: optinSite }
}
};