There is a collection with this structure:
{
"user": 1,
"timestamp": 216354113151,
"message": "asddsaasddsaa",
"data": [
"name = Jack & hash = s5w464t35w145df13s5df4sdg & id = 2"
"name = Mansonack & hash = xga5fd7h68745v46ed2 & id = 18"
]
}
I need to find an element of this collection, in the key data has a value that contains a string that contains the substring xga5fd7h68745v46ed2
Can I write this query? How will it look like?
You can loop over the array items like over any other list:
FOR d IN collection_name
FILTER IS_LIST(d.data)
FOR data IN d.data
FILTER CONTAINS(data, "xga5fd7h68745v46ed2")
RETURN d
If your data array has multiple occurances of substring you are searching for and want to return only the unique documents that contains search value, then use query like this:
// what you are looking for in string
LET searchValue = 'xga5fd7h68745v46ed2'
// go through items of your collection
FOR item IN collection_name
// store result of subquery in temporary variable
LET wasItemFound = FIRST((
// go through elements of your data array
FOR str IN item.data
// search for desired value within the element
FILTER CONTAINS(str, searchValue)
// we need to find only single occurance of searchValue
LIMIT 1
// return true if it was found
RETURN true
))
// give me only items which contained value you searched for
FILTER wasItemFound == true
RETURN item
Query which pluma suggested might do the job for your usecase if values you are looking for are all unique for all data array elements. If there are multiple occurances of substrings you search for within data array elements then the query will return duplicate documents. For example consider this example dataset:
[
{
"user": 1,
"data": [
"name = Jack & hash = abc & id = 2",
"name = Mansonack & hash = abcd & id = 18"
]
},
{
"user": 2,
"data": [
"name = Jack & hash = abb & id = 2",
"name = Mansonack & hash = abc & id = 18",
"name = xxx& hash = aaa & id = 18"
]
}
]
If you use this query
FOR d IN collection_name
FILTER IS_LIST(d.data)
FOR data IN d.data
FILTER CONTAINS(data, "ab")
RETURN d
It will return 4 documents even when the collection contains only 2. However the very first query in this post will return only 2. To be clear I'm not saying that pluma's solution is wrong it just depends what do you expect to be returned.
Related
When using terraform's for_each you have to specify a unique id to be used as a way to link the generated recource with its source definition.
I'd like to use a natural index for this, rather than an arbitrary unique value. In this case I'm working with DNS, so the natural index would be the DNS record name (FQDN)... Only that isn't always unique; i.e. you can have multipe A records for example.com to allow load balancing, or you may have multiple TXT records for providing verification to multiple vendors.
Is there a way to combine the natural index with a calculated value to provide a unique value; e.g. so we have the natural index followed by a 1 if it's the first time this value's seen, a 2 for the first duplicate, etc?
Specific Requirement / Context
I'm working on migrating our DNS records to be managed via IaC using Terraform/Terragrunt (this is for scenarios where the entries are manually managed, rather than those where the related service is also under IaC).
I'm hoping to hold the record data in CSVs (or similar) to avoid those managing the records day to day from requiring familiarity with TF/TG; instead allowing them to just update the data, and have the pipeline take care of the rest.
The CSV format would be something like this:
myId
RecordName
Type
Value
1
A
1.2.3.4
2
A
2.3.4.5
3
test
A
3.4.5.6
4
test
A
4.5.6.7
5
www
cname
example.com
Note: I'm considering each DNS Zone would have a folder with its name, and a CSV formatted as above which gives the records for that zone; so the above would be in the /example.com/ folder, and thus we'd have 2 A records for example.com, 2 for test.example.com and one CName for www.example.com which pointed to example.com.
locals {
instances = csvdecode(file("myDnsRecords.csv"))
}
resource aws_route53_zone zone {
name = var.domainname
provider = aws
}
resource aws_route53_record route53_entry {
for_each = {for inst in local.instances : inst.myId => inst}
name = "${each.value.RecordName}${each.value.RecordName == "" ? "" : "."}${var.domainname}"
type = each.value.Type
zone_id = aws_route53_zone.zone.zone_id
ttl = 3600
records = [each.value.Value]
}
I don't want the myId column though; as that doesn't add value / has no real relationship to the records; so if we were to remove/insert a record early in the CSV and renumber the following records it would result in a number of changes being required to records which hadn't really changed, just because their related "index" had changed.
I also don't want those working with these CSVs to have to manually manage such fields; i.e. I could provide another column and ask that they populate this as below... but that's asking for human error and adding complexity:
myId
RecordName
Type
Value
1
A
1.2.3.4
2
A
2.3.4.5
test1
test
A
3.4.5.6
test2
test
A
4.5.6.7
www1
www
cname
example.com
Question
Is there a way I can use a for_each loop with CSV data such as below, whilst working around the unique constraint?
RecordName
Type
Value
A
1.2.3.4
A
2.3.4.5
test
A
3.4.5.6
test
A
4.5.6.7
www
cname
example.com
You can add unique keys to the data structure:
locals {
instances = csvdecode(file("myDnsRecords.csv"))
instance_map = zipmap(range(0,length(local.instances)), local.instances)
}
resource "..." "..." {
for_each = local.instance_map
...
}
Terraform's for expressions when constructing a mapping have a "grouping mode" where it allows duplicate keys in return for the values of the map all being lists of potentially-multiple values that all had the same key.
I would therefore start by using that to project the CSV data into a map(list(map(string))) value where the keys are build from the record name and type, like this:
locals {
records_raw = csvdecode(file("${path.module}/myDnsRecords.csv"))
records_grouped = tomap({
for row in local.records_raw :
"${row.RecordName} ${row.RecordType}" => row...
})
}
The resulting data structure would be shaped like this:
records_grouped = tomap({
" A" = tolist([
{ RecordName = "", Type = "A", Value = "1.2.3.4" },
{ RecordName = "", Type = "A", Value = "2.3.4.5" },
])
"test A" = tolist([
{ RecordName = "test", Type = "A", Value = "3.4.5.6" },
{ RecordName = "test", Type = "A", Value = "4.5.6.7" },
])
"www CNAME" = tolist([
{ RecordName = "www", Type = "CNAME", Value = "example.com" },
])
})
Collecting the records with common keys into lists means that we now have a list index for each one that's unique only within the records with the common key.
So now we can project this one more time into a flat map of maps (map(map(string))) by incorporating those list indices into the map keys:
locals {
records = tomap(merge([
for group_key, group in local.records_grouped : {
for idx, record in group :
"${group_key} ${idx}" => group
}
]...))
}
This should produce a data structure like the following:
records = tomap({
" A 0" = { RecordName = "", Type = "A", Value = "1.2.3.4" }
" A 1" = { RecordName = "", Type = "A", Value = "2.3.4.5" }
"test A 0" = { RecordName = "test", Type = "A", Value = "3.4.5.6" }
"test A 1" = { RecordName = "test", Type = "A", Value = "4.5.6.7" }
"www CNAME 0" = { RecordName = "www", Type = "CNAME", Value = "example.com" }
})
That data structure is suitably-shaped for a for_each expression, so finally:
resource "aws_route53_record" "example" {
for_each = local.records
name = "${each.value.RecordName}${each.value.RecordName == "" ? "" : "."}${var.domainname}"
type = each.value.Type
zone_id = aws_route53_zone.zone.zone_id
ttl = 3600
records = [each.value.Value]
}
This will produce instance unique instance keys for each entry in the source CSV file while keeping all of the distinct (name, type) pairs separated so that you can add new ones without disturbing any existing records:
aws_route53_record.example[" A 0"]
aws_route53_record.example[" A 1"]
aws_route53_record.example["test A 0"]
...etc
You mentioned wanting a separate instance for each row in your CSV file but I also wanted to note that the aws_route53_record resource type is already designed to manage multiple records with the same name and type together, and so I think it would actually be fine to leave the records grouped together. (The name `aws_route53_record is a bit of a misnomer because each instance of this resource type manages a record set, not just a single record.)
Here's a variation that works that way:
locals {
records_raw = csvdecode(file("${path.module}/myDnsRecords.csv"))
record_groups = tomap({
for row in local.records_raw :
"${row.RecordName} ${row.RecordType}" => row...
})
recordsets = tomap({
for group_key, group in local.record_groups : group_key => {
name = group[0].Name
type = group[0].Type
values = group[*].Value
}
})
}
resource "aws_route53_record" "example" {
for_each = local.recordsets
name = "${each.value.name}${each.value.name == "" ? "" : "."}${var.domainname}"
type = each.value.type
zone_id = aws_route53_zone.zone.zone_id
ttl = 3600
records = each.value.values
}
This time the final map has one element per recordset instead of one element per record, after grouping all of the individual records together using their names and types. Now you don't need any synthetic indices at all because the name and type pair is the natural unique identifier for a Route53 recordset.
Does this get what you're looking for?
Dataset
RecordName,Type,Value
,A,1.2.3.4
,A,2.3.4.5
test,A,3.4.5.6
test,A,4.5.6.7
www,cname,example.com
main.tf
locals {
records = [for pref in {for _, key in distinct([for i, v in csvdecode(file("myDnsRecords.csv")): v.RecordName]): key => [for r in csvdecode(file("myDnsRecords.csv")): r if key == r.RecordName]}: {for i, r in pref: ("${r.RecordName}_${i}") => r}]
}
output "test" {
value = local.records
}
Output
Changes to Outputs:
+ test = [
+ {
+ _0 = {
+ RecordName = ""
+ Type = "A"
+ Value = "1.2.3.4"
}
+ _1 = {
+ RecordName = ""
+ Type = "A"
+ Value = "2.3.4.5"
}
},
+ {
+ test_0 = {
+ RecordName = "test"
+ Type = "A"
+ Value = "3.4.5.6"
}
+ test_1 = {
+ RecordName = "test"
+ Type = "A"
+ Value = "4.5.6.7"
}
},
+ {
+ www_0 = {
+ RecordName = "www"
+ Type = "cname"
+ Value = "example.com"
}
},
]
You can apply this plan to save these new output values to the Terraform state, without changing any real infrastructure.
Is it possible to order results of grouped items by the calculation results for each group? It seems that when I set the initialSort (or when I don't)it sorts by the order of the items within each group, rather than by the total calculation of each group.
For example, if I have data that looks something like this:
[
{id:1, company:"company 1", quantity:"10"},
{id:2, company:"company 1", quantity:"10"},
{id:3, company:"company 1", quantity:"10"},
{id:4, company:"company 2", quantity:"20"},
{id:5, company:"company 2", quantity:"1"},
{id:6, company:"company 2", quantity:"1"},
{id:7, company:"company 3", quantity:"9"},
{id:8, company:"company 3", quantity:"9"},
{id:9, company:"company 3", quantity:"9"},
]
I would end up with groups ordered:
Company 2: 22 // Highest qty 20
company 1: 30 // Highest qty 10
company 3: 27 // Highest qty 9
What I am trying to get is:
company 1: 30
company 3: 27
Company 2: 22
I can see the calculation results, but I'm not sure how to resort the groups, assuming it's possible. If anyone can point me in the right direction I will be quite grateful.
Rows can only be sorted according to a field stored in the row data.
Rows are sorted individually and then grouped, with groups appearing in the order of the sorted data.
In order to sort your table in this way, you would need to analyse your row data first before ingesting it into the table, and then set a field on each row with the desired order for the group that contains it.
The other option is to manually specify the groups you want using the groupValues option, with this approach you specify the exact groups you want and the order they should appear:
groupValues:[["male", "female", "smizmar"]]
Thanks Oli for pointing me in the direction of 'groupValues'. I ended up writing a function that uses the calcResults to get the sort order I want and then push them into groupValues to get the ordering I'm looking for.
const sortByGroupCalc = ( thisTable, thisSortField, thisSortDirection ) => {
// Temp arrays
let tempCalcArray = [];
let tempSortArray = [];
// Get calculation results
const calcResults = thisTable.getCalcResults();
// Populate Array with key:value pairs from the caculation results
for( const [key, value] of Object.entries(calcResults)) {
tempCalcArray.push([key, value.top[thisSortField]])
};
// Sort the object by value and direction and then create an array of the keys
tempCalcArray
.sort(function(a, b) {
if( thisSortDirection === 'asc' ) {
return a[1] - b[1];
} else {
return b[1] - a[1];
}})
.map(
x => tempSortArray.push(x[0])
);
// Set group order according to sort order
thisTable.setGroupValues([tempSortArray]);
}
I'm calling it in the dataSorted function. It's not perfect for all occasions, I imagine. But it seems to do what I need.
Thanks again.
I am using searchData() to count rows under different conditions for display to the user, without affecting the table itself. Specifically one col is "available" and I want to show how many rows there are depending on the filter criteria set, for both available and unavailable states. Most filters are normal default type header filters.
To do this I use the dataFiltered() callback. Inside the callback I get all the filters in a list, then do a count with and without an "available" filter added to the list. The problem is that I have one column with a custom header filter, and Tabulator complains if this one is active when searchData() is called. The custom filter works as expected in the table. Relevant code snippets are:
//table def
dataFiltered:function(filters, rows) { SetCounts(filters, rows) },
// col def
{title:"Sex", field:"sex", visible:true, hozAlign:"center", headerSort:false,
headerFilter:true, headerFilterFunc:customSexFilter,
},
// custom filter
function customSexFilter(headerValue, rowValue, rowData, filterParams) {
return (headerValue == rowValue) || (headerValue == "M" && rowValue == "N") || (headerValue == "F" && rowValue == "S");
}
// dataFiltered callback (simplified; one of two cases shown)
var availFilter = { field: "avail", type: "like", value: "A"};
function SetCounts(filters, rows) {
if (!table) return;
var filt = table.getHeaderFilters().concat(filters.slice());
var nAll = rows.length;
var fA = filt.slice();
fA.push( availFilter );
var nAvail = table.searchData(fA).length;
\$("#rowCount").html([nAll, '/', nAvail, 'A'].join(''));
}
The error that I get is Filter Error - No such filter type found, ignoring: function o(o)
and the filter itself shows as
field: "sex"
type:o(o)
value: "F"
I've studied the docs but can't figure out what I'm doing wrong. Or is this not supported? Version 4.6.
I'm designing a script that takes an object (jsonData[data]) and inputs its values into a different sheet based on which product it is.
Currently the script inputs all the data into a new row each time the form reaches a new stage, however the form goes through 4 stages of approval and so I'm finding each submission being entered into 4 different rows. Each submission has an "Id" value within the object which remains the same (but each submission could also be on any row in the sheet as it's used a lot).
I'm checking whether the ID exists in the sheet and using iteration to find the row number:
function updatePlatformBulkInfo(jsonData) {
var sheetUrl = "https://docs.google.com/spreadsheets/d/13U9r9Lu2Fq1WTT8pQ128heCm6_gMmH1R4O6u8e7kvBo/edit#gid=0";
var sheetName = "PlatformBulkSetup";
var doc = SpreadsheetApp.openByUrl(sheetUrl);
var sheet = doc.getSheetByName(sheetName);
var rowList = [];
var formId = jsonData["Id"];
var allSheetData = sheet.getDataRange().getValues();
setLog("AllSheetData = " + allSheetData[1][11]) //Logs to ensure data is collected correctly
var rowEdited = false;
var rowNumber = 0;
//Check whether ID exists in the sheet
for (var i = 0; i < allSheetData.length; i++) {
if(allSheetData[i][11] == formId) {
rowEdited = true;
} else {
rowNumber += 1;
}
}
My issue is with the next part:
//Append row if ID isn't duplicate or update row if duplicate found
if (rowEdited == false) {
for (var data in jsonData) {
rowList.push(jsonData[data])
}
setLog("***Row List = " + rowList + " ***");
setLog("***Current Row Number = " + rowNumber + " ***");
sheet.appendRow(rowList);
} else if(rowEdited == true){
var newRowValue = jsonData[data];
sheet.getRange(rowNumber, 1).setValues(newRowValue);
}
Everything works fine if the duplicate isn't found (the objects values are appended to the sheet). But if a duplicate is found I'm getting the error:
Cannot find method setValues(string)
This looks to me like i'm passing a string instead of an object, but as far as I'm aware I've already converted the JSON string into an object:
var jsonString = e.postData.getDataAsString();
var jsonData = JSON.parse(jsonString);
How can I modify my script to write the updated data to the matched row?
It's unclear based on your code whether or not you will actually write to the correct cell in the case of a duplicate. As presented, it looks as though you loop over the sheet data, incrementing a row number if the duplicate is not found. Then, after completing the loop, you write to the sheet, in the row described by rowNumber, even though your code as written changes rowNumber after finding a duplicate.
To address this, your loop needs to exit upon finding a duplicate:
var duplicateRow = null, checkedCol = /* your column to check */;
for(var r = 0, rows = allSheetData.length; r < rows; ++r) {
if(allSheetData[r][checkedCol] === formId) {
// Convert from 0-base Javascript index to 1-base Range index.
duplicateRow = ++r;
// Stop iterating through allSheetData, since we found the row.
break;
}
}
In both cases (append vs modify), you seem to want the same output. Rather than write the code to build the output twice, do it outside the loop. Note that the order of enumeration specified by the for ... in ... pattern is not dependable, so if you need the elements to appear in a certain order in the output, you should explicitly place them in their desired order.
If a duplicate ID situation is supposed to write different data in different cells, then the following two snippets will need to be adapted to suit. The general idea and instructions still apply.
var dataToWrite = [];
/* add items to `dataToWrite`, making an Object[] */
Then, to determine whether to append or modify, test if duplicateRow is null:
if(dataToWrite.length) {
if(duplicateRow === null) {
sheet.appendRow(dataToWrite);
} else {
// Overwriting a row. Select as many columns as we have data to write.
var toEdit = sheet.getRange(duplicateRow, 1, 1, dataToWrite.length);
// Because setValues requires an Object[][], wrap `dataToWrite` in an array.
// This creates a 1 row x N column array. If the range to overwrite was not a
// single row, a different approach would be needed.
toEdit.setValues( [dataToWrite] );
}
}
Below is the most basic solution. At the end of this post, I'll expand on how this can be improved. I don't know how your data is organized, how exactly you generate new unique ids for your records, etc., but let's assume it looks something like this.
Suppose we need to update the existing record with new data. I assume your JSON contains key-value pairs for each field:
var chris = {
id:2,
name: "Chris",
age: 29,
city: "Amsterdam"
};
Updating a record breaks down into several steps:
1) Creating a row array from your object. Note that the setValues() method accepts a 2D array as an argument, while the appendRow() method of the Sheet class accepts a single-dimension array.
2) Finding the matching id in your table if it exists. The 'for' loop is not very well-suited for this idea unless you put 'break' after the matching id value is found. Otherwise, it will loop over the entire array of values, which is redundant. Similarly, there's no need to retrieve the entire data range as the only thing you need is the "id" column.
IMPORTANT: to get the row number, you must increment the array index of the matching value by 1 as array indices start from 0. Also, if your spreadsheet contains 1 or more header rows (mine does), you must also factor in the offset and increment the value by the number of headers.
3) Based on the matching row number, build the range object for that row and update values. If no matching row is found, call appendRow() method of the Sheet class.
function updateRecord(query) {
rowData = [];
var keys = Object.keys(query);
keys.forEach(function(key){
rowData.push(query[key]);
})
var ss = SpreadsheetApp.getActive();
var sheet = ss.getSheets()[0];
var headers = sheet.getRange(1, 1, 1, sheet.getLastColumn()).getValues()[0];
var idColumn = 1;
var ids = sheet.getRange(2, idColumn, sheet.getLastRow() - 1, 1).getValues();
var i = 0;
var matchedRow;
do {
if (ids[i] == query.id) { matchedRow = i + 2; }
i++;
} while (!matchedRow && i < ids.length);
if (matchedRow) {
var row = sheet.getRange(matchedRow, idColumn, 1, rowData.length);
row.setValues([rowData]);
} else {
sheet.appendRow(rowData);
}
}
NOTE: if your query contains only some fields that need to be updated (say, the 'id' and the 'name' field), the corresponding columns for these fields will be
headers.indexOf(query[key]) + 1;
Possible improvements
If the goal is to use the spreadsheet as a database and define all CRUD (Create, Read, Write, Delete) operations. While the exact steps are beyond the scope of the answer, here's the gist of it.
1) Deploy and publish the spreadsheet-bound script as a web app, with the access set to "anyone, even anonymous".
function doGet(e) {
handleResponse(e);
}
function doPost(e) {
handleRespone(e);
}
function handleResponse(e) {
if (e.contentLength == -1) {
//handle GET request
} else {
//handle POST request
}
}
2) Define the structure of your queries. For example, getting the list of values and finding a value by id can be done via GET requests and passing parameters in the url. Queries that add, remove, or modify data can be sent as payload via POST request. GAS doesn't support other methods besides GET and POST, but you can simulate this by including relevant methods in the body of your query and then selecting corresponding actions inside handleResponse() function.
3) Make requests to the spreadsheet URL via UrlFetchApp. More details on web apps https://developers.google.com/apps-script/guides/web
I need to add multiple values (ID fields of another custom list) to a Multiple Lookup field using Sp-services.
What is the correct format of the data to be used ?
I have tried like ( 5,9,6 ) but only selecting the first item.
I am not quite sure why do you want to do this, because these items you will add - they can't be saved as values, because there is a relationship between the column and lookup list, i.e. if you have Lookup column to the List1 and you add a new value from the List2 with id 99 and save, it will save the reference to the list item with id 99 in the List1.
but if anything, it is possible, this is how I am appending multiple lookup selected values:
var lines = additionalTechnologies.split(';#');
$.each(lines, function (index) {
if (lines[index].length < 3) {
$("select[title='Additional Technologies selected values']:first").append("<option value=" + lines[index] + " title=" + lines[index + 1] + ">" + lines[index + 1] + "</option>");
$("select[title='Additional Technologies possible values'] option[value=" + lines[index] + "]").remove();
}
});
and remove them from the all items list. just do it vice versa.
I have found a way to do it.
// "list1Id" contains the array of LIST1 ID fields that you want to add...
// "MULTIPLELOOKUPFIELD" is the multiple lookup field in the LIST2...
var multipleLookupValue ="";
for(i = 0; i < list1Id.length ; i++)
{
multipleLookupValue = multipleLookupValue + list1Id[i]+";#data;#";
}
var method = "UpdateListItems";
$().SPServices({
operation: method,
async: false,
batchCmd: "New",
listName: "LIST2" ,
valuepairs: [["MULTIPLELOOKUPFIELD",multipleLookupValue]],
completefunc: function (xData, Status) {
//alert("Added new item to LIST2 list");
}
});
May be it will help someone...