bigQuery: PartialFailureError on table insert - node.js

I'm trying to insert data row to bigQuery table as follows:
await bigqueryClient
.dataset(DATASET_ID)
.table(TABLE_ID)
.insert(row);
But I get a PartialFailureError when deploying the cloud function.
The table schem has a name (string) and campaigns (record/repeated) fields which I created manually from the console.
hotel_name STRING NULLABLE
campaigns RECORD REPEATED
campaign_id STRING NULLABLE
platform_id NUMERIC NULLABLE
platform_name STRING NULLABLE
reporting_id STRING NULLABLE
And the data I'm inserting is an object like this:
const row = {
hotel_name: hotel_name,//string
campaigns: {
id: item.id,//string
platform_id: item.platform_id,//int
platform_name: item.platform_name,//string
reporting_id: item.reporting_id,//string
},
};
The errors logged don't give much clue about the issue.

These errors suck. The actual info about what went wrong can be found in the errors property on the PartialFailureError. In https://www.atdatabases.org we reformat the error to make this easier using: https://github.com/ForbesLindesay/atdatabases/blob/0e1a033264aac33deaff2ab753796049e623ab86/packages/bigquery/src/implementation/BigQueryDriver.ts#L211

According to my test it seems that there are 2 errors here. First is that you have campaign_id in schema while id in JSON.
2nd thing is related with format of REPEATED mode data in JSON. The documentation mentions following:
. Notice that the addresses column contains an array of values (indicated by [ ]). The multiple addresses in the array are the repeated data. The multiple fields within each address are the nested data.
It's not so straight in mentioned document (probably can be found somewhere else) however when you use REPEATED mode you should use brackets [].
I tested it shortly on my side and it seems that it should work like this:
const row = {
hotel_name: hotel_name,//string
campaigns: [ {
campaign_id: item.id,//string
platform_id: item.platform_id,//int
platform_name: item.platform_name,//string
reporting_id: item.reporting_id,//string
}, ]
};

Related

Bulk Insert records in Postgres using nodejs?

I am trying to do multiple insert row using node-postgres, I want id to be created dynamically and return those ids
Following approach, I am using
const insertArray = [['kunal#test.com',1000,'abcdef'],['kunal1#test.com',1000,'fedkcn']]
let query1 = format(`INSERT INTO users VALUES %L returning id`, insertArray);
const newClient = new Client();
await newClient.connect();
let {rows} = await newClient.query(query1);
I am getting errors as
invalid input syntax for type integer: "kunal#test.com",
how can we skip id?
Tried using CSV option via copyfrom also getting the same issue
const stream = client.query(copyFrom(`COPY users FROM STDIN (format csv, DELIMITER ',', HEADER)`))
const readStream = fs.createReadStream('./tmp/copy.csv');
copy.csv
"email","amount","address"
'kunal#test.com',1000,'abcdef'
USERS table schema
id email amount address
Integer. String. Integer. String.
Not sure how to auto-generate id and return the rows containing new id.
Cannot provide the col name also, as this functionality will also be used by other schemas.
If I provide id it works fine
Thanks for help
Inserting (either by insert or copy) without specifying column names is positional operational in order of column definition. In other words here insert into users values ... is the same as insert into users(id, email, amount, address) values .... However, your data does not have the same structure. The solution is to specify your columns:
... INSERT INTO users(email, amount, address) VALUEs...
... COPY users(email, amount, address) FROM STDIN ...

Get multiple documents from collection using nodejs and mongodb

Hi I have two mongodb collections. The first one returns json data (array) and with the output of this, I want to return documents that match.
When I run Console.log (req.bidder.myBids) I get the following output:
[{"productId":"3798b537-9c7b-4395-9e41-fd0ba39aa984","price":3010},{"productId":"3798b537-9c7b-4395-9e41-fd0ba39aa984","price":3020},{"productId":"4c4bd71c-6664-4d56-b5d3-6428fe1bed19","price":1040},{"productId":"4c4bd71c-6664-4d56-b5d3-6428fe1bed19","price":1050},{"productId":"4c4bd71c-6664-4d56-b5d3-6428fe1bed19","price":1060},{"productId":"4c4bd71c-6664-4d56-b5d3-6428fe1bed19","price":1070},{"productId":"4c4bd71c-6664-4d56-b5d3-6428fe1bed19","price":1090},{"productId":"4c4bd71c-6664-4d56-b5d3-6428fe1bed19","price":1100}]
The productId has duplicates, I want to remove duplicates and then call a routine that finds all the products that match and output as json.
So far I have this code that only outputs one document, but cant figure out how to add the array of productId's and then fetch all corresponding products.
var agencyId = req.body.agencyId;
var productId = req.body.productId;
if (!validate.STRING(agencyId)) {
res.apiError(messages.server.invalid_request);
} else {
dbProduct.find({productId:{$in:['3798b537-9c7b-4395-9e41-fd0ba39aa984','4c4bd71c-6664-4d56-b5d3-6428fe1bed19']}
}).then(dbRes => {
console.log(dbRes);
Updated code and works with hard-wired productId and updated above code. Looking at how to get the array data and transpose replacing the hard-wired productId's
The $in operator is what you want. See the docs here: https://docs.mongodb.com/manual/reference/operator/query/in/

Using Power Query with Excel, how do I replace a record filled will null values with a single null value?

Problem Summary
I have a column in my Power Query table which contains a custom linked data type. Creating a custom linked data type filled with all null values is not desired. Instead, if all the values contained in the custom data type is null, I would like the value in the column to be null.
Background
I have a table which holds API response JSON text. This JSON text contains a list of search results (also in JSON), representing movies which match search criteria delivered in the request. There can be any number of search results, including zero. Using Power Query M, I parse these JSON texts with the built-in parser, which generates a list containing one record per search result. I then extract the first record in the list, expand that record into new columns, and combine those new columns into a custom data type.
Example
Here is an example query simulating only the problem area of my query. This example is fully contained and can be used to reproduce my issue exactly.
let
// These two variables holds the API response JSON text obtained from calls to Web.Contents().
// I've eliminated the actual calls in this example because that part of my query works fine.
Search_Fast_and_Furious_Response =
"{ ""total-results"":""2"", ""results"":[
{ ""title"":""Fast & Furious"", ""year"":""2009"" },
{ ""title"":""The Fast and the Furious"", ""year"":""2001"" } ] }",
Search_mmmmm_Response =
"{ ""total-results"":""0"", ""results"":[] }",
// Create the table to hold the response text.
Source = Table.FromRecords( { [#"API Response"=Search_Fast_and_Furious_Response],
[#"API Response"=Search_mmmmm_Response] }),
// Parse the JSON and put the output (a record) in a new column.
#"Insert Parsed JSON" = Table.AddColumn(Source, "JSON", each Json.Document([API Response])),
// Expand the record in the parsed JSON column. Each field in the record becomes a new column.
#"Expand JSON" = Table.ExpandRecordColumn(#"Insert Parsed JSON", "JSON",
{"total-results", "results"}, {"Result Count", "Results List"}),
// Add a new column to hold the first search result in the responses results list.
// This is also a record, like the parsed JSON two steps ago.
#"Add Result #1 Column" = Table.AddColumn(#"Expand JSON", "Result #1", each
try _[Results List]{0}
otherwise null), // In case the list is empty
// Expand the record in the Result #1 column.
#"Expand Result #1" = Table.ExpandRecordColumn(#"Add Result #1 Column", "Result #1",
{"title", "year"}, {"Title", "Year"}),
// Combine the newly expanded columns into a single column.
// Make the Display Name be the value in the Title field/column,
// and make the Type Name be "Excel.DataType."
// This is what creates the custom linked data type.
#"Combine Result #1" = Table.CombineColumnsToRecord(#"Expand Result #1", "Result #1",
{"Title", "Year"}, [ DisplayNameColumn = "Title", TypeName="Excel.DataType" ])
in
#"Combine Result #1"
The list in the very last line before the in statement, i.e. the fourth parameter to the Table.CombineColumnsToRecord function, allows the record to be used as a custom data type used with Excel's new linked data feature. I'm not certain, but I believe Power Query/Excel stores them as records with additional metadata, such as DisplayNameColumn and TypeName (the latter of which I'm sure is the most important part).
Problem and Goal
Here is the resulting table created by the example query. The bottom-right cell is selected. Its contents are shown at the bottom of the image. The cell itself contains a value, specifically a record with all values set to null. Because the Title field is null, the record's display text is "null."
This next picture shows my desired output. Notice again the bottom-right cell. This time, the cell is empty. It no longer contains a record with all values being null; now it contains nothing, so the display shown in this view is null, italicized so as to indicate a null value as opposed to the word "null." (Note: I've been unable to change the "null" cell in the first image to a literal null value, so to demonstrate, I simply added a new column of null values.)
Unfortunately, because of my otherwise clause after the try, the column "Result #1" may be null if the API returned zero search results. If this value is null in any row, then all of the new columns created by #"Expand Result #1" will contain null in that row, also. Finally, when all the null values are combined in the last step, I'm left with a record with all null values. Instead, what I hope to achieve is to have a single null value (of type null) in that cell.
Efforts So Far
I have tried the Table.ReplaceValues function, passing null as the new value and many different values as the old value (the one to be replaced), such as a new record with all null values. All those attempts have either been syntactically incorrect or resulted in expected and unwanted behavior. I have also tried using the "Replace Values" option in the Power Query GUI, but the same result occurs. In case ReplaceValues didn't like nulls, I've also tried using a different value in the otherwise clause, such as "N/A" of type text, then doing a ReplaceValues on that different value. This yielded the same result.
Conclusion
Is there any way I can replace a record—which is filled with null values and is stored in a column containing records—with a singular null value? The linked data type feature is a high priority in this situation, so I would prefer a solution that retains that feature (though of course all solutions are welcome).
I have "solved" my problem. While not technically a solution to the question I posted, I've achieved the desired result using a workaround.
Instead of dealing with the object full of null fields, I ensure that object is not converted to the custom object to begin with. I achieve this by moving all records with a null value after extracting the first List item in the Results List column; this is done before I expand that extracted item. After putting the nulls in a new table (which I call the Null Table), I delete those nulls from the first table (which I call the Non-Null Table). I perform the regular operations on the Non-Null Table to create the custom linked data type for only those rows that were not null. Afterward, I merge the two tables together again.
The full code containing the solution with my representative example is below, with new steps "highlighted" with non-indented comments.
let
// These two variables holds the API response JSON text obtained from calls to Web.Contents().
// I've eliminated the actual calls in this example because that part of my query works fine.
Search_Fast_and_Furious_Response =
"{ ""total-results"":""2"", ""results"":[
{ ""title"":""Fast & Furious"", ""year"":""2009"" },
{ ""title"":""The Fast and the Furious"", ""year"":""2001"" } ] }",
Search_mmmmm_Response =
"{ ""total-results"":""0"", ""results"":[] }",
// Create the table to hold the response text.
Source = Table.FromRecords( { [#"API Response"=Search_Fast_and_Furious_Response],
[#"API Response"=Search_mmmmm_Response] }),
// Parse the JSON and put the output (a record) in a new column.
#"Insert Parsed JSON" = Table.AddColumn(Source, "JSON", each Json.Document([API Response])),
// Expand the record in the parsed JSON column. Each field in the record becomes a new column.
#"Expand JSON" = Table.ExpandRecordColumn(#"Insert Parsed JSON", "JSON",
{"total-results", "results"}, {"Result Count", "Results List"}),
// Add a new column to hold the first search result in the responses results list.
// This is also a record, like the parsed JSON two steps ago.
#"Add Result #1 Column" = Table.AddColumn(#"Expand JSON", "Result #1", each
try _[Results List]{0}
otherwise null), // In case the list is empty
// New step
// Filter down to only rows with null in the new column. Save this new table for later.
#"Filter In Null" = Table.SelectRows(#"Add Result #1 Column", each _[#"Result #1"] = null),
// New step
// Filter down to only rows with NOT null in the new column.
#"Filter Out Null" = Table.SelectRows(#"Add Result #1 Column", each _[#"Result #1"] <> null),
// Expand the record in the Result #1 column.
#"Expand Result #1" = Table.ExpandRecordColumn(#"Filter Out Null", "Result #1",
{"title", "year"}, {"Title", "Year"}),
// Combine the newly expanded columns into a single column.
// Make the Display Name be the value in the Title field/column,
// and make the Type Name be "Excel.DataType."
// This is what creates the custom linked data type.
#"Combine Result #1" = Table.CombineColumnsToRecord(#"Expand Result #1", "Result #1",
{"Title", "Year"}, [ DisplayNameColumn = "Title", TypeName="Excel.DataType" ]),
// New step
// Convert the Null Table into a list of records.
#"Convert Table" = Table.ToRecords(#"Filter In Null"),
// New step
// Append the Null Table from earlier to the main table.
#"Combine Tables" = Table.InsertRows(#"Combine Result #1", Table.RowCount(#"Combine Result #1"),
#"Convert Table")
in
#"Combine Tables"

Why am I unable to change values of a Mongoose object to a different type ‘directly’?

I've just spent a good hour figuring out something mind-boggling (at least to me, as a JS noob) and I'd like to understand the underlying logic (or just why it works this way, because I think it's illogical and quite unexpected).
Suppose I'm using Mongoose to retrieve documents from a database, some or all of which include a date property (created with new Date()), a numeric property, and a string property.
[{
string: 'foo',
date: '2018-10-13T21:11:39.244Z',
number: 10
},
...
{
string: 'bar',
date: '2018-10-13T21:12:39.244Z',
number: 20
}]
I thus obtain an array of objects and now want to take the date property for each object and change the value to a string, so I do something like:
doc.find({}, (err, list) => {
list.forEach((item, index) => {
list[index].date = 'new value'
})
})
But I can't do that!
I can do list[index].string = 'new value' as well as list[index].date = new Date() but I can't change values that are of a different type, in this example date and number.
However, when I do list[index]._doc.date = 'new value', which took so long to figure out because I didn't know Mongoose objects weren't just plain old objects and focused on solving problems I didn't have instead, I can modify the value just fine.
It appears that the mongoose object somehow translates obj.key to obj._doc.key only if the type of the value matches but I'd appreciate a more detailed explanation than my uneducated guesses.
I suppose you want to use multi type on a document field, Mongoose support this by "Mixed" type when you define the Schema.
You can get more detail from https://mongoosejs.com/docs/schematypes.html#mixed.

Inserting array into postgresql

I want to insert an array of id's to a file, but while creating a table I used int, and it has the reference of another table now how can I store array value to the specific field?
I understand that the columns have to be the same type and I also saw that some tried to tackle this foreign key on array issue already:
also i tried ELEMENT according to this doc but no success. please someone help me here https://blog.2ndquadrant.com/postgresql-9-3-development-array-element-foreign-keys/
db.query('CREATE TABLE IF NOT EXISTS departments_users(_id SERIAL NOT NULL UNIQUE PRIMARY KEY,user_id int REFERENCES companyUser ON DELETE CASCADE,department_id int REFERENCES departments ON DELETE CASCADE)');
exports.addCompanyUser = function(data, callback) {
db.query('INSERT INTO departments_users(user_id,department_id) VALUES($1,$2) RETURNING *', [companyuseraccess.rows[0]._id, data.department]).then(function(departments_users) {
})
}
You dont have any field with ARRAY dataType so essentially you cannot insert array into the field. If you wish to put in array in a field change its data type to ARRAY first and then you can
INSERT INTO "some_table" ("id","array_field","some_text_field")
VALUES (1,ARRAY['value1','value2']::TEXT[],'active') RETURNING *
Hope this helps. Let me know if i can help further

Resources