Bulk Insert records in Postgres using nodejs? - node.js

I am trying to do multiple insert row using node-postgres, I want id to be created dynamically and return those ids
Following approach, I am using
const insertArray = [['kunal#test.com',1000,'abcdef'],['kunal1#test.com',1000,'fedkcn']]
let query1 = format(`INSERT INTO users VALUES %L returning id`, insertArray);
const newClient = new Client();
await newClient.connect();
let {rows} = await newClient.query(query1);
I am getting errors as
invalid input syntax for type integer: "kunal#test.com",
how can we skip id?
Tried using CSV option via copyfrom also getting the same issue
const stream = client.query(copyFrom(`COPY users FROM STDIN (format csv, DELIMITER ',', HEADER)`))
const readStream = fs.createReadStream('./tmp/copy.csv');
copy.csv
"email","amount","address"
'kunal#test.com',1000,'abcdef'
USERS table schema
id email amount address
Integer. String. Integer. String.
Not sure how to auto-generate id and return the rows containing new id.
Cannot provide the col name also, as this functionality will also be used by other schemas.
If I provide id it works fine
Thanks for help

Inserting (either by insert or copy) without specifying column names is positional operational in order of column definition. In other words here insert into users values ... is the same as insert into users(id, email, amount, address) values .... However, your data does not have the same structure. The solution is to specify your columns:
... INSERT INTO users(email, amount, address) VALUEs...
... COPY users(email, amount, address) FROM STDIN ...

Related

bigQuery: PartialFailureError on table insert

I'm trying to insert data row to bigQuery table as follows:
await bigqueryClient
.dataset(DATASET_ID)
.table(TABLE_ID)
.insert(row);
But I get a PartialFailureError when deploying the cloud function.
The table schem has a name (string) and campaigns (record/repeated) fields which I created manually from the console.
hotel_name STRING NULLABLE
campaigns RECORD REPEATED
campaign_id STRING NULLABLE
platform_id NUMERIC NULLABLE
platform_name STRING NULLABLE
reporting_id STRING NULLABLE
And the data I'm inserting is an object like this:
const row = {
hotel_name: hotel_name,//string
campaigns: {
id: item.id,//string
platform_id: item.platform_id,//int
platform_name: item.platform_name,//string
reporting_id: item.reporting_id,//string
},
};
The errors logged don't give much clue about the issue.
These errors suck. The actual info about what went wrong can be found in the errors property on the PartialFailureError. In https://www.atdatabases.org we reformat the error to make this easier using: https://github.com/ForbesLindesay/atdatabases/blob/0e1a033264aac33deaff2ab753796049e623ab86/packages/bigquery/src/implementation/BigQueryDriver.ts#L211
According to my test it seems that there are 2 errors here. First is that you have campaign_id in schema while id in JSON.
2nd thing is related with format of REPEATED mode data in JSON. The documentation mentions following:
. Notice that the addresses column contains an array of values (indicated by [ ]). The multiple addresses in the array are the repeated data. The multiple fields within each address are the nested data.
It's not so straight in mentioned document (probably can be found somewhere else) however when you use REPEATED mode you should use brackets [].
I tested it shortly on my side and it seems that it should work like this:
const row = {
hotel_name: hotel_name,//string
campaigns: [ {
campaign_id: item.id,//string
platform_id: item.platform_id,//int
platform_name: item.platform_name,//string
reporting_id: item.reporting_id,//string
}, ]
};

Use value from a column as paramater for json request and combine the table

I am using power query to load some json data in a table (matches). I want to use a specific part of that data (fixture_id) as a parameter for another json request in another query (predictions), and then combine that output in my main (matches) table. Anyone can point me in the right direction on how to do this ?
So here is my matches table:
And then in my fixtures table i can maybe i have:
apiKey = Excel.CurrentWorkbook(){[Name="ApiKey"]}[Content]{0}[Column1],
fixtureID = "?",
Source = Json.Document(Web.Contents("https://v2.api-football.com/predictions/" & fixtureID, [Headers=[#"X-RapidAPI-Key"=apiKey]])),
If i hardcode the fixtureID, i get this output:
But i want to calculate it dynamically, and then merge the output to the matches table.
The first step is to turn your request into a function that accepts parameters. Put your request on a new blank query:
let
fnGetData = (fixtureID as text) =>
let
apiKey = Excel.CurrentWorkbook(){[Name="ApiKey"]}[Content]{0}[Column1],
fixtureID = "?",
Source = Json.Document(Web.Contents("https://v2.api-football.com/predictions/"
& fixtureID, [Headers=[#"X-RapidAPI-Key"=apiKey]]))
in
Source
in
fnGetData
Rename it to fnGetData.
Then, go to your table and click on Add Column/Add Custom Function. Select fnGetData and the input parameter is your fixtureID column. This should make all the requests and you'll just have to expand the new column results.

how to create one input text and save the result to separated fields?

i have a question. how to create one input text and fill it with this format:
, then save the result to database with separated fields:
- name
- age
- address
like example input : "Rangga Lawe 22 Jl Soekarno Jakarta"
so, it will save to database as follows:
- “Rangga Lawe“ to name field
- “22” to age field
- “Jl Soekarno Jakarta” to address field
please help me, thankyou.
im using mongoDb as my database and use node.js framework
Try splitting the value on the basis of some parameters. like instruct the user to input comma separated values. OR
You have to make input field fixed length for each of your values..
if you only have one input means that you have only one string that you want to parse into an array and then insert this fields into your database.
like this:
var str = "Rangga Lawe 22 Jl Soekarno Jakarta";
var result = str.split(" ");
//now you can work with the array
//console.log(result[0]) will contain "Rangga"
You can split the one input into multiple ones at the Schema level if you use Mongoose ORM for your mongoDB database using the .pre middleware.
Here is an example :
UserSchema.pre('save', function(next) {
const user = this
const data = user.fieldYouWantToSplit.split(" ")
user.address = data[0]
user.number = data[1]
return next()
})
})

TimeZone proplem in sequelize

Table name like users following columns are there
*Id -------------- (int type)
*DateandTime --------(Date type)
*Name -------------(varchar type)
I already insert one row like
Id = 123
Name = MR.X
DateandTime = 2018-01-12 06:22:06
Now select query in sequelize
model.user.findAll({
attributes: ['ID','DateandTime','NAME'],
where: {ID: useId} })
.then((userResult) => {
....... })
but getting DateandTime "2018-01-12T06:22:06.000Z"
You are getting out exactly what you put in. You have inserted a row as a string and it has been converted to a UTC date, it has not been stored the same as the string you put in.
When requesting it back out again, it is returning the date you put in, but from its database date form, not the original string.
You need to either store the date string as a string (varchar) if you just want the value as-is. Or, if you need to use the date for any date related functions, when you retrieve it back to display, you need to convert it to the string value you want.
I suggest using moment.js
const stringDate = moment(date).format('YYYY-MM-DD HH:mm');

Using string in place of property name (LINQ)

I have been able to get the values from tables using linq.
var q=(from app in context.Applicant
where app.ApplicantName=="")
Now what I want is this:
var q=(from app in context.Applicant
where app.stringIhave =="") // so instead of column name I have string which has same name as column.
Is it possible to specify string in Select as this is not sure what I will get in each case, I need different data all the time.
Is it possible to do so?
If no, then I will figure out something else.
Update
I have a GlobalString, which holds the column name of a table.
So when I query that table, I only specify from string which column value I want to get:
var q=(from app in context.Applicants
where app.ID==1013
select GlobalString //which is specifying that I want to get value from which column, as column name is not fixed.
//where GlobalString can have values like: app.FirstName..app.LastName etc
Update1:
var q = context.Applicants.Select("new(it.ApplicantFirstName as FirstName, it.ApplicantLastName as LastName)");
Error Message:
The query syntax is not valid. Near keyword 'AS'
You can use Dynamic Linq (available from NuGet) for that:
var q = context.Applicant.Where(app.stringIhave + " = #0", "");
for select you can try something like this
var q = context.Applicant.Select("new(it.FirstName as FirstName, it.LastName as LastName)");
so you only need construct string for that format

Resources