How to construct a query with a bulk insert?
let (query, values) = Query::insert()
.into_table(DbAppointmentService::Table)
.columns([
DbAppointmentService::Id,
DbAppointmentService::AppointmentId,
DbAppointmentService::Name,
DbAppointmentService::Price,
DbAppointmentService::Duration,
])
.unexisting_bulk_insert_method(values) // What is the method to use to make a bulk insert?
.build(PostgresQueryBuilder);
To make bulk insert in sea_query you can "split" builder declaration into multiple statements.
// Step 1: Create builder
let builder = Query::insert()
.into_table(DbAppointmentService::Table)
.columns([
DbAppointmentService::Id,
DbAppointmentService::AppointmentId,
DbAppointmentService::Name,
DbAppointmentService::Price,
DbAppointmentService::Duration,
]).to_owned();
// Step 2: Execute `values_panic` on each loop step
for item of items.into_iter() { // where items is a vec of row's values
builder.values_panic(vec![
item.first.into(),
item.second.into(),
item.third.into()
]);
}
// Step 3: Build the query by using `build` method
let (query, values) = builder.build(PostgresQueryBuilder);
// Step 4: Use `query` and `values` as usual
Related
Using Knex to make queries to my Postgres DB. I have a function that provides a "base" query using the Knex QueryBuilder. This works fine until I need to add something raw to the SELECT statement. From what I can tell, running .raw always wants to return a result. I just need it to be added to the QueryBuilder, though, so it can be executed by a different part of my app.
const baseQuery = knex
.select(newUserFields)
.from('users')
.leftJoin('user_roles', 'users.id', 'user_roles.user_id')
.leftJoin('roles', 'user_roles.role_id', 'roles.id')
.leftJoin('role_permissions', 'roles.id', 'role_permissions.role_id')
.leftJoin(
'permissions',
'permissions.id',
'role_permissions.permission_id'
)
.groupBy(
'users.id',
'users.email',
'users.name',
'users.status',
'users.created_at',
'users.password_reset_expiration',
'users.password',
'users.password_reset_token'
)
.orderBy('users.created_at', 'desc');
I need to add the following to the select:
knex.raw('to_json(array_agg(distinct roles.name)) as roleNames')
knex.raw('to_json(array_agg(distinct permissions.name)) as permissionNames')
How can I add these raw selects to the base query so that the base query can then be passed to a different function as a QueryBuilder and added to?
The cool thing about knex is that it is a queryBuilder, which allows you to call the methods without any limits about the order of calls. That means that you can just construct your base query in a function, and then attach to it additional things (such as additional columns).
In your case you just can call another time to select (knex will join the select calls)
// base-query.js
export const getBaseQuery = () => knex
.select(newUserFields)
.from('users')
.leftJoin('user_roles', 'users.id', 'user_roles.user_id')
.leftJoin('roles', 'user_roles.role_id', 'roles.id')
.leftJoin('role_permissions', 'roles.id', 'role_permissions.role_id')
.leftJoin('permissions', 'permissions.id', 'role_permissions.permission_id')
.groupBy(
'users.id',
'users.email',
'users.name',
'users.status',
'users.created_at',
'users.password_reset_expiration',
'users.password',
'users.password_reset_token'
)
.orderBy('users.created_at', 'desc');
// other-file.js
import {getBaseQuery} from 'base-query';
const enhancedQuery = getBaseQuery().select([
knex.raw('to_json(array_agg(distinct roles.name)) as roleNames'),
knex.raw('to_json(array_agg(distinct permissions.name)) as permissionNames'),
]);
const results = await enhancedQuery;
Another cool way that I'm using heavily, solves the following requirement: Sometimes I need to change an internal query from the outside, I use the modify
For example, I have a getProducts method which execute a select query and do some data transformation.
In order to implement getProductById which needs to return the same data structure (it is just need to filter the base query) I pass a queryModifier method which modifies the original query.
async function getProducts(queryModifier) {
const products = await knex
.select('*')
.from('products')
.modify((queryBuilder) => {
if (typeof queryModifier === 'function') {
return queryModifier(queryBuilder);
}
});
return products.map(someDataTransformation);
}
async function getProductById(id) {
return getProducts((qb) => {
return qb.where('id', id);
});
}
I'm using SQLite with sql.js on my project and I have been having some trouble with my implementation. Seems like the queries are being run on the database twice because for the for the INSERT statements I get 2 records in the DB.
The way I do it, I create the SQL and then pass it on to this method (the opts variable contains all of the data being put into the database):
prepareStatementAndCompileResults(db, sql, opts){
const stmt = db.prepare(sql);
const result = stmt.getAsObject(opts);
var rows = [];
if(!this.isEmpty(result)){ // isEmpty is a simple method that checks for empty objects
rows.push(result);
}
while(stmt.step()) {
var row = stmt.getAsObject();
rows.push(row);
}
this.saveToFile(db);
stmt.free();
return rows;
},
Here is a sample SQL INSERT that is being run twice
INSERT OR IGNORE INTO tag_event (tag_id, event_id, unique_string)
VALUES (:tag_id,:event_id, :unique);
Here is what the opts variable would look like for this query:
var opts = {
[':tag_id']: 1,
[':event_id']:1,
[':unique']: '1-1'
}
Because you're pushing it into row 2 time.
// if not empty will add to row
if(!this.isEmpty(result)){ // isEmpty is a simple method that checks for empty objects
rows.push(result);
}
// not sure what step() does but I'm assuming this will also run
while(stmt.step()) {
var row = stmt.getAsObject();
rows.push(row);
}
Verify by using a debugger or just console.log(rows) after the while loop before the save
So, what it turns out I needed to do was bind the variables to the prepared statement before getting the rather than binding them through getAsObject. This is much more efficient. My API response time on a local test went from 785ms to 14.5ms
prepareStatementAndCompileResults(db, sql, opts){
const rows = [];
const stmt = db.prepare(sql);
stmt.bind(opts);
while(stmt.step()) {
var row = stmt.getAsObject();
rows.push(row);
}
this.saveToFile(db);
stmt.free();
return rows;
},
I have two queries:
a) select id from ingredietns where name = my_param;
b) select word_id from synonyms where name = my_param;
Both return 0 or 1 row. I can also add limit 1 if needed (or in knex first()).
I can translate each into knex like this:
knex("ingredients").select('id').where('name', my_param) //do we need first()?
knex("synonyms").select('word_id').where('name', my_param) //do we need first()?
I need function called "ingredientGetOrCreate(my_param)". This function would
a) check if any of above queries return result
b) if any of these return, then return ingredients.id or synonyms.word_id - only one could be returned
c) if record doesn't eixst in any of tables, I need to do knex inesrt aand return newly added id from function
d) later I am not sure I also understand how to call this newly create function.
Function ingredientGetOrCreate would be used later as seperate function or in the following scenario (like "loop") that doesn't work for me either:
knex("products") // for each product
.select("id", "name")
.map(function (row) {
var descriptionSplitByCommas = row.desc.split(",");
Promise.all(descriptionSplitByCommas
.map(function (my_param) {
// here it comes - call method for each param and do insert
ingredientGetOrCreate(my_param)
.then(function (id_of_ingredient) {
knex('ingredients_products').insert({ id_of_ingredient });
});
...
I am stuck with knex and Promise queries because of asynchronouse part. Any clues, please?
I though I can somehow use Promise.all or Promise.some to call both queries.
P.S. This is my first day with nodejs, Promise and knex.
As far as I decode your question, it consists of two parts:
(1) You need to implement upsert logic (get-or-create logic).
(2) Your get part requires to query not a single table, but a pair of tables in specific order. Table names imply that this is some sort of aliasing engine inside of your application.
Let's start with (2). This could definitely be solved with two queries, just like you sense it.
function pick_name (rows)
{
if (! rows.length) return null
return rows[0].name
}
// you can sequence queries
function ingredient_get (name)
{
return knex('ingredients')
.select('id').where('name', name)
.then(pick_name)
.then(name =>
{
if (name) return name
return knex('synonyms')
.select('word_id').where('name', name)
.then(pick_name)
})
}
// or run em parallel
function ingredient_get (name)
{
var q_ingredients = knex('ingredients')
.select('id').where('name', name)
.then(pick_name)
var q_synonyms = knex('synonyms')
.select('word_id').where('name', name)
.then(pick_name)
return Promise.all([ q_ingredients, q_synonyms ])
.then(([name1, name2]) =>
{
return name1 || name2
})
}
Important notions here:
Both forms works well and return first occurence or JS' null.
First form optimizes count of queries to DB.
Second form optimizes answer time.
However, you can go deeper and use more SQL. There's a special tool for such task called COALESCE. You can consult your SQL documentation, here's COLASCE of PostgreSQL 9. The main idea of COALESCE is to return first non-NULL argument or NULL otherwise. So, you can leverage this to optimize both queries and answer time.
function ingredient_get (name)
{
// preparing but not executing both queries
var q_ingredients = knex('ingredients')
.select('id').where('name', name)
var q_synonyms = knex('synonyms')
.select('word_id').where('name', name)
// put them in COALESCE
return knex.raw('SELECT COALESCE(?, ?) AS name', [ q_ingredients, q_synonyms ])
.then(pick_name)
This solution guarantees single query and furthermore DB engine can optimize execution in any way it sees appropriate.
Now let's solve (1): We now got ingredient_get(name) which returns Promise<string | null>. We can use its output to activate create logic or return our value.
function ingredient_get_or_create (name, data)
{
return ingredient_get(name)
.then(name =>
{
if (name) return name
// …do your insert logic here
return knex('ingredients').insert({ name, ...data })
// guarantee homohenic output across get/create calls:
.then(() => name)
})
}
Now ingredient_get_or_create do your desired upsert logic.
UPD1:
We already got ingredient_get_or_create which returns Promise<name> in any scenario (both get or create).
a) If you need to do any specific logic after that you can just use then:
ingredient_get_or_create(…)
.then(() => knex('another_table').insert(…))
.then(/* another logic after all */)
In promise language that means «do that action (then) if previous was OK (ingredient_get_or_create)». In most of the cases that is what you need.
b) To implement for-loop in promises you got multiple different idioms:
// use some form of parallelism
var qs = [ 'name1', 'name2', 'name3' ]
.map(name =>
{
return ingredient_get_or_create(name, data)
})
var q = Promise.all(qs)
Please, note, that this is an agressive parallelism and you'll get maximum of parallel queries as your input array provides.
If it's not desired, you need to limit parallelism or even run tasks sequentially. Bluebird's Promise.map is a way to run map which analogous to example above but with concurrency option available. Consider the docs for details.
There's also Bluebird's Promise.mapSeries which conceptually is an analogue to for-loop but with promises. It's like map which runs sequentially. Look the docs for details.
Promise.mapSeries([ 'name1', 'name2', 'name3' ],
(name) => ingredient_get_or_create(name, data))
.then(/* logic after all mapSeries are OK */)
I believe the last is what you need.
I've traded node-DBI for knex because it has more function that I require.
So far I'd make the same choice again but only one thing is holding me back: writing abstract methods that take a options variable where params like where, innerjoin and such are contained in.
Using node-dbi I could easily forge a string using these variables but I can't seem to create the knex chain dymanicly because after using a switch, you'd get knex.method is not a function.
Any idea how to resolve this?
I'm looking for something as in
`getData(table,options){
var knex=knex
if(options.select)
/** append the select data using knex.select()
if(options.where)
/** append the where data using knex.where(data)*/
if(options.innerJoin)
/** append innerjoin data*/
}`
This way I can avoid having to write alot of DB functions and let my Business Logical Layers handel the requests
/*This function serves as the core of our DB layer
This will generate a SQL query and execute it whilest returning the response prematurely
#param obj:{Object} this is the options object that contain all of the query options
#return Promise{Object}: returns a promise that will be reject or resolved based on the outcome of the query
The reasoning behind this kind of logic is that we want to abstract our layer as much as possible, if evne the slightest
sytnax change occurs in the near future, we can easily update all our code by updating this one
We are using knex as a query builder and are thus relying on Knex to communicate with our DB*/
/*Can also be used to build custom query functions from a data.service. This way our database service will remain
unpolluted from many different functions and logic will be contained in a BLL*/
/* All available options
var options = {
table:'table',
where:{operand:'=',value:'value',valueToEqual:'val2'},
andWhere:[{operand:'=',value:'value',valueToEqual:'val2'}],
orWhere:[{operand:'=',value:'value',valueToEqual:'val2'}],
select:{value:['*']},
insert:{data:{}},
innerJoin:[{table:'tableName',value:'value',valueToEqual:'val2'}],
update:{data:{}}
}*/
/*Test object*/
/*var testobj = {
table:'advantage',
where:{operand:'>',value:'id',valueToEqual:'3'},
select:{value:['*']},
innerJoin:{table:'User_Advantage',value:'User_Advantage.Advantageid',valueToEqual:'id'}
}
var testobj = {
table:'advantage',
where:{operand:'>',value:'id',valueToEqual:'3'},
select:{value:['*']},
innerJoin:{table:'User_Advantage',value:'User_Advantage.Advantageid',valueToEqual:'id'}
}
queryBuilder(testobj)*/
function queryBuilder(options){
var promise = new Promise(function (resolve, reject) {
var query;
for (var prop in options) {
/*logger.info(prop)*/
if (options.hasOwnProperty(prop)) {
switch (prop) {
case 'table':
query = knex(options[prop]);
break;
case 'where':
query[prop](options[prop].value, options[prop].operand, options[prop].valueToEqual);
break;
/*andWhere and orWhere share the same syntax*/
case 'andWhere':
case 'orWhere':
for(let i=0, len=options[prop].length;i<len;i++){
query[prop](options[prop][i].value, options[prop][i].operand, options[prop][i].valueToEqual);
}
break;
case 'select':
query[prop](options[prop].value);
break;
/*Same syntax for update and insert -- switch fallthrough*/
case 'insert':
case 'update':
query[prop](options[prop].data);
break;
case 'innerJoin':
for(let i=0, len=options[prop].length;i<len;i++){
query[prop](options[prop][i].table, options[prop][i].value, options[prop][i].valueToEqual);
}
break;
}
}
}
return query
.then(function (res) {
return resolve(res);
}, function (error) {
logger.error(error)
return reject(error);
})
return reject('Options wrongly formatted');
});
return promise
}
Thanks to Molda I was able to produce the code above. This one takes a Object called options in as a parameter and will build the knex chain based on this value. See the comments for the syntax of Object
Not every knex query option has been included but this will serve as a good base for anyone trying to achieve a similar effect.
Some examples to use this:
/*Will return all values from a certain table
#param: table{String}: string of the table to query
#param: select{Array[String]}: Array of strings of columns to be select -- defaults to ['*'] */
function getAll(table,select) {
/*Select * from table as default*/
var selectVal=select||['*']
var options={
table:table,
select:{value:selectVal}
}
return queryBuilder(options)
}
or a more specific use case:
function getUserAdvantages(userid){
var options = {
table:'advantage',
innerJoin:[{table:TABLE,value:'advantage.id',valueToEqual:'user_advantage.Advantageid'}],
where:{operand:'=',value:'user_advantage.Userid',valueToEqual:userid}
}
return sqlService.queryBuilder(options)
}
Note: the sqlService is a module of node that I export containing the queryBUilder method.
Edit: I wanted to add that the only roadblock I had was using the .from / .insert from Knex. I no longer use these methods as they resulted in errors when using them. Ive used knex(table) as commented.
here my issue:
I've an IQueryable object and I need to execute a new query on Db (just like a refresh) by launching the same contained query in my IQueryable obj.
An example:
myObj = objCtx.Person
.Where(p => p.IdPerson...)
.OrderBy(p => ...)
.Select(..some field..);
//...
// From another function I just want re-execute
// the same query
// -> Well, that retrieve full lambda
var et = this.myObj.Expression
// This doesn't work:
var anotherObj = objCtx.Person.Where(et);
//or..
var tmp = Expression.Lambda<Func<T, bool>>(et);
// This doesn't work too:
var anotherObj = objCtx.Person.Where(tmp);
Is it possibile to achieve? What am I missing?
Thanks
Ok:
I would be able to retrieve full query (select statement) from an IQueryable object and execute it in order to get all my updated data.
This is the original query:
this.myIQueryableObj = objCtx.Person
.Where(p => p.IdPerson...)
.OrderBy(p => ...)
.Select(..some field..);
//..And in some button click, I want execute again the above query, but all what I know is myIQueryableObjOnly. Can you help me?