I have an API written in Express and MariaDB as DBMS, I'm trying to update multiple records at the same time using an array but it doesn't work.
Example:
const data = [ [ '2', '130' ], [ '4', '10' ] ]
try {
const updatedInventory = await pool.query('UPDATE inventory SET qty_shipped = qty_shipped + VALUES ? WHERE sku_id = VALUES ?', [data])
res.status(201).json(updatedInventory)
} catch (error) {
res.status(500).json({ message: error.message })
console.error(error.message)
}
However I get this error:
You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'VALUES ('2', '130'), ('4', '10') WHERE sku_id = VALUES ?' at line 1
I did something similar with the Insert endpoint where I'm doing bulk inserts also using an array and that works just fine:
const newInventory = await pool.query('INSERT INTO inventory (sku_id, qty_received, expiration_date) VALUES ? ON DUPLICATE KEY UPDATE qty_received = qty_received + VALUES(qty_received) ', [data])
Any ideas on how to get the bulk update part working?
According to the error you put in the question, the second "?" isn't grounded. If you look at the error, you see that the first value set is grounded to the data variable, but the second one isn't.
If you expect to ground the data to both "?" in your statement, then you might need to add the data variable to your statement twice.
Also, it might be that you can't ground the data variable twice. I wouldn't expect that; you should be able to use the data variable twice in our statement. If you need to different variables, then create a second variable, copy data into the second variable and then ground both both variables in your statement.
Related
I have an object that looks like this:
let queries = [
{
name: "checkUsers",
query: 'select * from users where inactive = 1'
},
{
name: "checkSubscriptions",
query: 'select * from subscriptions where invalid = 1'
}
]
I am making an AWS Lambda function that will iterate these queries, and if any of them returns a value, I will send an email.
I have come up with this pseudo code:
for (let prop in queries) {
const result = await mysqlConnector.runQuery(prop.query).catch(async error => {
// handle error in query
});
if (result.length < 0){
// send email
}
}
return;
I am wondering is this ideal approach? I need to iterate all the object queries.
I don't see anything wrong with what you are trying to achieve but there are few changes you could do
Try to use Promise.all if you can. This will speed up the overall process as things will execute in parallel. It will depend on number of queries as well.
Try leverage executing multiple statements in one query. This way you will make one call and then you can add the logic to identify. Check here
I've a node.js api in which user sends the required fields as an array to be fetched from the mongodb database. I need to find the data of that fields using Find query. I've written forEach statement to loop through that array and got the array elements. But when I try to get the results by inserting the array elements in the query, it doesn't giving the required results. Could any one please help me in resolving the issue by seeing the code below?
templateLevelGraphData: async function(tid,payload){
let err, templateData, respData = [], test, currentValue;
[err,templateData] = await to(Template.findById(tid));
var templateId = templateData.templateId;
payload.variables.forEach(async data=>{
console.log(data); //data has the array elements like variables=["humidity"]
[err, currentValue] = await to(mongoose.connection.db.collection(templateId).find({},{data:1}).sort({"entryDayTime":-1}).limit(1).toArray());
console.log(currentValue);
});
return "success";
}
The expected output is,
[ { humidity: 36 } ]
But I'm getting only _id like,
[ { _id: 5dce3a2df89ab63ee4d95495 } ]
I think data is not applying in the query. But I'm printing the data in the console where it's giving the correct results by displaying the array elements like, humidity. What I need to do to make it work?
When you are passing {data: 1} you are passing an array where is expecting name of column.
You have to create an object where the keys are going to be the elements of the array and set them to 1.
const projection = data.reduce((a,b) => (a[b]=1, a), {});
[...] .find({}, projection) [...]
Actually I got the solution.
for(let i=0;i<payload.variables.length;i++){
var test = '{"'+ payload.variables[i] +'":1,"_id":0}';
var query = JSON.parse(test);
[err, currentValue] = await to(mongoose.connection.db.collection(templateId).find({"deviceId":deviceId},query).sort({"entryDayTime":-1}).limit(1).toArray());
console.log(currentValue); //It's giving the solution
}
I am building a simple API that also deal with geometries (store into postGIS).
I am using knex-postgis to access the ST_ spatial functions in postGIS with knex.
I have used this example to insert a point and it work when its hard coded.
But my lack of experience is leave me hanging, how can I make the query dynamic? I want to create a form with input for x and y value and send it to the ST_geomFromText function to save it to the geom type in the db.
Is this when you would use parameters? Could someone point me in the right direction?
// insert a point
const sql1 = db.insert({
id: 1,
geom: st.geomFromText('Point(0 0)', 4326)
}).into('points').toString();
console.log(sql1);
// insert into "points" ("geom", "id") values
(ST_geomFromText('Point(0 0)'), '1')
So far I have tried
router.post('/', (req, res, next) => {
queries.create(req.body).then(poi => {
res.json(poi[0]);
});
});
Insert query
create(poi) {
const sql = db.insert(poi).returning('*').into('poi');
return sql;
},
In Postman I am sending this in its body
{
"place": "test",
"comments": "new",
"numbers": 6,
"geom": "st.geomFromText('Point(-71.064544 44.28787)', 4326)"
}
But get an error "Unhandled rejection error: parse error - invalid geometry"
The hard coded object looks the same and is work fine.
I have a feeling that I am using the st.geomFromText wrong, but i dont I am not sure?
This is what i get if i console.log returned query
insert into "poi" ("comments", "geom", "numbers", "place") values ('new', 'st.geomFromText(''Point(-71.064544 44.28787)'', 4326)', 6, 'test')
(see how it does not change st.geom.. to st_geom..? Also the quotation marks around the ST function and the Point object is not right)
This string work when I run it in pgAdmin
insert into "poi" ("comments", "geom", "numbers", "place") values ('new', st_GeomFromText('Point(-71.064544 44.28787)', 4326), 6, 'test')
EDIT:
I console.log the hardcoded version too. It does no appear to add the extra '' on the geometry value
insert into "poi" ("geom") values (ST_geomFromText('Point(-71.064544 44.28787)', 4326))
Any advice?
You pass coordinates from postman as string value in JSON object
"geom": "st.geomFromText('Point(-71.064544 44.28787)', 4326)"
And expect your code to transform it into a function call
st.geomFromText('Point(-71.064544 44.28787)', 4326)
What knex does in this case. It takes your string and convert it as it is string field (here you get a single quote ' replacement with double '' for Postgres to escape it).
What you can do. Pass your geometry to your API like this
"geom": "Point(-71.064544 44.28787)"
And in your route handler
create(poi) {
const sql = db.insert({
place: poi.place,
comments: poi.comments,
numbers: poi.numbers,
geom: st.geomFromText(poi.geom, 4326)
}).returning('*').into('poi');
return sql;
},
I have around 5,00,000 records in my database. I'm using DataTables to build an admin panel to manage the records.
I have Node.js as backed with MongoDB.
I have used this library - https://www.npmjs.com/package/datatables-query
So far on page load I have successfully loaded results a shown in the below Image.
Whenever I type something in the search box, I get 500 error as shown in the screenshot.
What could be the problem here?
Is DataTable a good option for showing grid with huge amount of data or is there any better option considering Node.js, Express and MongoDB combo?
Here is my server side code.
app.post('/getUsersData',function(req, res) {
var Model = require('./models/user'),
datatablesQuery = require('datatables-query'),
params = req.body,
query = datatablesQuery(Model);
query.run(params).then(function (users) {
var data = JSON.stringify(users);
// var data = JSON.stringify(users);
res.end(data);
}, function (err) {
res.status(500).json(err);
});
});
I have a table in MongoDB named User with 3 columns
1) Name
2) Email
3) Password
$(document).ready(function() {
var table = $('#datatable').DataTable({
// dom: 'Bfrtip',
processing: true,
serverSide: true,
order: [[1, 'asc']],
"aoColumnDefs": [ { "sClass": "hide_me", "aTargets": [ 0 ], visible: false } ], // first column in visible columns array gets class "hide_me"
ajax: {
url: "/getUsersData",
type: 'POST',
dataSrc: "data"
},
columns: [
{ data : "_id"},
{ data : "name" },
{ data : "email" },
{ data : "password" },
],
responsive: true
});
});
I would probably create an index on the data and search that index rather than searching the data itself. If you do however create a full text index you need to have a combination off all your cols in your collection and mongo allows only 1 full text index per collection.
As for alternatives, you could look into AWS's Elastic Search (which works just fine with MongoDB) or Sphinx Index (based on PostreSQL)
Edit:
I know this answer doesn't actually answer the question in the slightest but
I fear that the 500 error is not a memory issue in the application but rather on the DB (mongo is not like SQL, so don't design your applications like you would in SQL).
Some reading material, if you plan on changing db structure
http://rachbelaid.com/postgres-full-text-search-is-good-enough/
https://about.gitlab.com/2016/03/18/fast-search-using-postgresql-trigram-indexes/
The key here is figuring out what exactly went wrong on the DB or server side layer. The error message that you are alerting out is a generic message from the datatables library which points you to the following documentation: https://datatables.net/manual/tech-notes/7
The documentation linked to above says that this error will occur anytime a non-200 response is returned from the server. In this case you are explicitly throwing a 500 error in the error handler/callback of the datatables query call, but that doesn't give you/us the information we need!
The next step is to look at the response body in the network tab of your browser's console, since you are serializing the error as json and sending it along with your 500 response, or to console.log out the err in the error handler/callback in your node code. This will get you an understanding of what is actually going wrong, and with that information we can more accurately prescribe a solution. Post an update with whatever you log out please!
here is solution i tried to fix this issue:
goto ur jquery.datatable.min.js and then comment out the error message and you are good to go.
function K(a, b, c, d) {
// c = "DataTables warning: " + (a ? "table id=" + a.sTableId + " - " : "") + c;
// d && (c += ". For more information about this error, please see http://datatables.net/tn/" + d);
// if (b) E.console && console.log && console.log(c); else if (b = n.ext, b = b.sErrMode || b.errMode, a && r(a, null, "error", [a, d, c]), "alert" == b) alert(c); else {
// if ("throw" == b) throw Error(c);
// "function" ==
// typeof b && b(a, d, c)
// }
}
I have the following code:
var db = require("redis");
var dbclient1 = db.createClient();
dbclient1.zadd("myprivateset", 3, {"guid":"abab-baba", "data-persistent":"xxxx", "size":"20"})
dbclient1.zadd("myprivateset", 2, {"guid":"abab-baba3", "data-persistent":"xxxx", "size":"20"})
dbclient1.zrangebyscore("myprivateset", 1, 4)
dbclient1.hgetall("myprivateset", function(err, rep){
console.log(rep);
});
I wish to store my objects (in JSON format) in a sorted set, which determine by the score (3 & 2 in our case).
For some reason, when I print this table (rep), I get undefined.
What I do wrong?
Issue 1 -- sorted set keys
Try stringifying the JSON you are using as the keys of your sorted set. For example,
dbclient1.zadd("myprivateset", 3, {"guid":"abab-baba", "data-persistent":"xxxx", "size":"20"})
needs to be:
dbclient1.zadd("myprivateset", 3, JSON.stringify({"guid":"abab-baba", "data-persistent":"xxxx", "size":"20"}))
Without stringifying the keys, every zadd will use the key [object Object] overwriting each time. That is, you'll only ever have one item in your sorted set that is unidentifiable (other than by [object Object]).
Issue 2 -- fetching data
Also, hgetall is not the redis command to use for retrieving data in a redis sorted set. You'll want to focus on sorted set specific commands. A list of redis commands are listed here: http://redis.io/commands
My two cents, building on comments by #leonid-beschastny and #cpentra1. I recommend using redis.multi(). It allows for several calls in a batch, and as you can see in the example, as soon as the three elements are added to the ordered set, we can perform a zrangebyscore in the same multi batch and get the expected results. Instructions can be created dynamically. The replies array when multi.exec() is invoked returns the results for each of the multi operations, in order.
var db = require("redis");
var dbclient1 = db.createClient();
var multi = dbclient1.multi();
// We use JSON.stringify() as suggested by #cpentra1
multi.zadd("myprivateset", 3, JSON.stringify({"guid":"abab-baba", "data-persistent":"xxxx", "size":"20"}));
multi.zadd("myprivateset", 2, JSON.stringify({"guid":"abab-baba3", "data-persistent":"xxxx", "size":"20"}));
multi.zadd("myprivateset", 2, JSON.stringify({"guid":"abab-dafa3", "data-persistent":"yyyy", "size":"21"}));
multi.zrangebyscore("myprivateset", 1, 4);
multi.zcard("myprivateset"); // The total number of elements in the set
multi.exec(function(err, replies) {
console.log(replies)
// Will output something like:
// [ 1,
// 1,
// 1,
// [ '{"guid":"abab-baba3","data-persistent":"xxxx","size":"20"}',
// '{"guid":"abab-dafa3","data-persistent":"yyyy","size":"21"}',
// '{"guid":"abab-baba","data-persistent":"xxxx","size":"20"}' ],
// 3 ]
});
Note: if you run the same example twice, instead of 1s in the first three elements of the replies array, you'll get 0s as the same member with the same score cannot be added twice.
Seems like with Redis 6.2 the format has changed to and object with score and value attributes or an array of those, like this:
async function sortedSet() {
let client;
try {
client = createClient();
client.on("error", (err) => console.log("Redis Client Error", err));
await client.connect();
console.log("connected");
await client.zAdd("user:0:followers", [{score: "1", value: "John"}, {score: "2", value: "Other John"}]);
console.log("sorted set added");
} finally {
await client.quit();
}
}
sortedSet("duto_guerra", "with hashes");
In case you are wondering, I figured this out by reading the source code for node-redis ZADD