I am developing something with nodejs and I have a table on DynamoDB. Therefore, I am using aws-sdk for nodejs to work with these two things.
Here is an example. ID and name are the combined key to the table.
ID name otherData
----------------------
1 Matt 123
1 Mary 1234
2 Mary 2312
4 Mary 3123
4 Pet 3123
What I want is to get a list of distinct ID. So, the expected output is
ID = [1, 2, 4]
How to do this?
1) Do a DynamoDB scan to get all (non-unique) ID values. The ProjectionExpression limits the result attributes returned, in this case to ID only. Note this doesn't make your scan any faster. When you execute the scan you will get a JSON object back containing your results. You can use JMES to get the ID array (data.Items.ID).
2) Make the array unique in your nodejs code (e.g. using the ArrNoDupe code function below)
var params = {
ProjectionExpression: "ID",
TableName: "YOUR_TABLE"
};
ddb.scan(params, function(err, data) {
if (err) {
console.log("Error", err);
} else {
console.log(ArrNoDupe(data.Items.ID));
});
}
});
function ArrNoDupe(a) {
var temp = {};
for (var i = 0; i < a.length; i++)
temp[a[i]] = true;
var r = [];
for (var k in temp)
r.push(k);
return r;
}
Related
I'd like to write a wrapper function function select(db: any, ids: number[]): Cat[] that returns an array of Cat rows fetched from the DB by ID. The function should return the entire array of rows.
Below is one approach I've written. Instead of calling db.each on every ID in a for-loop as I do below, is it possible to pass my entire ids: number[] array as a parameter to a db.all / db.each query function?
// dbmethods.ts
async function select(db: any, ids: number[]): Promise<Cat[]> {
let query = "SELECT * FROM cats_table WHERE id = ?;";
let cats_back: Cat[] = [];
for (let i = 0; i < ids.length; i++) {
let cat: Promise<Cat> = new Promise(async function (resolve, reject) {
await db.get(query, ids[i], (err: Error, row: any) => {
if (err) {
reject(err);
} else {
let cat: Cat = {
index: row.id,
cat_type: row.cat_type,
health: row.health,
num_paws: row.num_paws
};
resolve(cat);
}
});
});
cats_back.push(await cat);
}
return cats_back;
}
and
// index.ts
let ids = create_many_ids(10_000); // returns an array of unique ordered ints between 0 and 10K
let res = await select(db, ids);
console.log(res); // successfully prints my cats
Benchmarks on my select function above suggest that it takes 300ms to select 10_000 rows by ID. It seems to me that's a little long; 10K rows shouldn't take that long for sqlite's select by id functionality... How can I be more efficient?
SELECT * FROM cats_table WHERE id IN (SELECT value FROM json_each(?));
The query parameter is a string representing a JSON array of ids, e.g., '[1, 2, 4]'
See this tutorial for further details
I am working on constructing a query to my database to return some data. Here is the link to a previous post describing my intentions Finding database data that best fits user variable responses. I want to return all of the columns for each data object, however the id that is returned is not correct and an additional VALUE field is being returned.
My database is set up like this
venues
id name parking decorations hotel
1 park 1 2 1
2 beach 1 2 2
3 theater 2 2 2
4 yard 2 1 1
and an enum table
id value
1 TRUE
2 FALSE
3 MAYBE
I am building a query on my backend as follows:
let searchConstraintsTrue = 'WHERE';
let firstItemTrue = 0;
for (const prop in req.body) {
if (req.body[prop] === 'TRUE') {
if (firstItemTrue === 0) {
searchConstraintsTrue += ` ${prop} = 1`;
firstItemTrue++;
} else {
searchConstraintsTrue += ` AND ${prop} = 1`;
}
}
}
let searchConstraintsMaybe = 'ORDER BY';
let firstItemMaybe = 0;
for (const prop in req.body) {
if (req.body[prop] === 'MAYBE') {
if (firstItemMaybe === 0) {
searchConstraintsMaybe += ` (${prop} = 1)::integer`;
firstItemMaybe++;
} else {
searchConstraintsMaybe += ` + (${prop} = 1)::integer`;
}
}
}
res.setHeader('Access-Control-Allow-Origin', 'http://localhost:3000');
let sqlText = `SELECT * FROM venues
INNER JOIN response_enum rp ON rp.id = venues.parking
INNER JOIN response_enum rd ON rd.id = venues.decorations
INNER JOIN response_enum rh ON rh.id = venues.hotel
${searchConstraintsTrue} ${searchConstraintsMaybe} DESC`;
I realize that my searchConstraintsTrue and searchConstraintsMaybe are not properly using the enum table but right now I am just trying to get things working.
An example response looks like this:
[ {
id: 1,
name: 'beach',
parking: 1,
decorations: 2,
hotel: 1,
value: 'TRUE'
},
{
id: 2,
name: 'yard',
parking: 1,
decorations: 2,
hotel: 2,
value: 'FALSE'
}]
So I am returning the desired data however the id's are incorrect and there is a value column which doesn't exist in my database.
SELECT * will select all fields from the joined tables. You need to specify a list of fully qualified field names like so:
SELECT v.id,v.name,v.parking,v.decorations,v.hotel FROM venues v
INNER JOIN response_enum rp ON rp.id = venues.parking
INNER JOIN response_enum rd ON rd.id = venues.decorations
INNER JOIN response_enum rh ON rh.id = venues.hotel
${searchConstraintsTrue} ${searchConstraintsMaybe} DESC
I have two collections, both have a structure like this:
id trips_in
1 5
2 10
id trips_out
1 6
2 8
My question is how can I combine them into a single collection like such:
id trips_in trips_out
1 5 6
2 10 8
I found out about mapReduce, but its functionality looks like more than what I need. I wrote the following query:
tripsInMap = function() {
var values = {
trips_in: this.trips_in
};
emit(this._id, values);
};
tripsOutMap = function() {
var values = {
trips_out: this.trips_out
};
emit(this._id, values);
};
reduce = function(key, values) {
var result = {
"trips_out" : "",
"trips_in" : ""
};
values.forEach(function(value) {
if(value.trips_out !== null) {result.trips_out = value.trips_out;}
if(value.trips_in !== null) {result.trips_in = value.trips_in;}
});
return result;
}
db.tripsIn.mapReduce(tripsInMap, reduce, {"out": {"reduce": "joined"}});
db.tripsOut.mapReduce(tripsOutMap, reduce, {"out": {"reduce": "joined"}});
However I end up with "trips_in": undefined. I wonder if there is a better method.
While this may not be the fastest way, you could try something like this:
// Create the new collection with data from the tripsIn collection
db.tripsIn.find().forEach( function(trip) {
db.tripsJoined.insert({ _id: trip._id, trips_in: trip.trips_in, trips_out: 0 });
})
// Update the trips_out field in the tripsJoined collection
// using upsert:true to insert records that are not found
db.tripsOut.find().forEach( function(trip) {
db.tripsJoined.update(
{ _id: trip._id},
{$inc: {trips_in: 0, trips_out: trip.trips_out}},
{upsert: true});
})
The first line will iterate through each document in the tripsIn collection and insert a corresponding document in the tripsJoined collection with the trips_out field set.
The second line will iterate over the tripsOut collection, and for each document it will update the corresponding tripsJoined document with the trips_out value.
Note that I added {$inc: {trips_in: 0... and upsert:true. This was done so that if any documents of trips exist in the tripsOut collection that do not have a corresponding _id value in the tripsIn collection, the document is inserted and the trips_in field is initialized to 0.
I am stuck with mongo query. I have a mongo collection structure which i can not modify at this time as it is very large data.
I need to carry out some results from the collection , so tried all ways round to get it.
Here is my collection json schema:-
{
"date": "2017-01-01T00:00:00.000Z",
"bob":"P",
"jacob":"P",
"wilson":"A",
"dev":"SL"
},
{
"date": "2017-01-02T00:00:00.000Z",
"bob":"P",
"jacob":"A",
"wilson":"A",
"dev":"SL"
},
{
"date": "2017-01-03T00:00:00.000Z",
"bob":"P",
"jacob":"P",
"wilson":"A",
"dev":"SL"
},
{
"date": "2017-01-04T00:00:00.000Z",
"shashikant":"P",
"jacob":"P",
"wilson":"SL",
"dev":"SL"
}
....
As output I am looking for below kind of structure:-
from 1st jan 2017 to 30th jan 2017
bob P 17
bob A 2
wilson P 10
dev SL. 1
.....
I am using loopback for my backend but still i can use normal mongodb query to get the output.
Please help
MongoDB allows $unwind only for the arrays. But you could use a simple mapReduce to achieve what you want:
//Define the time frame here
var from = new Date('2015-01-01T00:00:00.000Z');
var to = new Date('2025-01-01T00:00:00.000Z');
db.getCollection('test').mapReduce(function () {
var keys = Object.keys(this);
//If there is no date found on a record, simply skip
if (!this.date) {
return;
}
var date = new Date(this.date);
//Skip the records that do not fit into [form; to] interval
if (!(date >= from && date <= to)) {
return;
}
for (var i = 0; i < keys.length; i++) {
var key = keys[i];
//Emit each combination of key and value
if (key !== 'date' && key !== '_id') {
emit({key: key, value: this[key]}, {count: 1});
}
}
},
function (key, values) {
var reduced = {count: 0};
for (var i = 0; i < values.length; i++) {
var value = values[i];
//Simply counting the combinations here
reduced.count += value.count;
}
return reduced;
},
{
//Passing the dates to mongo
//so 'from' and 'to' will be avail in map, reduce and finalize
scope: {from: from, to: to},
finalize: function (key, reducedValue) {
//Changing the data structure just for a convenience
return {
propertyName: key.key,
propertyValue: key.value,
from: from,
to: to,
count: reducedValue.count
};
},
out: {inline: 1}
}
);
I tested this in Mongo console, but map-reduces are also supported by mongo native Node.js and for mongoose as well.
I am using nodejs and back-end DB as PostgreSQL and the ORM is SequalizeJS.
I want to add list of records into a table. The records are array of objects.
I am iterating this array and pushing each record into database . But sometimes , the order is not maintained.
Can you suggest some other way to fix this? . I want to add record one by one serially.
var users = <array of users>;
var createdUsers = [];
for (var Index = 0; Index < users.length; Index++) {
logger.debug("Insert User" + users[Index].user_name);
models.User.create(users[Index]).then(function (user) {
logger.debug("Inserted User" + user.user_name);
createdUsers.push(user);
if (createdUsers.length === users.length) {
response.status(200).json(createdUsers);
}
}).catch(function (error) {
response.status(500).json(error);
});
}
users contains [{user_name:"AAA"},{user_name:"BBB"},{user_name:"CCC"},{user_name:"DDD"},{user_name:"EEE"},{user_name:"FFF"}].
After insert , sometimes the order will be BBB,AAA,FFF,EEE,DDD.