So I am trying to make a discord bot for me and my friends for tracking stats in CS GO 10 mans, and I am using cheerio for webscraping from the site that provides us the stats, and then pass them into mongodb. The scraping functionality works fine, but im trying to figure out how to avoid creating duplicate documents for each user. If I enter *userid 857575 it pulls the stats for that user, and puts in the DB, but if i call that multiple times, its making multiple documents in the DB. My question is, how would I get mongodb to update the document based on if the message author in discord matches the username in the db? So if username bob sends *userid3939 and bob already exists in the db, update the document. If bob doesnt exist, create document. code below, appreciate any tips.
module.exports.run = async (bot, message, args) => {
console.log(args);
var userUrl = 'https://popflash.site/user/' +args;
console.log(userUrl);
console.log(message.member.user.tag);
rp(userUrl)
.then(function (html) {
const arr = [];
var i = 0;
$('.stat-container', html).each(function (key, value) {
arr[i++] = $(this).find(".stat").text();
});
const stats = new Stats({
_id: mongoose.Types.ObjectId(),
userName: message.member.user.tag,
userId: args,
HLTV: arr[0],
ADR: arr[1],
HS: arr[2],
W: arr[3],
L: arr[4],
T: arr[5],
win_percent: arr[6]
});
stats.save()
.then(function (result) {
let botembed = new Discord.RichEmbed()
.setDescription(message.member.user + "'s 10 Man stats")
.setColor("#15f153")
.addField("stats", result)
return message.channel.send(botembed);
})
})
}
module.exports.help = {
name: "userid"
}
Through db.collection.update, you can specify the upsert: true option to get the behavior I think you're desiring. It will update an existing record if matched, otherwise it will create a new record.
Related
I am fairly new to writing functions and cannot figure out how to solve my issue or even search for it properly.
I have three collections:
current_projects
vendors
vendor_projects
when a new project is created I want the function to take all documents in vendors, add certain fields from them to vendor_projects, and include the project_id field that is created in current_projects.
Do I need a for loop to accomplish this or is there other syntax that could be utilized?
My current function is below. This creates on document using the new project_id field but doesnt take any of the fields from vendors. Any input is greatly appreciated.
exports.createProjVen = functions.firestore.document("/Current_projects/{id}")
.onCreate((snap, context)=>{
console.log(snap.data());
const id = snap.data().id;
// const collection = context.params.Current_projects;
// const id = context.params.id;
const projectVendors = admin.firestore().collection("project_vendors");
// const vendors = admin.firestore().collection("vendors");
return projectVendors.doc(id).set({
actual: "",
budget: "",
id: "23121",
project_id: id,
toggle: "true",
type: "Fixtures",
vendor_company: "tes",
vendor_contact: "tes",
vendor_email: "jj#j.com",
vendor_id: "test",
vendor_phone_number: "test"});
});
Adding more details:
When a new project is added it creates a record in current_projects.
I want the function to be able to query all documents in the vendor collection when a new record is created in current_projects.
From there, I want to grab the highlighted fields from the vendor documents, add the id from that was created from current_projects (highlighted in the first screen shot), and create a new document in project_vendors (third screen shot below) for every document in the vendors table.
If you are trying to created a document in project_vendors collection for every vendor in vendors after a project is created then you can map an array of promises and then use Promise.all() as shown below:
exports.createProjVen = functions.firestore.document("/Current_projects/{id}")
.onCreate((snap, context) => {
const docSnap = snap.data();
const id = context.params.id;
const vendorsSnap = await admin.firestore().collection("vendors").get();
const vendorsData = vendorsSnap.docs.map((d) => ({ id: d.id, ...d.data() }))
const promises = [];
const vendorPrsCol = admin.firestore().collection("project_vendors");
vendorsData.forEach((vendor) => {
const data = {
projectId: id,
email: vendor.email,
// other vendor fields
}
promises.push(vendorPrsCol.add(data));
})
await Promise.all(promises);
console.log("Vendor Projects added");
return null;
});
I have users and companies and want to store a company for each user and all of the users of each company in Firebase.
user={
"id":"tjkdEnc3skdm2Jjknd"
"name":"Adam",
"street":"Sideway 4",
"company":"dHend4sdkn25"
}
companies={
"id":"dHend4sdkn25",
"name":"Comp Ltd.",
"members":[
{
"id":"tjkdEnc3skdm2Jjknd"
"name":"Adam"
},{
"id":"dfjnUkJKB3sdn8n2kj"
"name":"Berta"
}
]
}
All explanations say that duplicate data is the best way to deal with and so I want to write some cloud functions to keep thigs in sync when editing on one of the sides.
Basically I started with
exports.userChangedCompany = functions.firestore
.document('users/{userId}')
.onUpdate((change, context) => {
const data = change.after.data();
const previousData = change.before.data();
if (data.company == previousData.company) {
return null;
}
else{
admin.firestore().doc('companies/'+data.company).set({ ... });
}
});
to update the companies when a user changed the company. Unfortunately I haven't found any hint how to set the new company-data properly.
Can someone please help me?
It sounds like you just need to remove user from members array of old company and add in that array of new company. You just need IDs of both companies.
async function updateCompanies(userId, username, oldCompanyId, newCompanyId) {
const companiesRef = await admin.firestore().collection("companies")
const userObj = {id: userId, name: username}
// Removing from old company and adding in new company
await Promise.all([
companiesRef.doc(oldCompanyId).update({members: admin.firestore.FieldValue.arrayRemove(userObj)}),
companiesRef.doc(newCompanyId).update({members: admin.firestore.FieldValue.arrayUnion(userObj)})
])
return true
}
You can just call this function in your cloud function. Just make sure you pass correct params. The reason why you need to pass the username as well is you have array of objects (members) and hence you need the complete object to add/remove using arrayUnion/arrayRemove.
What is the best approach to do a batch update or transaction, that reads a value of the first update, then uses this value to make further updates?
Here is an example:
//create person
const id = await db
.collection("person")
.add({ ...person })
.then(ref => ref.id)
//then do a series of updates
let batch = db.batch()
const private_doc = db
.collection("person")
.doc(id)
.collection("private")
.doc("data")
batch.set(private_doc, {
last_modified,
version: 1,
versions: []
})
const some_index = db.collection("data").doc("some_index")
batch.update(some_index, {
[id]: { first_name: person.first_name, last_name: person.last_name, last_modified }
})
const another_helpful_doc = db.collection("some_other_collection").doc("another_helpful_doc")
batch.update(another_helpful_doc, {
[id]: { first_name: person.first_name, last_name: person.last_name, image: person.image }
})
return batch.commit().then(() => {
person.id = id
return person
})
You can see here if there is an error any of the batch updates, the person doc will still be created - which is bad. I could add in a catch to delete the person doc if anything fails, however interested to see if this is possible with transactions or batches.
You can call the doc() method, without specifying any path, in order to create a DocumentReference with an auto-generated ID and, then, use the reference later. Note that the document corresponding to the DocumentReference is NOT created.
So, the following would do the trick, since all the writes/updates are included in the batched write:
const new_person_ref = db.collection("person").doc();
const id = new_person_ref.id;
let batch = db.batch()
batch.set(new_person_ref, { ...person })
const private_doc_ref = db // <- note the addition of ref to the variable name, it could help avoiding errors, as this is not a DocumentSnapshot but a DocumentReference.
.collection("person")
.doc(id)
.collection("private")
.doc("data")
batch.set(private_doc_ref, {
last_modified,
version: 1,
versions: []
})
//....
I am using NodeJS, PostgreSQL and the amazing pg-promise library. In my case, I want to execute three main queries:
Insert one tweet in the table 'tweets'.
In case there is hashtags in the tweet, insert them into another table 'hashtags'
Them link both tweet and hashtag in a third table 'hashtagmap' (many to many relational table)
Here is a sample of the request's body (JSON):
{
"id":"12344444",
"created_at":"1999-01-08 04:05:06 -8:00",
"userid":"#postman",
"tweet":"This is the first test from postman!",
"coordinates":"",
"favorite_count":"0",
"retweet_count":"2",
"hashtags":{
"0":{
"name":"test",
"relevancetraffic":"f",
"relevancedisaster":"f"
},
"1":{
"name":"postman",
"relevancetraffic":"f",
"relevancedisaster":"f"
},
"2":{
"name":"bestApp",
"relevancetraffic":"f",
"relevancedisaster":"f"
}
}
All the fields above should be included in the table "tweets" besides hashtags, that in turn should be included in the table "hashtags".
Here is the code I am using based on Nested transactions from pg-promise docs inside a NodeJS module. I guess I need nested transactions because I need to know both tweet_id and hashtag_id in order to link them in the hashtagmap table.
// Columns
var tweetCols = ['id','created_at','userid','tweet','coordinates','favorite_count','retweet_count'];
var hashtagCols = ['name','relevancetraffic','relevancedisaster'];
//pgp Column Sets
var cs_tweets = new pgp.helpers.ColumnSet(tweetCols, {table: 'tweets'});
var cs_hashtags = new pgp.helpers.ColumnSet(hashtagCols, {table:'hashtags'});
return{
// Transactions
add: body =>
rep.tx(t => {
return t.one(pgp.helpers.insert(body,cs_tweets)+" ON CONFLICT(id) DO UPDATE SET coordinates = "+body.coordinates+" RETURNING id")
.then(tweet => {
var queries = [];
for(var i = 0; i < body.hashtags.length; i++){
queries.push(
t.tx(t1 => {
return t1.one(pgp.helpers.insert(body.hashtags[i],cs_hashtags) + "ON CONFLICT(name) DO UPDATE SET fool ='f' RETURNING id")
.then(hash =>{
t1.tx(t2 =>{
return t2.none("INSERT INTO hashtagmap(tweetid,hashtagid) VALUES("+tweet.id+","+hash.id+") ON CONFLICT DO NOTHING");
});
});
}));
}
return t.batch(queries);
});
})
}
The problem is with this code I am being able to successfully insert the tweet but nothing happens then. I cannot insert the hashtags nor link the hashtag to the tweets.
Sorry but I am new to coding so I guess I didn't understood how to properly return from the transaction and how to perform this simple task. Hope you can help me.
Thank you in advance.
Jean
Improving on Jean Phelippe's own answer:
// Columns
var tweetCols = ['id', 'created_at', 'userid', 'tweet', 'coordinates', 'favorite_count', 'retweet_count'];
var hashtagCols = ['name', 'relevancetraffic', 'relevancedisaster'];
//pgp Column Sets
var cs_tweets = new pgp.helpers.ColumnSet(tweetCols, {table: 'tweets'});
var cs_hashtags = new pgp.helpers.ColumnSet(hashtagCols, {table: 'hashtags'});
return {
/* Tweets */
// Add a new tweet and update the corresponding hash tags
add: body =>
db.tx(t => {
return t.one(pgp.helpers.insert(body, cs_tweets) + ' ON CONFLICT(id) DO UPDATE SET coordinates = ' + body.coordinates + ' RETURNING id')
.then(tweet => {
var queries = Object.keys(body.hashtags).map((_, idx) => {
return t.one(pgp.helpers.insert(body.hashtags[i], cs_hashtags) + 'ON CONFLICT(name) DO UPDATE SET fool = $1 RETURNING id', 'f')
.then(hash => {
return t.none('INSERT INTO hashtagmap(tweetid, hashtagid) VALUES($1, $2) ON CONFLICT DO NOTHING', [+tweet.id, +hash.id]);
});
});
return t.batch(queries);
});
})
.then(data => {
// transaction was committed;
// data = [null, null,...] as per t.none('INSERT INTO hashtagmap...
})
.catch(error => {
// transaction rolled back
})
},
NOTES:
As per my notes earlier, you must chain all queries, or else you will end up with loose promises
Stay away from nested transactions, unless you understand exactly how they work in PostgreSQL (read this, and specifically the Limitations section).
Avoid manual query formatting, it is not safe, always rely on the library's query formatting.
Unless you are passing the result of transaction somewhere else, you should at least provide the .catch handler.
P.S. For the syntax like +tweet.id, it is the same as parseInt(tweet.id), just shorter, in case those are strings ;)
For those who will face similar problem, I will post the answer.
Firstly, my mistakes:
In the for loop : body.hashtag.length doesn't exist because I am dealing with an object (very basic mistake here). Changed to Object.keys(body.hashtags).length
Why using so many transactions? Following the answer by vitaly-t in: Interdependent Transactions with pg-promise I removed the extra transactions. It's not yet clear for me how you can open one transaction and use the result of one query into another in the same transaction.
Here is the final code:
// Columns
var tweetCols = ['id','created_at','userid','tweet','coordinates','favorite_count','retweet_count'];
var hashtagCols = ['name','relevancetraffic','relevancedisaster'];
//pgp Column Sets
var cs_tweets = new pgp.helpers.ColumnSet(tweetCols, {table: 'tweets'});
var cs_hashtags = new pgp.helpers.ColumnSet(hashtagCols, {table:'hashtags'});
return {
/* Tweets */
// Add a new tweet and update the corresponding hashtags
add: body =>
rep.tx(t => {
return t.one(pgp.helpers.insert(body,cs_tweets)+" ON CONFLICT(id) DO UPDATE SET coordinates = "+body.coordinates+" RETURNING id")
.then(tweet => {
var queries = [];
for(var i = 0; i < Object.keys(body.hashtags).length; i++){
queries.push(
t.one(pgp.helpers.insert(body.hashtags[i],cs_hashtags) + "ON CONFLICT(name) DO UPDATE SET fool ='f' RETURNING id")
.then(hash =>{
t.none("INSERT INTO hashtagmap(tweetid,hashtagid) VALUES("+tweet.id+","+hash.id+") ON CONFLICT DO NOTHING");
})
);
}
return t.batch(queries);
});
}),
I have the following code in my chat application based on NodeJS and MongoDB to change admin for room:
export function setAdmin(room, user) {
const userGuid = getGuid(user);
if (room.users && room.users.length) {
// filter current and new
room.users = room.users.filter(guid =>
guid !== room.adminGuid && guid !== userGuid
);
} else {
room.users = [];
}
room.users.push(userGuid);
room.adminGuid = userGuid;
return roomsCollection.save(room, { w: 1 });
}
Each room have only 2 users: admin and customer.
To update several rooms:
const rooms = await roomsCollection.find({ adminGuid: currentAdminGuid }).toArray();
for (const room of rooms) {
await setAdmin(room, newAdminUser);
}
I had some problems under highload with my application. They were resolved with help of indexes.
But after, working with MongoDB dump I found out that I have rooms with 3 users guids in room.users array. I think that save works as update for exist document, but how it updates array? If $set, why I have such 3 users rooms. Any thoughts how it would be possible?
save() behaves as insert if document does not exist. You have to use update() with {upsert: false}
some resources to update array: Array Query, array Operator
simple tutorial to nest array inside document, here
advanced example of dealing with multiple nested array, here