How to store a list in firebase firestore? - node.js

I am trying to store a list with firebase firestore, but I always get a encoding error:
UnhandledPromiseRejectionWarning: Error: Cannot encode value: a,b,c
I have tried the following code snippets for node js:
const data = {
a: "hello world",
b: ["a", "b", "c"]
};
const res = await db.collection('data').doc('one').set(data);
and
const data = {
a: "hello world",
b: new Array(["a", "b", "c"])
};
const res = await db.collection('data').doc('one').set(data);
and
const data = {
a: "hello world",
b: []
};
const res = await db.collection('data').doc('one').set(data);
of which these all result in the same encoding error.
I have also tried to store a list as the following the blog post here: https://firebase.googleblog.com/2014/04/best-practices-arrays-in-firebase.html
{
0: "a",
1: "b",
2: "c"
}
but this stored it as a object not a list and I need to store it as a list.
I can create new documents with lists in the firebase console, but i cant do it programmatically. I am using node js.
Thank you for aany help you can offer.

So, I found a solution after looking at the firebase docs a bit more.
This page : https://firebase.google.com/docs/firestore/manage-data/add-data told me that I could use array union to do this. Here is the working code sample:
const data = {
a: "hello world",
b: admin.firestore.FieldValue.arrayUnion("a")
};
const res = await db.collection('data').doc('one').set(data);

Related

Get map of map field from Firestore

I have a field of type map that contains Maps of data in firestore.
I am trying to retrieve this data using a cloud function in node.js. I can get the document and the data from the field but i can't get it in a usable way. I have tried every solution i can find on SO and google but the below is the only code that can give me access to the data. I obviously need to be able to access each field with in the Map individually. in swift i build an array of String:Any but i can get that to work in Node.JS
const docRef = dbConst.collection('Comps').doc('XEVDk6e4AXZPkNprQRn5Imfcsah11598092006.724980');
return docRef.get().then(docSnap => {
const tagets = docSnap.get('targets')
console.log(tagets);
}).catch(result => { console.log(result) });
this is what i am getting back in the console.
In Swift i do the following and am so far not able to find an equivalent in typescript. (i don't need to build the custom object just ability to access the keys and values)
let obj1 = doc.get("targets") as! [String:Any]
for objs in obj1{
let obs = objs.value as! [String:Any]
let targObj = compUserDetails(IDString: objs.key, activTarg: obs["ActivTarget"] as! Double, stepTarg: obs["StepTarget"] as! Double, name: obs["FullName"] as! String)
UPDATE
After spending a whole day working on it thought i had a solution using the below:
const docRef = dbConst.collection('Comps').doc('XEVDk6e4AXZPkNprQRn5Imfcsah11598092006.724980');
return docRef.get().then(docSnap => {
const tagets = docSnap.get('targets') as [[string, any]];
const newDataMap = [];
for (let [key, value] of Object.entries(tagets)) {
const tempMap = new Map<String,any>();
console.log(key);
const newreWorked = value;
tempMap.set('uid',key);
for(let [key1, value1] of Object.entries(newreWorked)){
tempMap.set(key1,value1);
newDatMap.push(tempMap);
};
};
newDatMap.forEach(element => {
const name = element.get('FullName');
console.log(name);
});
However the new data map has 6 seperate mapped objects. 3 of each of the original objects from the cloud. i can now iterate through and get the data for a given key but i have 3 times as many objects.
So after two days of searching an getting very close i finnaly worked out a solution, it is very similar to the code above but this works. it may not be the "correct" way but it works. feel free to make other suggestions.
return docRef.get().then(docSnap => {
const tagets = docSnap.get('targets') as [[string, any]];
const newDatarray = [];
for (let [key, value] of Object.entries(tagets)) {
const tempMap = new Map<String,any>();
const newreWorked = value;
tempMap.set('uid',key);
for(let [key1, value1] of Object.entries(newreWorked)){
tempMap.set(key1,value1);
};
newDatarray.push(tempMap);
};
newDatarray.forEach(element => {
const name = element.get('FullName');
const steps = element.get('StepTarget');
const avtiv = element.get('ActivTarget');
const UID = element.get('uid');
console.log(name);
console.log(steps);
console.log(avtiv);
console.log(UID);
});
}).catch(result => { console.log(result) });
I made this into a little function that gets the underlying object from a map:
function getMappedValues(map) {
var tempMap = {};
for (const [key, value] of Object.entries(map)) {
tempMap[key] = value;
}
return tempMap;
}
For an object with an array of maps in firestore, you can get the value of the first of those maps like so:
let doc = { // Example firestore document data
items: {
0: {
id: "1",
sr: "A",
},
1: {
id: "2",
sr: "B",
},
2: {
id: "3",
sr: "B",
},
},
};
console.log(getMappedValues(doc.items[0]));
which would read { id: '1', sr: 'A' }

Filtering Out unnecessary portions of scraped data

I am trying to make a scraper that scrapes Post ID and Poster's ID from a Facebook public post link, using puppeteer and nodejs.
(async() => {
let url = 'https://m.facebook.com/photo/?fbid=1168301430177531&set=gm.1386874671702414'; //demo link
let brw = await puppeteer.launch();
let page = await brw.newPage();
await page.goto(url,{ waitUntil:'networkidle2'});
let data = await page.evaluate(()=>{
let ids = document.querySelector('div[class="_57-p"] > a[class="_57-s touchable"]').search; // for
image post
return{
ids
}
});
console.log(data);
and I get output like:
{
ids: '?fbid=1168301430177531&id=100009930549147&set=gm.1386874671702414&refid=13&__tn__=%2B%3E'
}
how can I filter out the unnecessary portions?(I just want fbid and id values)
Thanks in advance
It seems this is the most reliable and simple way:
const href = document.querySelector('div[class="_57-p"] > a[class="_57-s touchable"]').href;
const searchParams = new URL(href).searchParams;
return {
fbid: searchParams.get('fbid'),
id: searchParams.get('id'),
};
Try use query-string.
it will help you to parse the query strings
let search = '?foo=bar'
const parsed = queryString.parse(search);
console.log(parsed);
//=> {foo: 'bar'}
it is just a simple example of what you should do
You can use a match() method with a regex like /\Wfbid=(\w+)(\W|$)/, the search result under index 1 of the capturing groups will contain the desired parameter value.
let ids = '?fbid=1168301430177531&id=100009930549147&set=gm.1386874671702414&refid=13&__tn__=%2B%3E'
const fbid = ids.match(/\Wfbid=(\w+)(\W|$)/)[1] // 1168301430177531
const id = ids.match(/\Wid=(\w+)(\W|$)/)[1] // 100009930549147
Without [1] you'd get all matches, e.g.:
ids.match(/\Wid=(\w+)(\W|$)/)
=>
["&id=100009930549147&", "100009930549147", "&", index: 22, input: "?fbid=1168301430177531&id=100009930549147&set=gm.1386874671702414&refid=13&__tn__=%2B%3E", groups: undefined]
And you need the string between the capturing & characters, the 2nd element of the array (so: [1]).

Stringifying JSON data loses object I'm trying to stringify

I'm new to NodeJS and I'm trying to learn it a bit better by writing a Discord bot. But I'm having an issue with stringifying an object to JSON. I think it's having an issue with the array I'm setting, but I'm not sure how else I would do this. If I'm not supposed to set an array and using my guildMembers example below, how else should I insert this data into my JSON file?
I've looked through a few examples here on StackOverflow and found this particular article: JSON Stringify Removing Data From Object. However, it's not clear to me given what I'm trying to achieve.
var o = {};
var guildKey = guild.id;
o[guildKey] = [];
o[guildKey]['guildMembers'] = {};
var guildMembers = []
guild.members.forEach(function(guildMember, guildMemberId) {
if (!guildMember.user.bot){
var memberData = {
id: guildMember.id,
data: {
userName: guildMember.user.username,
nickName: guildMember.nickname,
displayName: guildMember.displayName,
joinedAt: guildMember.joinedAt
}
}
guildMembers.push(memberData);
};
});
o[guildKey]['guildMembers'] = guildMembers;
json = JSON.stringify(o);
I am expecting the data to show the guildMembers object with the array under the guildKey object. However, the JSON shows only the guildKey object with an empty array:
{"guildKey":[]}
You make guildKey an array and then try to use it as an object ...
Solution is:
o[guildKey] = {};
Just like in the mentioned post.
It is because you are assigning a key-value pair to an array with
o[guildKey]['guildMembers'] = { }; <= o[guildKey]
When iterating over an array to display the contents, JavaScript does not access non-integer keys. You should store o[guildKey] as an object instead.

how to get one page data list and total count from database with knex.js?

I have a user table with some records(such as 100), how can I get one page data and total count from it when there are some where conditions?
I tried the following:
var model = knex.table('users').where('status',1).where('online', 1)
var totalCount = await model.count();
var data = model.offset(0).limit(10).select()
return {
totalCount: totalCount[0]['count']
data: data
}
but I get
{
"totalCount": "11",
"data": [
{
"count": "11"
}
]
}
, how can I get dataList without write where twice? I don't want to do like this:
var totalCount = await knex.table('users').where('status',1).where('online', 1).count();
var data = await knex.table('users').where('status',1).where('online', 1).offset(0).limit(10).select()
return {
totalCount: totalCount[0]['count']
data: data
}
Thank you :)
You probably should use higher level library like Objection.js which has already convenience method for getting pages and total count.
You can do it like this with knex:
// query builder is mutable so when you call method it will change builders internal state
const query = knex('users').where('status',1).where('online', 1);
// by cloning original query you can reuse common parts of the query
const total = await query.clone().count();
const data = await query.clone().offset(0).limit(10);

collection.find on an item in a subarray

I have the following Object Structure:
[
{
name: "someThing"
,securities: [ {"id": "2241926"} ]
}
]
I want to be able to return all objects in the outer array, that has at least one child secuirty with an id that starts with a value. I have tried a few things and keep running up short. On the mongoo console, this query works:
db.caseSummary.find({"securities.id": /^224.*/i})
We are using ES6, so please apologies for the generator syntax:
const q = require("q");
const startsWith = function(term){
return new RegExp('^' + term + '.*', 'i')
}
const find = function*(collection, criteria){
const command = q.nbind(collection.find, collection),
myCriteria = criteria || {},
results = yield command(myCriteria),
toArray = q.nbind(results.toArray, results) ;
return yield toArray()
}
const searchBySecurity = function*(mongo, term) {
const collection = mongo.collection("caseSummary"),
criteria = {"securities.id": startsWith(term) };
return yield find(collection, criteria);
}
so searchBySecurity(this.mongo, '224') It returns an empty array, it should return the same results as the mongo console. I guess I am missing a pretty basic concept in translating my search criteria or invoking the find writing this in node from raw mongo console query.
Edit 1: to be clear I want to return all parent objects, which have subarrays that contain a value that starts with the term passed in...
Edit 2:
I changed the criteria to be:
criteria = { "securities": {
"$elemMatch": {
"id": "901774109" //common.startsWith(term)
}
}
};
Still the same results.
Edit 3:
Using nodejs - mongodb "version": "1.4.38"
Edit 4:
this ended up not being an issue

Resources