Parsing CSV string with nodeJS without a library - node.js

I want to create a cloud function in firebase that parses csv files and adds them as a firestore document. Somehow, none of the libraries I try seem to work. Whenever I add a csv library, the function won't even deploy. therefore, I want to parse a CSV file without using a library. However, I cant seem to figure out how to parse it correctly when using maps and arrays. I have the following string in string type. Replaced some data with placeholders.
subject,contacts,dateStart,dateEnd,timeStart,timeEnd,isRecurring,frequence,location,room,visitors,sendInvite
Sollicitatie,[{"name":"<name here>","email":"<email here>","phoneNumber":"<phone number here>"}],28-09-2020,28-09-2020,13:00,14:00,true,yearly,<location here>,2,[{"name":"<name here>","email":"<email here>","visitorImage":"imgRef","visitorType":"Relation","phoneNumber":"<telephone here>","licencePlate":"<licencePlate>"}],true
I want to get this data in firestore as the following:
subject: subject,
contacts:[
{
name: name,
email: email,
phoneNumber: phoneNumber
},
{
name: name,
email: email,
phoneNumber
}
],
dateStart: dateStart,
dateEnd, dateEnd,
timeStart, timeEnd,
isRecurring, isRecurring,
frequence, frequence,
location: location,
room: room,
visitors:[
{
name: name,
email: email,
phoneNumber: phoneNumber
},
{
name: name,
email: email,
phoneNumber
//Whatever data also needs to be added
}
]
Is there a library that will work/is there a problem with my package.json, or is there a script that will work? I tried to write a script that will do this, but it is far from working. I have tried the following and got the right strings, but I cant get the arrays the right way.
const convert = (from, to) => str => Buffer.from(str, from).toString(to);
const hexToUtf8 = convert('hex', 'utf8');
const string = hexToUtf8(contents);
const data = string.split('\n');
for(i=0;i<=data.length - 1;i++){
if(i == 0){
continue;
}
const row = data[i];
const rowValues = [];
const elems = row.split(",");
console.log(elems);
var arrayParse;
var parsing = 0;
for(j=0;j<=elems.length - 1;j++){
const elem = elems[j];
if(parsing == 1){
arrayParse += "," + elem;
continue;
}else if(elem[0] == "["){
arrayParse += elem;
parsing = 1;
continue;
}else if(elem[elem.length - 1] == "]"){
arrayParse += elem;
parsing = 0;
rowValues.push(arrayParse);
arrayParse = "";
continue;
}else{
rowValues.push(elem);
}
}
console.log(rowValues);

Related

How to upload big data to mongodb

I need to upload a large amount of data to mongodb.
not a file, but rather a very large amount of key value pairs.
example:
let payload = [];
for ( let i =0 ;i<1000000;i++){
payload.push({
"first name": "juan",
"hair color": ""+i,
"gender" :"male"
})
}
var body = {
"channelId":"63dd281360e269e2a9399939",
"recordCount":payload.length,
"minBidUSD": 5,
"payload": payload
}
the script above creates a huge payload. that payload is then put on the body of a POST request to our system.
I need to be able to store very large amounts of data in that payload.
context of functionality: I am working on a website where people can sell data leads. ex: I am looking for males in florida that are 30 years old. the possible data structure for that lead payload would be:
{gender:"male",state:"florida",age:30}
this is the question that have::
[1] is there a better way to store this data?
--note: the attributes or this payload change, so i cant create a model against it.
[2] if the best way to store this data is using gridfs-file-storage, how do I do it?
Additional notes about the question::
here is the model for the collection that holds the payload
const mongoose = require("mongoose");
const channelDataSchema = mongoose.Schema({
channelId: { type: String, required: true },
payload: { type: [Object], required:true },
});
module.exports = mongoose.model("ChannelData", channelDataSchema);
Gridfs is for binary data and is chunked up to avoid the 16mb MongoDB document size limitation. I don't recommend gridfs for basic data.
For rapid data ingestion, I recommend using bulkwrite. Here is a mongoshell example. This example will loop creating a bunch of fake data elements made up using the random() function. It is for illustration only. If you need to insert data from a source, such as a file, or another database system you can use bulkwrite using a custom programming language, such as Node, and using the MongoDB supported database driver - which includes bulkwrite features.
MongoShell Example:
use mydatabase;
function getRandomInteger(min, max) {
// NOT INCLUSIVE OF MAX. IF NEED TO INCLUDE MAX, BUMP IT UP BY ONE
return Math.floor(Math.random() * (max - min) + min);
}
function getRandomAlphaNumeric(length) {
var result = [];
var characters = 'abcdefghijklmnopqrstuvwxwyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789';
var charactersLength = characters.length;
for ( var i = 0; i < length; i++ ) {
result.push(characters.charAt(Math.floor(Math.random() * charactersLength)));
}
return result.join('');
}
function generateDocument() {
return {
stringFixedLength01: getRandomAlphaNumeric(1),
stringFixedLength02: getRandomAlphaNumeric(2),
stringFixedLength03: getRandomAlphaNumeric(3),
stringFixedLength04: getRandomAlphaNumeric(4),
stringFixedLength05: getRandomAlphaNumeric(5),
stringVariableLength: getRandomAlphaNumeric(getRandomInteger(5, 50)),
integer1: NumberInt(getRandomInteger(0, 2000000)),
long1: NumberLong(getRandomInteger(0, 100000000)),
date1: new Date(),
guid1: new UUID()
};
}
for (var j = 0; j < 1000000; j++) {
var batch=[];
for (var i = 0; i < 50000; i++) {
batch.push( { insertOne: { document: generateDocument() } } );
if(i % 10000 == 0) {
print("outerloop: " + j + ", innerloop: " + i);
}
}
print("BulkWrite command submitted...");
db.mycollection.bulkWrite(batch, {ordered: false});
}

Get map of map field from Firestore

I have a field of type map that contains Maps of data in firestore.
I am trying to retrieve this data using a cloud function in node.js. I can get the document and the data from the field but i can't get it in a usable way. I have tried every solution i can find on SO and google but the below is the only code that can give me access to the data. I obviously need to be able to access each field with in the Map individually. in swift i build an array of String:Any but i can get that to work in Node.JS
const docRef = dbConst.collection('Comps').doc('XEVDk6e4AXZPkNprQRn5Imfcsah11598092006.724980');
return docRef.get().then(docSnap => {
const tagets = docSnap.get('targets')
console.log(tagets);
}).catch(result => { console.log(result) });
this is what i am getting back in the console.
In Swift i do the following and am so far not able to find an equivalent in typescript. (i don't need to build the custom object just ability to access the keys and values)
let obj1 = doc.get("targets") as! [String:Any]
for objs in obj1{
let obs = objs.value as! [String:Any]
let targObj = compUserDetails(IDString: objs.key, activTarg: obs["ActivTarget"] as! Double, stepTarg: obs["StepTarget"] as! Double, name: obs["FullName"] as! String)
UPDATE
After spending a whole day working on it thought i had a solution using the below:
const docRef = dbConst.collection('Comps').doc('XEVDk6e4AXZPkNprQRn5Imfcsah11598092006.724980');
return docRef.get().then(docSnap => {
const tagets = docSnap.get('targets') as [[string, any]];
const newDataMap = [];
for (let [key, value] of Object.entries(tagets)) {
const tempMap = new Map<String,any>();
console.log(key);
const newreWorked = value;
tempMap.set('uid',key);
for(let [key1, value1] of Object.entries(newreWorked)){
tempMap.set(key1,value1);
newDatMap.push(tempMap);
};
};
newDatMap.forEach(element => {
const name = element.get('FullName');
console.log(name);
});
However the new data map has 6 seperate mapped objects. 3 of each of the original objects from the cloud. i can now iterate through and get the data for a given key but i have 3 times as many objects.
So after two days of searching an getting very close i finnaly worked out a solution, it is very similar to the code above but this works. it may not be the "correct" way but it works. feel free to make other suggestions.
return docRef.get().then(docSnap => {
const tagets = docSnap.get('targets') as [[string, any]];
const newDatarray = [];
for (let [key, value] of Object.entries(tagets)) {
const tempMap = new Map<String,any>();
const newreWorked = value;
tempMap.set('uid',key);
for(let [key1, value1] of Object.entries(newreWorked)){
tempMap.set(key1,value1);
};
newDatarray.push(tempMap);
};
newDatarray.forEach(element => {
const name = element.get('FullName');
const steps = element.get('StepTarget');
const avtiv = element.get('ActivTarget');
const UID = element.get('uid');
console.log(name);
console.log(steps);
console.log(avtiv);
console.log(UID);
});
}).catch(result => { console.log(result) });
I made this into a little function that gets the underlying object from a map:
function getMappedValues(map) {
var tempMap = {};
for (const [key, value] of Object.entries(map)) {
tempMap[key] = value;
}
return tempMap;
}
For an object with an array of maps in firestore, you can get the value of the first of those maps like so:
let doc = { // Example firestore document data
items: {
0: {
id: "1",
sr: "A",
},
1: {
id: "2",
sr: "B",
},
2: {
id: "3",
sr: "B",
},
},
};
console.log(getMappedValues(doc.items[0]));
which would read { id: '1', sr: 'A' }

Duplicate an object in array in mongoose

I want to duplicate an object in array in mongoose.
For example, if there is a object which projectName is "a", I want to create a complete same one, but the name is called "a duplicated".
I tried to the following code:
project.find({
"logininfo.username": username,
}, (err, users) => {
...
var user= users[0];
var dproject;
for(var i in user.projects){
if(user.projects[i].projectName === projectName)
{
dproject = user.projects[i];
break;
}
}
dproject.projectName += " duplicated";
dproject._id = new mongoose.Types.ObjectId();
user.projects.push(dproject)
projects.save((err, user) => {
...
However, this function will also change the previous object, which creates a completely same object (both names change to "a duplicated"), these two objects have the same _id.
How can I duplicate the object but keep the projectName and _id different?
Use JSON.parse(JSON.stringify(object)) to deep copy. It will make a copy but not a reference.
for (var i in projects.projects) {
if (projects.projects[i].projectName === projectName) {
dproject = JSON.parse(JSON.stringify(projects.projects[i])); //deep copy
break;
}
}
dproject._id = new mongoose.Types.ObjectId();
dproject.projectName += " duplicated";
projects.projects.push(dproject)

Creating an SMS group forwarder in Twilio with NodeJS

Here's what I'm trying to accomplish:
Set a list of names and numbers (my "group")
When a text message is sent to the Twilio number, forward it on to every member in the group
At a high-level, the idea seems straight forward enough. My programming / syntax skills are rusty, though, and I'd love some help.
I'm using Twilio Functions, and I've been able to send and receive messages successfully. Now I am stuck on how to carry this idea of iterating through a group.
Here's what I've written so far:
var groupmembers = {
jonathan:{
name: 'Jonathan',
number: '+0000000000'
},
joshua:{
name: 'Joshua',
number: '+1110000000'
}
}
exports.handler = function(context, event, callback) {
// Set some values for later use
this.fromNumber = event.From
this.body = event.Body || ''
let twiml = new Twilio.twiml.MessagingResponse();
groupmembers.forEach(function(member) {
// Skip sending if it's the same number
if (member.number === this.fromNumber ) {
return;
}
// Otherwise, let's send a mesage!
twiml.message("Hello World").to( member.number );
callback(null, twiml);
});
};
The issues I believe I have:
Not being sure how to properly set my array or "dictionary"
Not knowing the proper syntax for passing the "to" variable to the message
Not knowing the proper syntax for doing a loop in NodeJS (the Functions console is telling me that 'groupmembers.forEach is not a function')
Thank you for any and all feedback and for pointing me in the right direction!
The mistake you have is pretty simple. groupmembers is an object, you want an array.
You may want something akin to this instead:
var groupmembers = [{
name: 'Jonathan',
number: '+0000000000'
},
{
name: 'Joshua',
number: '+1110000000'
}]
Apart from that, it looks to me as okay.
With more searching and the point in the right direction from Slava I was able to figure it out! Here's the complete code:
/**
* Represents a search trough an array.
* #function search
* #param {Array} array - The array you wanna search trough
* #param {string} key - The key to search for
* #param {string} [prop] - The property name to find it in
* Props: https://stackoverflow.com/a/33097318/315818
*/
function search(array, key, prop){
// Optional, but fallback to key['name'] if not selected
prop = (typeof prop === 'undefined') ? 'name' : prop;
for (var i=0; i < array.length; i++) {
if (array[i][prop] === key) {
return array[i];
}
}
}
var groupmembers = [
{
name: 'Jonathan',
number: '+000000000'
},
{
name: 'Joshua',
number: '+111111111'
}
];
exports.handler = function(context, event, callback) {
let twiml = new Twilio.twiml.MessagingResponse();
// Search for the group member that matches the sender number
let sender = search(groupmembers, event.From, 'number');
// Now, loop through each of the group members
groupmembers.forEach(function(member) {
// Skip sending if it's the same number
if (member.number === event.From ) {
return;
}
// Now, forward on the message to the group member, using the sender's name
twiml.message(`${sender.name}: ${event.Body}`, {
to: member.number
});
});
// Loop ended
callback(null, twiml);
};

Mongoose handle duplicate url slug

I am storing url slugs from blog titles in MongoDB using Mongoose. So for instances when users enter the same blog title, I want to do it like how Wordpress handles it which is appending a number on the slug and increment it for every duplicates.
Example:
blog-title
blog-title-2
blog-title-3
Also when the slug evaluated from a blog title is blog-title-2 and there is already a blog-title-2 slug, it will be smart enough to append -2 at the back instead of incrementing the number. So it will be blog-title-2-2.
How do I implement that?
I managed to figure out how to implement it myself.
function setSlug(req, res, next) {
// remove special chars, trim spaces, replace all spaces to dashes, lowercase everything
var slug = req.body.title.replace(/[^\w\s]/gi, '').trim().replace(/\s+/g, '-').toLowerCase();
var counter = 2;
// check if there is existing blog with same slug
Blog.findOne({ slug: slug }, checkSlug);
// recursive function for checking repetitively
function checkSlug(err, existingBlog) {
if(existingBlog) { // if there is blog with the same slug
if(counter == 2) // if first round, append '-2'
slug = slug.concat('-' + counter++);
else // increment counter on slug (eg: '-2' becomes '-3')
slug = slug.replace(new RegExp(counter++ + '$', 'g'), counter);
Blog.findOne({ slug: slug }, checkSlug); // check again with the new slug
} else { // else the slug is set
req.body.slug = slug;
next();
}
};
}
I played around with Wordpress for a bit; publishing a lot of test blog posts with weird titles just to see how it handles the title conversion and I implemented the first step in converting the title according to my discoveries. Wordpress:
removes all special characters
trims away leading and ending spaces
converts remaining spaces to single dashes
lowercase everything
In your mongooese model you can give the field the unique option:
var post = new post({
slug: {
type: String,
unique: true
}
});
Then when you save it you can validate and know if the slug is not unique, in which case you can modify the slug:
var post = new post({ slug: "i-am-slug"});
var tempSlug = post.slug
var slugIsUnique = true;
car counter = 1;
do{
error = post.validateSync();
if(// not unique ){ //Make sure to only check for errors on post.slug here
slugIsUnique = false;
counter++;
tempSlug = post.slug + "-" + counter;
}
}while(!slugIsUnique)
post.slug = tempSlug;
post.save((err, post) => {
// check for error or do the usual thing
});
Edit:
This wont take blog-title-2 and output blog-title-2-2 but it will ouput blog-title-2-1 unless that already exists
I found the following code worked for me:
The general idea is to create an array of slugs you may potentially want to use and then use the MongoDB's $in query operator to determine if any of those slugs exist.
Note: The below solutions uses NPM's slugify. This is optional. I also use crytpo for randomizing since it has higher performance. Which is also optional.
private getUniqueSlug(title: string) {
// Returns a promise to allow async code.
return new Promise((resolve, reject) => {
const uniqueSlug: string = "";
// uses npm slugify. You can use whichever library you want or make your own.
const slug: string = slugify(title, { lower: true, strict: true });
// determines if slug is an ObjectID
const slugIsObjId: boolean = (ObjectId.isValid(slug) && !!slug.match(/^[0-9a-fA-F]{24}$/));
// creates a list of slugs (add/remove to your preference)
const slugs: string[] = [];
// ensures slug is not an ObjectID
slugIsObjId ? slugs.push(slug + "(1)") : slugs.push(slug);
slugs.push(slug + "(2)");
slugs.push(slug + "(3)");
slugs.push(slug + "(4)");
slugs.push(slug + "(5)");
// Optional. 3 random as fallback (crypto used to generate a random 4 character string)
for (let x = 0; x < 2; x++) {
slugs.push(slug + "(" + crypto.randomBytes(2).toString("hex") + ")");
}
// Uses a single find instance for performance purposes
// $in searches for all collections with a slug in slugs array above
const query: any = { slug: { $in: slugs } };
Collection.find(query, { slug: true }).then((results) => {
if (results) {
results.forEach((result) => {
slugs.every((s, si) => {
// If match found, remove from slugs since that slug ID is not valid.
if (s === result.slug) { slugs.splice(si, 1); return false; } else { return true; }
});
});
}
// returns first slug. Slugs are ordered by priority
if (slugs.length > 0) { resolve(slugs[0]); } else {
reject("Unable to generate a unique slug."); // Note: If no slug, then fails. Can use ObjectID as failsafe (nearly impossible).
}
}, (err) => {
reject("Find failed");
});
});
}

Resources