I am trying to populate my MongoDB using data from a CSV file. There are currently no databases or collections in my MongoDB and I would like to create these using an update function that creates objects parsed from a csv file.
I am using ya-csv to parse my csv file and the mongodb driver for node.
My code looks like this:
var csv = require('ya-csv');
var fs = require('fs');
var MongoClient = require('mongodb').MongoClient;
var Server = require('mongodb').Server;
var mongoclient = new MongoClient(new Server('localhost', 27017, {'native_parser' : true}));
var reader = csv.createCsvFileReader('YT5.csv', {columnsFromHeader:true,'separator': ','});
reader.addListener('data', function(data){
var nameHolder = data.name;
//I have no issue getting the needed variables from my csv file
mongoclient.db(nameHolder).collection('assets').update({assetId:data.assetId,name:data.name},{upsert:true},function(err,updated){if(err){console.log(err)}});
reader.addListener('end', function(data){
console.log("done");
}
I have not created the databases or collections for this, but can it do this for me with this update? I get an error:
[Error: Connection was destroyed by application]
When I run this, the databases get created but they're empty.
Any help would be greatly appreciated.
Unless there is a specific need to use NodeJS, say to not only reorder but make some complex modification to fields read from CSV file, use mongoimport.
If all you need to do is to skip and reorder some fields - work the fields with simple awk, skipping and changing order as needed:
cat /tmp/csv | awk -F',' 'BEGIN{OFS=","} {print $1,$2,$4,$3,$10}' | mongoimport --type csv --db test --collection csv_import
Additionally, if there is a need to change collection or db name based on the csv values (field 10 in this example is used as db name and field 11 as collection):
cat /tmp/csv | awk -F',' 'BEGIN{OFS=","} {print $1,$2,$4,$3,$10 | "mongoimport --type csv --db "$10" --collection "$11 }'
Or you could split the file into the per db/collection chunks first.
If you convert the csv rows in a javascript array (each row is an object), you can use https://github.com/bitliner/MongoDbPopulator
Related
I want to make use of multiple databases in a project, but I am having difficulty in calling the
secondary databases across many route controllers. I do not want to close connection to the main databases
the current method which I have used to insert data into multiple controllers I believe is not efficient
enough i.e.
------>> all these are called every time a file needs to connect to the secondary database
----------------------------------------------------------------------------------------|
var mongoose = require('mongoose'); |
var mongooseConnect = mongoose.createConnection('mongodb://localhost/27017/'+dbName); |
|
require('./models'); |
var modelData = mongooseConnect.model('models'); |
----------------------------------------------------------------------------------------|
modelData.save()
I am looking for a method which would allow me to
connect to the secondary database once and call it anytime i want to across the controllers
combine the above code into a one line code
I have called the main database through the normal means which is
route.js
var mongoose = require('mongoose');
mongoose.connect(mongodbhost/maindb)
then in the schema I just simply require mongoose before using it in the schema. and only the schema is called in the
route controllers.
Any help would be appreciated
You can use the same schema in multiple database but you would need to create the connections to both databases. Here's how you can do it:
const conn1 = mongoose.createConnection('mongodb://localhost:27017/test1');
const conn2 = mongoose.createConnection('mongodb://localhost:27017/test2');
const TestSchema = require('./TestSchema');
var TestModelDb1 = conn1.model('Model', TestSchema);
var TestModelDb2 = conn2.model('Model', TestSchema);
Hello friends I am new in nodejs and mongodb. Is it possible to upload multiple files inside single mongodb doc with other information.
You can make use of BSON (Binary JSON) to store files in MongoDB collections. However, BSON has a limit of 16MB. If you are planning to store files bigger than that, consider GridFS
You can write files to MongoDB like so in Node.js:
var Binary = require(‘mongodb’).Binary;
//Read the file that you want to store
var file_data = fs.readFileSync(file_path);
var db_doc = {};
db_doc.file_data= Binary(file_data);
var my_collection = db.collection(‘files’);
my_collection.insert(db_doc, function(err, result){
//more code..
....
Searching for an efficient and quick way to stream a large (10 million lines) of csv data into a mongoose database.
Problems that arise are dealing with streaming instead of importing which could be solve with fs.createReadStream (although still learning how to use it) and how to deal with inserting that large amount of data into the mongoDB using mongoose because overloading mongoose/mongo with insert requests could lead to some errors.
you simply need 'stream-to-mongo-db' and 'csvtojson' npm library.
here is the example code i use to dump millions of records from BIG csv files. It just works!
const fs = require('fs');
const csv = require('csvtojson');
const streamToMongoDB = require('stream-to-mongo-db').streamToMongoDB;
const csvFile = './bigFiles.csv';
const dbURL = 'mongodb://localhost/tweets';
const collection = 'tweets';
fs.createReadStream(csvFile)
.pipe(csv())
.pipe(streamToMongoDB({
dbURL : dbURL, collection : collection
}))
there is a insertMany() method in mongoose. but that only lets you insert 10 000 docs only per once so.. My solution is to you loop asyncly using that method and insert untll the stream finishes..
i would like to get data from a csv file into a mongoDB collection using Meteor js and i would be grateful for any help
You can use papa-parse and read csv file using Node file system like this:
var fs = Npm.require('fs');
// Assume that the csv file is in yourApp/public/data folder
var data = fs.readFileSync(process.env.PWD + '/public/data/yourCSV.csv', 'utf8');
var usersData = Papa.parse(data, {header: true});
The userData will in the JSON format, you can store it in the MongoDb as you want.
csv-parse is used to parse csv files. Loading the MongoDB collection can be done via the upsert method of Meteor collection.
I'm working on a web app with node/express/mongodb, using mongoose. I created a folder in my project called data and inside I created another folder called db
data
db
when I start mongo, I use the --dbpath parameter and point it to my project data folder that I created. It connects fine and everything. I've been writing some data to the database, and the application loads it correctly, but I can't find the data when I open mongo shell.
first thing is, when I open shell the only database I get is test. Is this auto created if I don't specify a database name? how do I specify a database name? I have the following in my app.js file:
var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/abduldb');
My guess is that a database named abduldb should be created correct? I don't see this in mongo shell.
Also, when using the test database, I run show collections and it shows no results... Where exactly is my data being written? I only have 1 model right now and it looks like this:
var mongoose = require('mongoose');
var ItemSchema = new mongoose.Schema({
title: String,
body: String
});
mongoose.model('item', ItemSchema);
and I'm adding new items in my route file using:
router.post('/items', function (req, res, next) {
var item = new Item(req.body);
item.save(function(err, item){
if(err){ return next(err); }
res.json(item);
});
});
When you are in the mongo shell, try this command: "use abduldb".
The reason you don't see your collections is because you are automatically connected to the test db. If you want to change that, run this when you start mongo shell:
mongo abduldb
Alternatively, you can change the default db in your .mongorc.js file by adding/changing this line:
db = db.getSiblingDB("abduldb")
The .mongorc.js file can be found at /etc/mongorc.js, but is overridden if it exists in your home directory (i.e. /home/abduldb/.mongorc.js).