trying to get a CSV dump of some data (~500Mb) in mongodb. Thought streams would be the way to go, to avoid building up an array in memory and then building the csv at once.
But, it seems the stream that mongoose creates and the one that csv expects are not the same thing.
var stream = Subscriber.find().stream()
stream.setEncoding = function() { }
csv().from.stream(stream).on('record', function(record, index) {
console.log(record)
console.log(index)
})
without the setEncoding() stub above, I get an error about that when csv calls setEncoding on the stream. With it, results in
TypeError: Object #<Object> has no method 'indexOf'
at [object Object].stringify (/home/project/node_modules/csv/lib/stringifier.js:98:35)
So, is this even the right approach? if so, what is the problem with the streams?
As zeMirco said: to get a CSV dump of a collection, I'd use the mongoexport tool that comes with MongoDB. Here's an example of exporting a collection called "users" in a database "mydatabase" to CSV format:
$ mongoexport --csv --host localhost:27017 --db mydatabase --collection users --fields name,email,age -o output.csv
And you'll get something that looks like this:
$ cat output.csv
name,email,age
renold,renold.ronaldson#gmail.com,21
jacob,xXxjacobxXx#hotmail.com,16
Something like this should work. Replace process.stdout with a filestream to write it to a file.
var csv = require('csv')
var through = require('through')
var Model = require('...')
_ = require('underscore')
var modelStream = Model.find().stream();
modelStream.pipe(through(write, end)).pipe(csv()).pipe(process.stdout);
function end(){ console.log('done'); }
function write(doc) {
this.queue(_.values(doc.toObject({getters:true, virtuals:false})));
}
If you want to download the csv from a webserver by accessing a URL and your using express you can do this:
var through = require('through');
var csv = require('csv')
var MyModel = require('./my_model');
app.get('/download_csv/', function(req, res) {
res.setHeader('Content-disposition', 'attachment; filename=attendances.csv');
res.contentType('csv');
res.write('property 1,property 2\n');
var modelStream = MyModel.find().stream();
modelStream.
pipe(through(write, end)).
pipe(csv.stringify()).
pipe(res);
function end() {
res.end();
console.log('done outputting file');
}
function write(doc) {
var myObject = doc.toObject({getters:true, virtuals:false});
this.queue([
myObject.property_1,
myObject.property_2
]);
}
});
NOTE: This is using the latest version of the csv module (v0.4) whereas the previous answers are using an older version of the module.
Related
I use an NPM library to parse markdown to HTML like this:
var Markdown = require('markdown-to-html').Markdown;
var md = new Markdown();
...
md.render('./test', opts, function(err) {
md.pipe(process.stdout)
});
This outputs the result to my terminal as intended.
However, I need the result inside the execution of my node program. I thought about writing the output stream to file and then reading it in at a later time but I can't figure out a way to write the output to a file instead.
I tried to play around var file = fs.createWriteStream('./test.html'); but the node.js streams rather give me headaches than results.
I've also looked into the library's repo and Markdown inherits from Readable via util like this:
var util = require('util');
var Readable = require('stream').Readable;
util.inherits(Markdown, Readable);
Any resources or advice would be highly appreciated. (I would also take another library for parsing the markdown, but this gave me the best results so far)
Actually creating a writable file-stream and piping the markdown to this stream should work just fine. Try it with:
const writeStream = fs.createWriteStream('./output.html');
md.render('./test', opts, function(err) {
md.pipe(writeStream)
});
// in case of errors you should handle them
writeStream.on('error', function (err) {
console.log(err);
});
I'm trying to generate a json file containing the filenames of all the files in a certain directory. I need this to create a cheatsheet for icons.
Currently I'm trying to run a script locally via terminal, to generate the json. That json will be the input for a react component that will display icons. That component works, the create json script doesn't.
Code for generating the json
const fs = require('fs');
const path = require('path');
/**
* Create JSON file
*/
const CreateJson = () => {
const files = [];
const dir = '../icons';
fs.readdirSync(dir).forEach(filename => {
const name = path.parse(filename);
const filepath = path.resolve(dir, filename);
const stat = fs.statSync(filepath);
const isFile = stat.isFile();
if (isFile) files.push({ name });
});
const data = JSON.stringify(files, null, 2);
fs.writeFileSync('../Icons.json', data);
};
module.exports = CreateJson;
I run it in terminal using
"create:json": "NODE_ENV=build node ./scripts/CreateJson.js"
I expect a json file to be created/overridden. But terminal returns:
$ NODE_ENV=build node ./scripts/CreateJson.js
✨ Done in 0.16s.
Any pointers?
You are creating a function CreateJson and exporting it, but you are actually never calling it.
You can get rid of the module.exports and replace it with CreateJson().
When you'll execute the file with node, it will see the function declaration, and a call to it, whereas with your current code there is no call.
I am new to node and file handling. below my code work perfectly
const data = 'Dummy data';
fs.appendFile('./../log/TempLog.txt', data, function(error){
});
But when same code place inside the function which is in other module file is not written i tried it in many ways please check my code below
const fs = require("fs");
function WriteData(data) {
fs.appendFile('./../log/TempLog.txt', data, function(error){
});
}
module.exports.WriteData = WriteData;
Also the Writedata function called form index module as shown below
const writeLog = require('./public/writeLog');
server.on("connection", function (socket) {
const decoder = new StringDecoder('utf8');
socket.on("data", function (d) {
writeLog.WriteData(decoder.write(d));
});
}
am i doing it wrong? Thanks in advance
after reviewing my own code i finally got the solution below :
The problem was with my path although the writedata.js module is in public folder but this will loaded into index.js as i imported into main module so i give path of text file referring to index file.
I was trying to use neo4j at backend. First I want to import csv to neo4j. (first tried to see how many lines csv file has)
But having problem, the code is following
var neo4j = require('neo4j-driver').v1;
var driver = neo4j.driver("bolt://localhost", neo4j.auth.basic("neo4j", "neo4j"));
function createGraphDataBase(csvfilepath)
{
var session = driver.session();
return session
.run( 'LOAD CSV FROM {csvfilepath} AS line RETURN count(*)',
{csvfilepath}
)
.then(result => {
session.close();
console.log(' %d lines in csv.file', result);
return result;
})
.catch(error => {
session.close();
console.log(error);
return error;
});
}
the "csvfilepath" is the path of csv file, it is as follows.
'/Users/.../Documents/Project/.../test/spots.csv';
is there something wrong with giving path like this?
I am calling that function on other module as
var api = require('./neo4j.js');
const csvFile = path.join(__dirname,csvFileName);
api.createGraphDataBase(csvFile);
I am having error as
Error: Connection was closed by server
....
I am new to these, please help!
The URL that you specify in a LOAD CSV clause must be a legal URL.
As stated in this guide:
Make sure to use the right URLs esp. file URLs.+ On OSX and Unix use
file:///path/to/data.csv, on Windows, please use
file:c:/path/to/data.csv
In your case, csvfilepath needs to specify the file:/// protocol (since you seem to be running on OSX) for a local file. Based on your example, the value should be something like this:
'file:///Users/.../Documents/Project/.../test/spots.csv'
I have an avro file.
I want to use nodejs to open and read its schema and iterate through its records.
How to do this? The avro libraries I see in nodejs appear to require you to pass in a schema instead of getting the schema out of the .avro file. Also, I want to be able to support arrays, which there does not seem to exist a node library that does (node-avro-io).
My avro/avroschema Contains:
A nested field {a:{suba: vala, subb: vala}}.
An array field {a:["A","B"]}. node-avro-io does not work.
Error I get with node-avro-io:
Avro Invalid Schema Error: Primitive type must be one of: ["null","boolean","int","long","float","double","bytes","string"]; got DependencyNode
In case you're still looking, you can do this with avsc. The code would look something like:
var avro = require('avsc');
// To stream the file's records (they will already be decoded):
avro.createFileDecoder('your/data.avro')
.on('data', function (record) { /* Do something with the record. */ });
// Or, if you just want the file's header (which includes the schema):
var header = avro.extractFileHeader('your/data.avro');
If you want to open a file, the following code found here : https://www.npmjs.com/package/node-avro-io will do the trick :
var DataFile = require("node-avro-io").DataFile;
var avro = DataFile.AvroFile();
var reader = avro.open('test.avro', { flags: 'r' });
reader.on('data', function(data) {
console.log(data);
});