I have an application that streams data to a file, can I use Node.js to read the file while it's being streamed to?
I tried using createReadStrem, but it only read one chunk and the stream ended
You could try watching for file changes with fs.watchFile(filename[, options], listener) or node-watch. In file change you could just read last line with read-last-lines.
Although I'm not sure how efficient it would be.
Related
I have a 6000 lines long data file which I'm going to load up in buffer parse it and write to another json file. What is a better way to accomplish this task ? Should I load the file in buffer, then parse it , and then write it to the file ? Or should I load chunk of file in buffer, process it, and write it to the while keeping tasks simultaneously ? Is this close to async function in javascript ? Is there examples in python for simple file loading and writing to a file ?
You can use aiofiles:
async with aiofiles.open('filename', mode='r') as f:
async for line in f:
print(line)
They have good usage documentation in their GitHub repo.
Is there any way to read a parquet file schema from Node.JS?
If yes, how?
I saw that there is a lib, parquetjs but as I saw it from the documentation it can only read and write the contents of the file.
After some investigation, I've found that the parquetjs-lite can do that. It does not read the whole file, just the footer and then it extracts the schema from it.
It works with a cursor and the way I saw it there is two s3.getobject calls, one for the size and one for the given data.
I'm trying to read the contents of some .gz log files using streams in node.
I've started simply with: fs.createReadStream('log.gz').pipe(zlib.createUnzip().
This works and I can pipe to process.stdout to verify. I'd like to pipe to this a new writeableStream, that I can have a data event to actually work with the contents. I guess I just don't fully understand how the streams work. I tried just creating a new writable stream, var writer = fs.createWriteStream() but this doesn't work because it requires a path.
Any ideas how I can go about doing this (without creating any other files to write to)?
var unzipStream = zlib.createUnzip()
unzipStream.on('data', myDataHandler).on('end', myEndHandler)
fs.createReadStream('log.gz').pipe(unzipStream)
That will get you the data and end events. Since it's log data you may also find split useful to get events line by line.
I'm using websockets in NodeJS to create a server and I want to read a CSV file or parse a CSV file and print data on a web page but that file is updated frequently so is there a way to keep track of the updating file and print the updated data like appending the print data whenever updated?
I am able to parse csv and print data but want to keep track of updation
Have a look at using fs to watch a file for changes.
When this event fires for your file, you can send a socket event with the latest version of the contents of the file to the client. You will need to re-fill your file stream when this event fires, and then send the results over the socket.
I have an application that will log data to a text file anytime the application is used. It may be used once a day or maybe once every second.
Is there anything wrong with me opening a handle to the text file and keeping it open as long as my application is running and never close it? This way I can just append data to the log without reopening the file handle. Are there consequences to this decision I should be aware of?
fs.open returns a file descriptor and if you don't close connection it may result in file descriptors leak. The file descriptor won't be closed, even if there's an error.
on the other hand, fs.readFile, fs.writeFile,fs.appendFile , fs.createReadStream and fs.createWriteStream don't return a file descriptor. They open the file , operate on it and then close the file.