I'm doing image modifications on a .png and then writing it on my server with
fs.writeFile
Problem is if some clients try to http get the .png from my server while i'm writing it, it will fail.
How can i prevent that ?
Save the modified files under a diferent name then rename it to the original name. The rename operation is an atomic one so it will be done instantly.
Steps:
Make the modification of original.png and write them to original.png.mod
Rename original.png.mod in original.png
Say you are currently writing to example.png.
Write to a different file, and then move it to replace example.png.
That removes the lag between opening the file for writing and finishing writing to it.
Related
I am writing a nodejs app which works with fonts. One action it performs is that it downloads a .ttf font from the web, converts it to a base64 string, deletes the .ttf and uses that string in other stuff. I need the .ttf file stored somewhere, so I convert it. This process takes like 1-2 seconds. I know heroku has an ephemeral file system but I need to store stuff for such a short time. Is there any way I can store my files? Using fs.writeFile currently returns this error:
Error: EROFS: read-only file system, open '/app\test.txt']
I had idea how about you make an action, That would get font, convert it and store it on a global variable before used by another task.
When you want to use it again, make sure you check that global variable already filled or not with that font buffer.
Reference
Singleton
I didn't know that you could store stuff in /tmp directory. It is working for the moment but according to the dyno/ephemeral system, it gets cleaned frequently so I don't know if it may cause other problems in the long run.
I have a file and I am reading it through chokidar. I am also writing to the same file in some another function using fs.writeFileSync.
I want to emit the watcher.on('change') function only when the file is changed manually (i.e. not through WriteFileSync). Is there a way of determining that ?
No, there isn't, not to my knowledge.
You can only know the file was changed, not what process/... changed it.
I have a rather peculiar file format to work with:
Every line begins with the checksum of its content, followed by a new-line-character.
It looks like this:
[CHECKSUM OF LINE_1][LINE_1]\n
[CHECKSUM OF LINE_2][LINE_2]\n
[CHECKSUM OF LINE_3][LINE_3]\n
...
My goal: To allow any application to work with these files like they would work with any other text file - unaware of the additional checksums at the beginning of each line.
Since I work on a linux machine with debian wheezy (kernel 3.18.26) I want to use the LD_PRELOAD-mechanism to override the relevant file functions.
I have seen something like this with zlibc on https://zlibc.linux.lu/index.html - with an explanation of how it works ( https://zlibc.linux.lu/zlibc.html#SEC8 ).
But I dont get it. They only replace the file-opening functions. No read. No write. no fseek. Nothing. So how does it work?
Or - which functions would I have to intercept to handle every read or write operation on this file and handle them accordingly?
I didn't exactly check how it works but the reason seems to be quite simple.
Possible implementation:
zlibc open:
uncompress file you wanted to open to some temporary file
open this temporary file instead of yours
zlibc close:
Compress temporary file
Override original file
In this case you don't need to override read/write/etc because you can use original ones.
In your case you have two possible solutions:
open, that make a copy of your file with striped checksums. close that calculates checksums and override original file
read and write that are able to skip/calculate checksums.
Ad 2.
From What is the difference between read() and fread()?:
fread() is part of the C library, and provides buffered reads. It is
usually implemented by calling read() in order to fill its buffer
In this case I believe that overriding open and close will be less error prone because you can safely reuse original read, write, fread, fseek etc.
I implemented big-text file reading using createReadStream. In my implementation i wanted to test for error handling.
So i started process and renamed file, still entire file was read and all data were printed on console. In second try i did same but this time i deleted the file, and results were same, entire file was read and printed on console.
In short, I am reading a file using createReadStream, and after reading started even if i rename/delete the file, it still reads till the end.
Here is a sample implementation,
const fs = require('fs');
const readableStream = fs.createReadStream('mySampleTextFile.txt');
readableStream.on('data', (chunk) => console.log(chunk));
readableStream.on('end', () => console.log('Read till end !'));
mySampleTextFile.txt contains 600k rows with just number on each line, and i verified the last number logged to be exactly same as in the original file.
I am deleting the file before execution ends, i have two terminals open and in one i start execution and in second i fire delete/rename command.
I tried also deleting permanently means directly delete, not in trash.
Am i missing some fundamental of Stream or something? Please help me on this, it's not breaking anything but why it is not breaking ?
At least on Unix-type OS'es, when you remove a file while it's open, the process that opened the file can still read from it (or write to it).
Removing such a file merely removes its directory entry so you can't see it anymore when running ls or a file browser, but the disk space it takes up won't be marked "available" until there are no more processes that are referencing the file (i.e. have it open).
What you could try is truncating the file instead of removing it:
cp /dev/null mySampleTextFile.txt
I wonder if you can configure logstash in the following way:
Background Info:
Every day I get a xml file pushed to my server, which should be parsed.
To indicate a complete file transfer afterwards I get an empty .ctl (custom file) transfered to the same folder.
The files both have the following name schema 'feedback_{year}{yearday}_UTC{hoursminutesseconds}_51.{extention}' (e.g. feedback_16002_UTC235953_51.xml). So they have the same file name but one is with .xml and the other is a .ctl file.
Question:
Is there a way to configure logstash to wait parsing the xml file until the according .ctl file is present?
EDIT:
Is there maybe a way to archiev that with filebeat?
EDIT2:
It would also be enough to be able to configure logstash in a way that it will wait x minutes before starting to process a new file, if that is easier.
Thanks for any help in advance
Your problem is that you don't want to start the parser before the file transfer hasn't been completed. So, why don't push the data to a file (file-complete.xml) when you find your flag file (empty.ctl)?
Here is the possible logic for a script and runs using crontab:
if empty.ctl exists:
Clear file-complete.xml
Add the content of file.xml to file-complete.xml.
Remove empty.ctl
This way, you'd need to parse the data from file-complete.xml. I think is simpler to debug and configure.
Hope it helps,