I have a huge JSON file called data.json filled with objects with equal properties.
A sample of an object is this:
{
"directory": "directory_here",
"posted": false,
"date": null
}
In my script, I am using fs.readFile reading the data.json file if posted is false, if it is then run some functions and then change posted to true and also change date from null to the current date. So I need to change the properties inside the data.json file. How can I do that?
Here arr is Array of object
arr.forEach((obj)=>{
if(!obj.posted) obj.posted = true;
if(!obj.date) obj.date = new Date()
})
Except for a few pathological cases you cannot directly edit the contents of a JSON text file. Instead, you would need to either (a) read-modify-write the data, or (b) switch to a different storage format that does allow in-place editing.
If you (have to) stick with JSON, you have two main choices:
Read the file in chunks, parsing to JSON as you go. When you get a complete object, modify it as required then write it to a new file. After the whole file has been read, and all objects written, you can replace the original file with the newly-created one.
Read and parse the whole file into an array of objects. Make the required changes to all objects and then write the entire array (in JSON format) over the top of the original file. (Or, for extra safety, you could write to a new file and replace the original if there were no problems).
The second approach is probably easier to write, but will consume more memory.
A possible alternative (that I suspect won't be appropriate in your case) would be to switch to a storage format that does allow in-place editing. This could be a file of fixed record length records, or something like an SQLite database.
Related
I would like to convert a string to a variablename, so it can be read as a already restored variable.
So, I look through a file, and look at all the files. I use RESTORE to use the file in IDL, restore names this object as something slightly different. It names it as an object which we'll call map_1 (in the code it's called filerestore_name). This is related to the file name and I can recreate this variable name - however, its saved as a string.
Now, I pass this onto the make_cool_video procedure. However, althoughthis string now is exactly the same as the varialbe name, its still a string!.
Thus, as its a string, the procedure can't work.
filenames=FILE_SEARCH('rxrt*')
filenames_withoutextension = STREGEX(filenames,'rxrt_[0-9]+[a-zA-Z_]+',/EXTRACT,/FOLD_CASE)
restore, '/home/tomi/Documents/actualwork/'+filenames_withoutextension(18)+'.idl_sav',
filerestore_name = STRJOIN(STRSPLIT(filenameswithout(18),'_[0-9]+',/EXTRACT,/REGEX),'')
PRINT, filerestorename
make_cool_video, EXECUTE(filerestore_name),filename=filerestorenames, outdir='/path/to.file/'
retall
What I tried: using the RESTORE function and the associated RESTORED_OBJECTS to store pointers in an array, and then referring to the array. But I couldn't get the restore function to form an array.
Using EXECUTE(filerestore_name) however, this doesn't convert it as I was expecting.
I would recommend using SCOPE_VARFETCH() instead (it isn't as limited as EXECUTE() and is probably more efficient). You can do something like:
make_cool_video, (SCOPE_VARFETCH(filerestore_name)), filename=filerestorenames, outdir='/path/to.file/'
I wrote this, then immediately thought of the answer.
So,
Convert everything to a string:
string1 = "makecooljes, "+ filerestore_name, outdir='file/to/path/'"
result= EXECUTE(string1)
I realize this is a pretty weird thing to want to do - it's mainly just to simplify my unittests.
I have a class whose __init__ takes a filename as an argument, which it open()s and reads a bunch of data from. It'd be really awesome if I could somehow "trick" that open() into reading from a string object instead without actually writing a temporary file to disk.
Is this possible, and if so, how would I go about it?
You can monkey-patch the module that contains the class before running the test, and restore it after the test:
def my_fake_open(fake_filename):
return object_with_read_and_close_that_will_feed_correct_data()
def test_something(self):
module_containing_test.open = my_fake_open
...run test...
del module_containing_test.open
Check out the mock library if you don't want to write your own mock objects.
Reading a bit too fast, I thought the answer would be in io.stringIO which allows you to create a file-like object with the content of a string.
But what you want, is an object that, passed to the standard open function would actually yield a file-like object from the contents of the your string.
The thing is that open takes a string (or a file descriptor) but anything else will pause a problem.
So I don't believe this way is practical.
But actually, it's not difficult to create a file using tempfile:
with tempfile.NamedTemporaryFile() as tmp_file:
tmp_file.write(your_string)
yourmodule.YourClass(tmp_file.name)
(If you're on Windows you might want to play with delete=False and close before sending the name for it to be opened.)
Another approach might be to just change the API: if all the init does with the name is to open a file, why not directly pass a file-like object.
I want to copy some files using Node.js. Basically, this is quite easy, but I have two special requirements I need to fulfill:
I need to parse the file's content and replace some placeholders by actual values.
The file name may include a placeholder as well, and I need to replace this as well with an actual value.
So, while this is not a complex task basically, I guess there are various ways how you could solve this. E.g., it would be nice if I could use a template engine to do the replacements, but on the other hand then I need to have the complete file as a string. I'd prefer a stream-based approach, but then - how should I do the replacing?
You see, lots of questions, and I am not able to decide which way to go.
Any hints, ideas, best practices, ...?
Or - is there a module yet that does this task?
You can write your own solution without reading the entire file. fs.readFile() should only be used when you are 100% sure that the files are no longer than a buffer chunk (typically 8KB or 16KB).
The simplest solution is to create a readable stream, attach a data event listener and iterate the buffer reading character by character. If you have a placeholder like this: ${label}, then check if you find ${, then set a flag to true. Begin storing the label name. If you find } and flag is true then you've finished. Set flag to false and the temporal label string to "".
You don't need any template engine or extra module.
If the whole file can be safely loaded into memory (isn't crazy big), then the library fs-jetpack might be very good tool for this use case.
const jetpack = require("fs-jetpack");
const src = jetpack.cwd("path/to/source/folder");
const dst = jetpack.cwd("path/to/destination");
src.find({ matching: "*" }).forEach((path) => {
const content = src.read(path);
const transformedContent = transformTheFileHoweverYouWant(content);
const transformedPath = transformThePath(path);
dst.write(transformedPath, transformedContent);
});
In the example code is synchronous, but you can easily make async equivalent.
Suppose i have two files a_b_c_d.txt and e_f_g_h.png S,At runtime i.e., by using command prompt i have to create b folder inside that c folder inside that d folder inside that a.txt and same also for another file f->g->h->e.png and i have some text in a and image in epng . .So,how can I get values from those existing file into created files. .
You can find all the file system operations inside the fs module. http://nodejs.org/api/fs.html
But like tapan says if you need to do complex synchronous execution that manipulates the file system something like Bash will be a lot better suited for that.
So if I'm understanding you correctly you want to take a file named "a_b_c_d.txt" in some folder, and move that into a nested folder as:
./a_b_c_d.txt -> ./b/c/d/a.txt
The general solution would be:
Grab the file name using process.argv if it varies.
For example, if you supply the file as an argument to node, e.g.
node move.js "a_b_c_d.txt", the argument, "a_b_c_d.txt", will be in the argv array.
Process the file name using a combination of string and array methods.
Nodes current directory is stored in the __dirname global variable
if you need it.
You can split the extension from the rest of the path
using string's split(...) method.
For the above argument, split('.') will result in the array ['a_b_c_d', 'txt']
You can then split 'a_b_c_d' using '_',
and use various array operations to pull the file name 'a'
out of the array, so that you're left with the path ['b', 'c', 'd']
and the file name and extension sitting in their own variables somewhere.
Use fs.mkdirSync(...) on the path array to make each nested folder,
starting with b (e.g. using array's forEach(...) method).
You could also use the async fs.mkdir(...) and supply callbacks,
but the sync version is easier in this case.
Finally use fs.renameSync(...) to move ./a_b_c_d.txt to ./b/c/d/a.txt.
As you can see, python or bash (as tapan suggested) would probably be simpler for this use case, but if for some reason you have to use node, the above advice will hopefully be enough to get you started.
I can easily skip the header of a data file using getline, but then when I parse through the data file and get to the footer of the file, I end up stuck in a loop because the program is trying to parse columns of data that no longer exist. Is there an easy way to stop reading when there is no longer data in the line? It looks like there is a blank line followed by some footer information, but I cannot guarantee that all of my data files will look like that (i.e. I need something pretty generic).
Looking at your existing code (edit your question and put it there, not in a comment), I see you have nested loops. But what you really want is one loop with two reasons to exit.
while ((q < 16) && (liness >> temp)) { ... }
Read the line into a string, parse if only if you see \n at the end.