I have a rather large JSON file that stores user information, and when my server starts it loads the entire file into memory. Obviously, this is not ideal. I have looked into using read/write streams, but I can't seem to understand quite how they'd work.
The data in the JSON file is formatted as such:
"accountName": {
"favoriteColor": "blue"
}
The process currently goes in this order:
Server starts, and data.json is loaded into a variable (dataVar),
User johndoe logs in, and their data is used to make an object.
The user changes their object's data, and dataVar is updated.
Server autosaves to the data.json file with the new contents.
I want to continue being able to access user data as needed, without loading everything into memory. I assume there are stream equivalents of things like dataVar.johndoe, but I can't seem to find that information.
Related
I am using Python 3.7 along with a library for AES CBC on Windows 10 to encrypt files and it works perfectly. Except, after decrypting them, they lose their metadata like the date they were created. Because I want the user to feel like they never 'deleted' or 'lost' the original file, I need to preserve that data.
This is what I'm doing to read the data:
f = open(file_name, "rb")
data = f.read()
f.close()
After I encrypt the data, I write the encrypted bytes into a new file. When I decrypt this new file, I would like the metadata to be preserved so that the file (like an image) is exactly like it was before encryption. (P.S. I don't know if this will help but overwriting the new data on the original file might help but I want to try and avoid this if possible)
How do I include metadata of the file in a variable WITH the data that I am encrypting, such that when I decrypt, I get the exact same file with the same date created etc.?
EDIT:
I found a way to get the file creation time but I STILL NEED to get all the metadata as the file can be in any format: for example an image or a video, or a doc file's author. I also want to store this in the decrypted file which I don't know how to.
os.path.getctime(file_name)
I know it has been asked a few times already, but - not in this context I think and other questions have been asked few years ago already, so I'm hoping maybe something changed.
So my issue is - I am uploading files to the Google Drive using Google Drive File Stream. However, while the uploading goes smoothly, I have a problem with files creation date - it is always changed to the timestamp of the time the file got uploaded, not the actual, local file creation date. It is a serious problem, as I am going to use this to back-up huge amounts of data and preserve all the meta-data I can and the creation date is crucial. Is there a way to either upload it with the creation date intact, or to change it after the upload? From what I've seen this seems not to be possible, but I have to try and make it work. Any help and insight will be appreciated. I'm using the Drive File Stream with Python.
EDIT: I didn't make it clear enough - the issue here is that I don't want to use Google Drive API at all, but rather deal with this using only Google Drive File Stream interface if it's possible.
create
If you check the documentation for files.create You will find that acceptable metadata for file creation does include a createdTime
You should then just add this to the metadata you use when uploading the file. As you did not post your code I have grabbed the standard example from the documentation and added the created time as follows.
file_metadata = {'name': 'photo.jpg', 'createdTime': 'THETIME'}
media = MediaFileUpload('files/photo.jpg', mimetype='image/jpeg')
file = drive_service.files().create(body=file_metadata,
media_body=media,
fields='id').execute()
print 'File ID: %s' % file.get('id')
Update
In the event that you want to update the ones you have already created you could use the following method.
If you check the documentation for file create you will find that the response is just a File resource
If you check file resource you will see that CreatedTime is write able.
You should run a file.update and reset the createdTime to the proper time.
I am new to node js
I want to print excel data without using any library as using libraries takes time and we have to deal with large memory files.
I tried reading a .xlsx file using fs.createReadStream() and logged the data to console, it prints some different characters but the actual data.
nodejs code
const stream=fs.createReadStream('example.xlsx');
stream.on('data',function(data){
console.log(Buffer.from(data, 'base64').toString('utf8'))
})
Can I know how to get the actual data or any library that can read large excel files?
let dataCer = '0�\u0007\u00060�\u0006��\u0003\u0002\u0001\u0002\u0002\u0010Q��\u0000����K��Z�Q��0\n\u0006\b*�\u0003\u0007\u0001\u0001\u0003\u00020�\u0001l1\u001e0\u001c\u0006\.............'
fs.writeFile('111.cer', dataCer);
let dataPdf = '%PDF-1.4\r\n1 0 obj\r\n<< \r\n/Length 9947\r\n/Filter /FlateDecode\r\n>>\r\nstream\r\nX��]�n#9p}���\u000f���\u0005\b\u0002X��<\'X \u001f�\u001b\u0010 \u0001���H�,6�R�Z�\u0014�N`�\n�T�t�ڼT\u0015���?ԋz��_�{IN_Bz�����O.............'
fs.writeFile('111.pdf', dataPdf);
The data dataCer and dataPdf I get from the application using the GET requests. I can only get this data in this encoding.
And now I need to save them as files.
Also, I will need to then save any data to the file in the same way (zip, rar, png, jpeg, ...).
When i use fs.writeFile, I get files that do not open.
fs.writeFile, can not keep the original state data, ignoring the encoding does not give me the desired result.
Please tell me how to get around this error?
Or which library can save data to any file in node.js, while ignoring the encoding?
The module requests provides a high level HTTP API. Using requests I'd like to send data via HTTP using a POST request. The documentation is very short about this, stating that a "file like object" should be provided without stating clearly what exactly requests would expect from that object. I've some binary data, but unfortunately this is generated data and I have not a file like object. How could I possibly implement a "file like object" myself that would conform to the expectations of requests? The documentation is quite poor in that regard and I wasn't able to clarify this by looking into the source code of requests myself. Has anyone done this before using the requests API?
File-like object is a standard Python term for an object that behaves like a file. This means that if you have a file, you have a file-like object and simply need to pass the file path to Requests. If you have a more complex situation you will need to give us a full description of the form of your data so we can help you more explicitly.
EDIT: To address your comment, here is the code to send a binary file to a host using Requests.
url = 'http://SomeSite/post'
files = {'files': ('mydata', open('mydata', mode='rb'), 'application/octet-stream')}
r = requests.post(url, files=files)
Opening the file with the Python open command creates the file-like object.
EDIT2: Whenever to open a file on disk you create a file-like object in the process of opening the objects. However, Python supports other object types that act like files. Some examples include the standard stdin, stdout and stderr. In addition, pipes can be access using os.pipe and via subprocess.Pipe. These objects behave like files, i.e. they can be accessed with a subset of the file API and their API's behave in the same way as the object that accesses a real file.
This is why they are called file-like because they use the same API's and act in the same way. You open, close, can read or write a pipe in the same way as you do a file.