I am writing a nodejs app which works with fonts. One action it performs is that it downloads a .ttf font from the web, converts it to a base64 string, deletes the .ttf and uses that string in other stuff. I need the .ttf file stored somewhere, so I convert it. This process takes like 1-2 seconds. I know heroku has an ephemeral file system but I need to store stuff for such a short time. Is there any way I can store my files? Using fs.writeFile currently returns this error:
Error: EROFS: read-only file system, open '/app\test.txt']
I had idea how about you make an action, That would get font, convert it and store it on a global variable before used by another task.
When you want to use it again, make sure you check that global variable already filled or not with that font buffer.
Reference
Singleton
I didn't know that you could store stuff in /tmp directory. It is working for the moment but according to the dyno/ephemeral system, it gets cleaned frequently so I don't know if it may cause other problems in the long run.
Related
I'm currently making an electron app that needs to play some 40Mbyte audio file from the file system, maybe it's wrong to do this but I found that the only way to play from anywhere in the file system is to convert the file to a dataurl in the background script and then transfer it using icp, after that I simply do
this.sound = new Audio(dataurl);
this.sound.preload = "metadata"
this.sound.play()
(part of a VueJS component hence the this)
I did a profling inside electron and this is what came out:
Note that actually transferring the 40Mbytes audio file doesn't take that long (around 80ms) what is extremely annoying is the "Second Task" which is probably buffering (I have no idea) which last around 950ms, this is way too long and ideally would need it under <220ms
I've already tried changing the preload option to all available options and while I'm using the native html5 audio right now I've also tried howlerjs with similar results (seemed a bit faster tho).
I would guess that loading the file directly might be faster but even after disabling security measures put by electron to block the file:/// it isn't recognized as a valid URI by XHR
Is there a faster way to load the dataurl since all the data is there it just needs to be converted to a buffer or something like that ?
Note: I can not "pre-buffer" every file in advance since there is about 200 of them it just wouldn't make sense in my opinion.
Update:
I found this post Electron - throws Not allowed to load local resource when using showOpenDialog
don't know how I missed it, so I followed step 1 and I now can load files inside electron with the custom protocol, however, nor Audio nor howlerjs is faster, it's actually slower at around 6secs from click to first sound, is it that it needs to buffer the whole file before playing ?
Update 2:
It appears that the 6sec loading time is only effective on the first instance of audio that is created. I do not know why tho. After that the use of two instances (one playing and one pre-buffering) work just fine, however even loading a file that isn't loaded is instantaneous. Seems weird that it only is the firs one.
OS - Debian Stable
I downloaded fonts from a website (that seemed legitimate to me) and transferred the contents to /usr/share/fonts/directory. There's a .uuid file being generated for every directory with a string like this as its sole content:
f25e9432-c6f1-4bbe-a33c-89289a8d17f1
This file regenerates right after I delete it. Is this a malicious program? Is this indexing by the OS itself or is it something like fc-cache running in the background? What could be the cause of this?
This has nothing to do with the fact that you've downloaded your own fonts. This is simply just fontconfigdoing it's job. It very well could just be the cached data created. Only rather, the binary data is being converted into a uuid string/unique ID.
So, I would say no. I do not believe this to be the cause of anything malicious. Nor are any of these occurrences a result from you downloading fonts from your web browser.
I would like to wrap a data file (~1MB) to golang app and then use that data in os.exec. The app runs on Linux.
How to define the data in the app, as a string or []byte, variable, or Const?
Should be defined in a global scope, or wrapped in a func?
How to pass the data from the app memory to the executed process ?
For building the data file(s) in to your program, you have a number of choices. You are correct in that you could manually copy/paste the data file(s) in to the program as types string, []byte as variables, but there are other packages/applications for go that handle this for you already that can also minimize your app's memory footprint. One example that comes to mind is go-bindata (link to repo) which does just this.
As for using the file in os/exec, I'm assuming you're trying to either pass the entire file to the program using either a file path or a raw string. For file paths, you would have to write the entire file to disk first. If you're trying to pass the entire file contents as a raw string, you can still use go-bindata or a string of the data file as arg in os/exec.Command.
P.S. - go-bindata has not seen updates in a while, so I would encourage you to seek more active forks or alternatives if you're having trouble using it.
When I run the command in the terminal back to back, it doesn't sync the second time. Which is great! It shouldn't. But, if I run my build process and run aws s3 sync programmatically, back to back, it syncs all the files both times, as if my build process is changing something differently the second time.
Can't figure out what might be happening. Any ideas?
My build process is basically pug source/ --out static-site/ and stylus -c styles/ --out static-site/styles/
According to this - http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html
S3 sync compares the size of the file and the last modified timestamp to see if a file needs to be synced.
In your case, I'd suspect the build system is resulting in a newer timestamp even though the file size hasn't changed?
AWS CLI sync:
A local file will require uploading if the size of the local file is
different than the size of the s3 object, the last modified time of
the local file is newer than the last modified time of the s3 object,
or the local file does not exist under the specified bucket and
prefix.
--size-only (boolean) Makes the size of each key the only criteria used to decide whether to sync from source to destination.
You want the --size-only option which looks only at the file size not the last modified date. This is perfect for an asset build system that will change the last modified date frequently but not the actual contents of the files (I'm running into this with webpack builds where things like fonts kept syncing even though the file contents were identical). If you don't use a build method that incorporates the hash of the contents into the filename it might be possible to run into problems (if build emits same sized file but with different contents) so watch out for that.
I did manually test adding a new file that wasn't on the remote bucket and it is indeed added to the remote bucket with --size-only.
This article is a bit dated but i'll contribute nonetheless for folks arriving here via google.
I agree with checked answer. To add additional context, AWS S3 functionality is different than standard linux s3 in a number of ways. In Linux, an md5hash can be computed to determine if a file has changed. S3 does not do this, so it can only determine based on size and/or timestamp. What's worse, AWS does not preserve timestamp when transferring either way, so timestamp is ignored when syncing to local and only used when syncing to s3.
I have a rather peculiar file format to work with:
Every line begins with the checksum of its content, followed by a new-line-character.
It looks like this:
[CHECKSUM OF LINE_1][LINE_1]\n
[CHECKSUM OF LINE_2][LINE_2]\n
[CHECKSUM OF LINE_3][LINE_3]\n
...
My goal: To allow any application to work with these files like they would work with any other text file - unaware of the additional checksums at the beginning of each line.
Since I work on a linux machine with debian wheezy (kernel 3.18.26) I want to use the LD_PRELOAD-mechanism to override the relevant file functions.
I have seen something like this with zlibc on https://zlibc.linux.lu/index.html - with an explanation of how it works ( https://zlibc.linux.lu/zlibc.html#SEC8 ).
But I dont get it. They only replace the file-opening functions. No read. No write. no fseek. Nothing. So how does it work?
Or - which functions would I have to intercept to handle every read or write operation on this file and handle them accordingly?
I didn't exactly check how it works but the reason seems to be quite simple.
Possible implementation:
zlibc open:
uncompress file you wanted to open to some temporary file
open this temporary file instead of yours
zlibc close:
Compress temporary file
Override original file
In this case you don't need to override read/write/etc because you can use original ones.
In your case you have two possible solutions:
open, that make a copy of your file with striped checksums. close that calculates checksums and override original file
read and write that are able to skip/calculate checksums.
Ad 2.
From What is the difference between read() and fread()?:
fread() is part of the C library, and provides buffered reads. It is
usually implemented by calling read() in order to fill its buffer
In this case I believe that overriding open and close will be less error prone because you can safely reuse original read, write, fread, fseek etc.