I am trying to determine what the difference is between the Transferred column and Size column. Does it have to do with the difference between compressed files and uncompressed?
I am not compressing my files on my server (Node.js Express server) so I don't know why there would be a difference in file size.
Your express application has gzip compression enabled as indicated by the Content-Encoding: gzip header, so the response body is compressed with gzip before sending over the network. Transferred size is when compressed, and size is decompressed in the browser. Express is doing this on the fly, so even though your file is not compressed on disk, it gets compressed before it is sent over the network.
Follow-up on your comments
You haven't posted any code, but it's probable that your express application is using the compression middleware (perhaps from the boilerplate you started with). If so, that will use mime-db to determine if the response content type is compressible. Looking up application/javascript in mime-db reveals it is marked as compressible:
mimeDb['application/javascript']
{ source: 'iana',
charset: 'UTF-8',
compressible: true,
extensions: [ 'js' ] }
Note that a .gz file extension is not involved anywhere here. There is no .gz file on disk, the compression is being done to a .js file in memory. Also note that just setting the Content-Encoding: gzip header without actually encoding the body as gzip is not something you want to do. It will cause encoding errors for the client.
Related
i am creating a api service using nodejs and express and to parse multipart request i am using multerjs. I have to receive a pdf file from rest request and it is coming as multipart/form-data and to verify file type i am checking the file.minetype
Now the issue is if someone changes the file extension like from mp4 -> pdf it also changes the mimetype to pdf and the file is accepted as pdf so how do i sniff the mimetype for content is their any way to do without any new packages. I really need to do it without using any packages as it'll take a really long to get a new package to approved
I'm using AWS S3 to host a static webpage, almost all assets are gzipped before being uploaded.
During the upload the "content-encoding" header is correctly set to "gzip" (and this also reflects when actually loading the file from AWS).
The thing is, the files can't be read and are still in gzip format although the correct headers are set...
The files are uploaded using npm s3-deploy, here's a screenshot of what the request looks like:
and the contents of the file in the browser:
If I upload the file manually and set the content-encoding header to "gzip" it works perfectly. Sadly I have a couple hundred files to upload for every deployment and can not do this manually all the time (I hope that's understandable ;) ).
Has anyone an idea of what's going on here? Anyone worked with s3-deploy and can help?
I use my own bash script for S3 deployments, you can try to do it:
webpath='path'
BUCKET='BUCKETNAME'
for file in $webpath/js/*.gz; do
aws s3 cp "$file" s3://"$BUCKET/js/" --content-encoding 'gzip' --region='eu-west-1'
done
My first deploy to AWS.
The files are all in place, and index.html loads.
There are two files in a subdir, one .js and once .css.
They both return 200 but fail to load. Chrome sais it's the 'parser'.
After trying a few things, I noted that this property is causing it: ContentEncoding: "gzip".
If I remove this property the files are found correctly.
Am I using this property incorrectly?
I am using the Node AWS SDK via this great project: https://github.com/MathieuLoutre/grunt-aws-s3
You can witness this behavior for yourself at http://tidepool.co.s3-website-us-west-1.amazonaws.com/
If you specify Content-Encoding: gzip then you need to make sure that the content is actually gzipped on S3.
From what I see in this CSS file:
http://tidepool.co.s3-website-us-west-1.amazonaws.com/08-26_6483218-dirty/all-min.css
the actual content is not gzipped, but the Content-Encoding: gzip header is present.
Also keep in mind that S3 is unable to compress your content on the fly based on the Accept-Encoding header in the request. You can either store it uncompressed and it will work for all browsers/clients or store it in a compressed format (gzip/deflate) and it will only work on some clients that can work with compressed content.
You could also take a look at the official AWS SDK for Node.js.
I'm using IIS to serve a static file. I've configured IIS (using the GUI) to enable both static and dynamic compression.
I'm using HttpClient to download it, as follows:
var client = new HttpClient();
var request = new HttpRequestMessage(HttpMethod.Get, requestUri);
request.Headers.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));
var response = client.SendAsync(request).Result;
var stream = new FileStream("foo", FileMode.Create);
response.Content.CopyToAsync(stream)
.ContinueWith(t => stream.Close())
.Wait();
I'm inspecting the traffic using Fiddler, and I can see that the first one or two responses are not compressed. I assume that IIS is compressing the file in the background and caching it somewhere. The file written to disk is about 14MB (expected).
Later requests are compressed. I can see a Content-Encoding: gzip header in the response, and the file that is downloaded is about 360KB. It's a gzip file, as identified by Cygwin's file command.
However, when I use gzip -d to decompress it, I end up with a 660KB file which is, itself a gzip-compressed file, also identified by file.
If I decompress that file, I get back the 14MB file that I was expecting.
So: why is my file being compressed twice?
In relation with my early question of how to add manifest cache in node.js, my question now is related with how to cache the HTML generated by node.js. As we didn't have a physical file like in php (index.php) we cannot cache such kind of files.
How we can cache a "non existing" page? Just adding in cache:
CACHE MANIFEST
CACHE:
# plain files to cache
/javascripts/client.js
/stylesheets/style.css
/stylesheets/style.styl
# generated files like /
/
/content
Any idea of how to solve this problem?
Thanks!
Solution:
Add router to return the cache.manifest file with the correct mime-type:
app.get("/offline.manifest", function(req, res){
res.header("Content-Type", "text/cache-manifest");
res.end("CACHE MANIFEST");
});
Found at stackoverflow
The cache manifest list URLs that should be cached. The client accessing those urls has no knowledge whether these are static html files on top of Apache or dynamic content generated by node.js or anything else.
You are basically instructing the client:
Read my list of urls
Go through each url
Download the response and store it someplace safe
Check back on my cache.manifest if it has changed and then proceed to step 1
So as long as your data generated by node.js is reachable via a URL there is no problem in defining it as a line in the cache manifest.
And if you are worried "how will I know which urls there are" you can always generate the cache.manifest file programmatically from node.js itself -- but remember to serve the correct content-type text/cache-manifest