Unable to get same result from FileSaver/Blob versus Node's fs - node.js

I'm trying to generate a midi file (with jsmidgen) and I'm able to output it with node this way :
fs.writeFileSync('test.mid', file.toBytes(), 'binary');
That midi file is working perfectly. Then I'm trying to generate it from a browser and I download it with FileSaver.js this way :
let blob = new Blob([file.toBytes()], {type: "audio/midi"});
saveAs(blob, "test.mid");
That midi file is corrupted. I tried various blob content types without success and I've also validated that the file.toBytes() output is the same in both circumstances. I compared both hex outputs it looks like an encoding issue but I'm not able to find how to fix this.
Hex code from the good file (save in Node with fs)
4d 54 68 64 00 00 00 06
00 00 00 01 00 80 4d 54
72 6b 00 00 00 5e 00 90
3c 5a 40 80 3c 5a 00 90
3e 5a 40 80 3e 5a 00 90
40 5a 40 80 40 5a 00 90
41 5a 40 80 41 5a 00 90
43 5a 40 80 43 5a 00 90
45 5a 40 80 45 5a 00 90
47 5a 40 80 47 5a 00 90
48 5a 40 80 48 5a 81 00
90 3c 5a 00 90 40 5a 00
90 43 5a 81 00 80 3c 5a
00 80 40 5a 00 80 43 5a
00 ff 2f 00
Hex code from the bad file (FileSaver/Blob) :
4d 54 68 64 00 00 00 06
00 00 00 01 00 c2 80 4d
54 72 6b 00 00 00 44 00
c2 90 3c 5a 40 c2 80 3c
5a 00 c2 90 3e 5a 40 c2
80 3e 5a 00 c2 90 40 5a
40 c2 80 40 5a 00 c2 90
41 5a 40 c2 80 41 5a 00
c2 90 43 5a 40 c2 80 43
5a 00 c2 90 45 5a 40 c2
80 45 5a 00 c2 90 47 5a
40 c2 80 47 5a 00 c2 90
48 5a 40 c2 80 48 5a 00
c3 bf 2f 00
Is there something wrong with the way I'm using Blob or could I try another approach?
The code I used to generate the midi file is the first example on the jsmidgen page, the one playing a C major scale.

Do you need to use Blob specifically? If not, then you can use btoa.
I use jsmidgen as a primary dependency for my module(which acts as a wrapper and does some other things) and I faced a similar issue hence came up with this:
const bytes = file.toBytes();
const b64 = btoa(bytes);
const uri = 'data:audio/midi;base64,' + b64;
const link=document.createElement('a');
link.href=uri;
link.download = 'music.mid';
link.click(); // this will start a download of the MIDI byte string
I ve documented this in the context of my module with a working JS Bin in there too: https://scribbletune.com/documentation/core/midi

Related

Manually Parse Form Data Node.js Express

How can I parse form-data manually at node js?
I get chunks of the large file at the server side. And can't merge it, because it will borrow RAM equals to the (file size + file info size).
As far as I understand, I can't use the libraries, such as multer/connect-busboy/connect-multiparty etc.
In simple terms, now I get following data at server side and can't figure out how to receive, example, a file name, and how to determine, which byte is responsible for the beginning of the file.
[1] Get request...
[1] data: <Buffer 2d 2d 2d 2d 2d 2d 57 65 62 4b 69 74 46 6f 72 6d 42 6f 75 6e 64 61 72 79 62 66 30 31 63 62 36 38 71 4d 41 4d 36 6d 51 31 0d 0a 43 6f 6e 74 65 6e 74 2d ... 65486 more bytes>
[1] data: <Buffer 02 10 00 00 01 ce 00 00 01 ef 00 00 14 e4 00 00 01 de 00 00 01 b2 00 00 01 d8 00 05 f0 1a 00 00 06 70 00 00 00 fa 00 00 00 cf 00 00 0a 3b 00 00 01 4a ... 65486 more bytes>
[1] data: <Buffer 31 52 2e 9a 5b df 11 19 5f c2 5d 38 ed db 23 00 06 05 bd 8f 71 e7 6b 03 9d 3a 88 e3 24 10 00 0f a1 10 43 82 ff 67 bd 5d eb 8e 59 fe a9 d3 e6 45 f2 9e ... 65486 more bytes>
[1] data: <Buffer 2b 6c cf 18 40 13 d8 fd 09 fa 56 25 e6 6d 6a 57 94 8c a4 9c d1 7b 99 bb 20 c0 21 e7 4a 1f 72 5a a5 3d 8e 84 5d 97 eb 1b 0f db b1 a9 81 f9 db ef 06 36 ... 65486 more bytes>
[1] data: <Buffer bb 29 4b 0d 06 1d 69 e1 c7 47 a7 99 c6 e4 c3 48 2e 85 6a b3 57 01 68 09 00 aa 6d b4 7b c7 07 2f 73 c0 c4 6b c4 48 9b 8f 81 64 8e 25 c7 a3 de c3 4f 2a ... 65486 more bytes>
[1] data: <Buffer 21 2b 1c dd 57 02 23 f5 43 a1 70 72 b4 8d 8d a8 4a 4b c1 e1 21 75 84 ed 07 00 34 de 5c 8d b0 0a 6f 99 60 28 34 b1 c5 bd a2 3c 3e d5 01 0f 62 57 a8 e8 ... 65486 more bytes>
[1] data: <Buffer 2d 96 e3 47 75 ce 25 63 77 26 15 96 b6 43 b9 d2 59 7d 12 c9 64 b2 11 f2 39 29 3e bf 83 1a f3 15 7b 14 ee f3 33 52 5c 39 34 75 06 41 2f e7 11 e8 26 4e ... 65486 more bytes>
[1] data: <Buffer 4a 9d 20 c7 a8 55 35 22 26 ff c7 b4 11 d9 6c 4d ce e1 a0 b2 b3 01 37 da 6a ad ae 98 fd 9b 8e 9c 8b 4a 27 8e e3 10 4a dc 80 f2 50 df 88 d9 c8 f9 ef 48 ... 65486 more bytes>
[1] data: <Buffer 52 ee b3 96 1f 7c af df fa 3f 9c 9a f7 01 20 7d 3f ea 4e 7e aa 4e d9 57 31 cb b5 f3 09 49 c7 ee 92 83 2e cb 58 d2 ea 1b b0 35 2a 3c 6c 27 75 0c d0 39 ... 65486 more bytes>
...more
...more
...and more
After receiving all the above data, I need to get file name and send each chunk of the file(not other info, such as name/mime type) to another client. (It's file sharing web resource)
Another solutions?
At the same time, I could to send the file separated manually by the chunks of type ArrayBuffer. Each chunk would be encoded additional information, like a "file name", "mime type". But in this case, I should to separate file at client side and get again the above problem the lack of memory.
A working solution, but a bit insecure. Send the file name and mime type separately from the main file data.
The part of the code responsible for sending data:
app.get(`/${senderId}/get`, (request, response) => {
let countChunks = 0;
// waiting for the sender's request now...
app.post(`/${senderId}/send`, (requestSender, responseSender) => {
// start sharing...
let countChunks = 0;
requestSender.on("data", (chunk) => {
if (countChunks === 0) {
// need get the name from the chunks
const fileName = "test_name";
response.setHeader("Content-Disposition", `attachment; filename="${fileName}"`);
}
countChunks += 1;
// need to get the first byte of the file data and start sending starting with it
response.write(chunk);
})
requestSender.on("end", () => {
response.end();
})
})
})

How ImageMagic determines the ColorSpace of a PNG?

Suppose I create a simple PNG with:
convert -size 1x1 canvas:red red.png
Here is a similar image (bigger size) for reference:
Then run the command identify on it. It tells me the ColorSpace of the image is sRGB but there seems to be NO indication of this inside the file. In fact running
$ hexdump -C red.png
00000000 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 |.PNG........IHDR|
00000010 00 00 00 01 00 00 00 01 01 03 00 00 00 25 db 56 |.............%.V|
00000020 ca 00 00 00 04 67 41 4d 41 00 00 b1 8f 0b fc 61 |.....gAMA......a|
00000030 05 00 00 00 20 63 48 52 4d 00 00 7a 26 00 00 80 |.... cHRM..z&...|
00000040 84 00 00 fa 00 00 00 80 e8 00 00 75 30 00 00 ea |...........u0...|
00000050 60 00 00 3a 98 00 00 17 70 9c ba 51 3c 00 00 00 |`..:....p..Q<...|
00000060 06 50 4c 54 45 ff 00 00 ff ff ff 41 1d 34 11 00 |.PLTE......A.4..|
00000070 00 00 01 62 4b 47 44 01 ff 02 2d de 00 00 00 07 |...bKGD...-.....|
00000080 74 49 4d 45 07 e5 01 0d 17 04 37 80 ef 04 02 00 |tIME......7.....|
00000090 00 00 0a 49 44 41 54 08 d7 63 60 00 00 00 02 00 |...IDAT..c`.....|
000000a0 01 e2 21 bc 33 00 00 00 25 74 45 58 74 64 61 74 |..!.3...%tEXtdat|
000000b0 65 3a 63 72 65 61 74 65 00 32 30 32 31 2d 30 31 |e:create.2021-01|
000000c0 2d 31 33 54 32 33 3a 30 34 3a 35 35 2b 30 30 3a |-13T23:04:55+00:|
000000d0 30 30 2d af d4 01 00 00 00 25 74 45 58 74 64 61 |00-......%tEXtda|
000000e0 74 65 3a 6d 6f 64 69 66 79 00 32 30 32 31 2d 30 |te:modify.2021-0|
000000f0 31 2d 31 33 54 32 33 3a 30 34 3a 35 35 2b 30 30 |1-13T23:04:55+00|
00000100 3a 30 30 5c f2 6c bd 00 00 00 00 49 45 4e 44 ae |:00\.l.....IEND.|
00000110 42 60 82 |B`.|
00000113
does not provide a clue, that I know of.
I understand that identifying the ColorSpace of an image, that does not contain that information, is a very hard problem -- see one proposed solution looking at the histogram of colors here.
So how identify, from the ImageMagick suite, determines the ColorSpace of this image?
It is common, but not standardized to assume that an image without an embedded or sidecar ICC profile or without an explicit encoding description is encoded according to IEC 61966-2-1:1999, i.e. sRGB specification.
This is just a bug in ImageMagick. You can use exiftool to check whether sRGB + intent chunk is present. In this case, no.
Gamma 2.2 is not sRGB. Thus ImageMagic is wrong here. That is a common problem on Wikipedia, all SVG images when converted to PNG have this and it destroys the colours. See: https://phabricator.wikimedia.org/T26768
We will have to reencode all images on Wikipedia, since we use ImageMagick. Sigh.

Nodejs WebSocket response is encrypted or not readable (connecting to a exchange socket)

I am trying to make a request to a WebSocket of a crypto-exchange to get live market data for analysis.
In the Browser the data is shown in readable Text as it is used to populate the orderbook.
When trying to connect to it via nodejs I am getting a hex code that when converted still doesn't make sense.
const WebSocket = require('ws');
const sendData = '{"method":"deals.subscribe","params":["ETHBTC"],"id":18}'
// Establish a WebSocket connection to the echo server
const ws = new WebSocket('wss://ws.exchnageAddres.com');
ws.addEventListener('open', () => {
console.log('Sending:', sendData);
// Send message to the WebSocket server
ws.send(sendData);
});
ws.addEventListener('message', event => {
console.log('Received:', event.data);
});
Response:
Received: <Buffer 1f 8b 08 00 00 00 00 00 00 03 ab 56 4a 2d 2a ca 2f 52 b2 52 c8 2b cd c9 d1 51 50 2a 4a 2d 2e cd 29 01 f2 ab 95 8a 4b 12 4b 4a 8b 81 4c a5 e2 d2 e4 e4 ... 23 more bytes>
Received: <Buffer 1f 8b 08 00 00 00 00 00 00 03 9d 9a df 8a 5c 47 0e c6 5f 25 cc 75 28 24 95 a4 52 ed 65 96 85 7d 80 dc 85 5c 78 d7 86 35 d8 89 89 ed 8b 10 f2 ee fb 95 ... 2012 more bytes>
Received: <Buffer 1f 8b 08 00 00 00 00 00 00 03 55 8c b1 0a 83 30 14 45 7f 45 de 5c 42 e2 6b 92 da d1 52 e8 07 b8 89 43 da 04 2a 18 0d 26 19 44 fc f7 e6 15 3a 74 3d f7 ... 106 more bytes>
How can I connect to the ws in node?
Do I have to use a certificate or anything? Would be awesome if someone could help me out here.
Testing the connection on a chrome extension is giving me a Blob object but I can't really convert that to an array.
Using Burp Decoder works but only there.
Thanks and best regards.

How is socket.io secured natively?

I'm looking at socket.io packets and they are TCP. When I review the value, I see encrypted data. Where and how is socket.io encrypting the messages that pass through the soccket? Is it really secure? This is a VM running with requests over http.
For example, I see
0000 bc ec 23 c3 64 6a 00 15 5d 01 59 06 08 00 45 08 ..#.dj..].Y...E.
0010 00 58 68 cc 40 00 40 06 4e 6c c0 a8 01 05 c0 a8 .Xh.#.#.Nl......
0020 01 0a 00 16 c6 51 15 15 7f 44 69 87 60 58 50 18 .....Q...Di.\XP.`
0030 0b 2e 6c ae 00 00 8b 6f 92 7f b9 1b c2 d6 54 60 ..l....o......T
0040 5e 24 65 2a 0c d6 87 90 fd 87 63 30 9d 69 11 26 ^$e*......c0.i.&
0050 4d 75 8c 7b 5e b2 ad 47 12 9d 05 d0 7c 3b 7c 9e Mu.{^..G....|;|.
0060 b1 0d a0 b7 f1 88 ......

Snort rules with content

This will generate an alert:
alert tcp any any <> any any (msg:"Test_A"; sid:3000001; rev:1;)
This will not:
alert tcp any any <> any any (msg:"Test_B"; content:"badurl.com"; http_header; sid:3000002; rev:1;)
I have tried: fast_pattern:only; metadata:service http; nocase; http_header; and others. I cannot get it to work at this generic level. Any ideas why the content attribute does not work? The packet has a URL.
Updated from the comments
0000 9c d2 4b 7d 96 60 3c 15 c2 dc 48 fa 08 00 45 00 ..K}.<. ..H...E.
0010 01 5c ac 2c 40 00 40 06 cf f5 c0 a8 c8 1e 41 fe .\.,#.#. ......A.
0020 f2 b4 dc 41 00 50 d0 e7 97 d0 ae b8 f9 ba 80 18 ...A.P.. ........
0030 ff ff da 1f 00 00 01 01 08 0a 34 03 84 d8 b7 cc ........ ..4.....
0040 3f 04 47 45 54 20 2f 20 48 54 54 50 2f 31 2e 31 ?.GET / HTTP/1.1
0050 0d 0a 48 6f 73 74 3a 20 6d 79 64 6f 6d 61 69 6e ..Host: mydomain
0060 2e 63 6f 6d 0d 0a 55 73 65 72 2d 41 67 65 6e 74 .com..Us er-Agent
The rule that you have provided will never fire with the example packet that you have provided. You have used a content:"POST"; with a http_method modifier but you are attempting to match a packet that is a GET request.
I think that the right content modifier should be http_uri, not http_header. Unless you are trying to capture the Host POST parameter.

Resources