I am trying to write a tcp server to get the data from Heacent 908 GPS tracker. After establishing the connection from the tracker I am getting the following buffer output.
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 06 64 be 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 06 64 be 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 06 64 be 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 06 64 be 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 06 64 be 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 06 64 be 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 06 64 be 0d 0a>
I am not sure how to decode this data into proper readable format.
Note: Off course I have tried to reach the manufacture but they are not responding at all.
What type of possible encoding formats are there for TCP protocol?
On next day I got data like this
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 07 75 37 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 07 75 37 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 07 75 37 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 07 75 37 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 07 75 37 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 07 75 37 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 07 75 37 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 07 75 37 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 07 75 37 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 1f 12 0e 02 14 13 01 14 c8 03 5f a6 50 07 f7 f8 c1 32 35 39 01 9a 04 0f a2 00 b0 5a 00 1a 9b 7a 0d 0a>
<Buffer 78 78 1f 12 0e 02 14 13 01 1e c8 03 5f ad bc 07 f7 f0 76 41 35 40 01 9a 04 0f a2 00 b0 5a 00 1b b6 31 0d 0a>
Something is being changed but not sure what is it...
You ask what possible encoding formats there are for TCP. That's a bit of an odd question: there are an unbounded number of encoding formats using TCP as the underlying protocol. But no matter, we can try to figure out this one!
You've posted some sample messages. Let's see if we can translate them:
byte 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
rev 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
----------------------------------------------------------
hex 78 78 0d 01 03 87 11 31 20 86 48 42 00 06 64 be 0d 0a
text x x \r -- -- -- -- 1 -- H B -- -- d -- \r \n
dec 13 1 3 17 0 6 100 13 10
be32 [218170247] [288432262] [ 419006]
----------------------------------------------------------
hex 78 78 0d 01 03 87 11 31 20 86 48 42 00 07 75 37 0d 0a
text -- u 7
dec 7 117 55
be32 [ 488759]
----------------------------------------------------------
hex 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a
text -- -- --
dec 8 141
be32 [ 560576]
----------------------------------------------------- byte 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
hex 78 78 1f 12 0e 02 14 13 01 14 c8 03 5f a6 50 07 f7 f8 c1 32 35 39 01 9a 04 0f a2 00 b0 5a 00 1a 9b 7a 0d 0a
text -- -- -- -- -- -- -- -- -- -- _ -- P -- -- -- -- 2 5 9 -- -- -- -- -- -- -- -- -- xx -- z \r \n
----------------------------------------------------------
hex 78 78 1f 12 0e 02 14 13 01 1e c8 03 5f ad bc 07 f7 f0 76 41 35 40 01 9a 04 0f a2 00 b0 5a 00 1b b6 31 0d 0a
text -- -- -- A 5 # -- xx -- 1
Some potentially interesting facts:
Starts with "xx\r\01" which more or less seems like a possible header. But later messages start with "xx" and something else. Anyway, given that NMEA has a prefix of "GP" I wouldn't be shocked if these devices used "xx" for "something that's not NMEA."
Has "HB" in the middle, which could mean "heartbeat" since this is repeating, perhaps waiting for a reply from the server.
Ends with "\r\n" which is a common line ending (on Windows in particular), though the rest doesn't appear to be entirely textual.
The earlier messages are 18 bytes long and the later ones 36 bytes. A guess would be the short ones are status updates or heartbeats and the long ones are actual location information. 36 bytes is enough if we figure:
4 byte latitude: 24 bits if you pinch (see), 25-32 bits more likely
4 byte longitude: same as latitude
6 byte timestamp: 39 bits if using epoch time with centiseconds, 32/48/64 bits more likely
2 byte altitude: I suspect this device doesn't publish altitude at all, given some of the docs
So I think what is going on is that these messages you see are just the device "pinging" the server and waiting for a response. What sort of response? Well, you could try to brute force it, but far, far easier would be to set up a bridge in your program that takes whatever it receives from the device, sends it to the manufacturer's server, and does the same thing in reverse for the responses to the device. This way you will quickly be able to gather a corpus of valid messages which will be very helpful if we really do need to reverse engineer this thing. Or if you're lucky it will turn out to use some standard protocol like NMEA after negotiating the initial session.
Edit: now that you've given us more messages from the device, we can see that it does seem to send something else with variable content. Maybe that's the location data, but I don't have time to try to reverse engineer it right now. One idea is to physically move the unit from west to east or north to south and capture the messages it sends during that time, to try to isolate which parts of the messages are the longitude and which are the latitude (and perhaps timestamp too).
I think it's fairly clear that the first two bytes are "xx" as a header, and the last two are "\r\n" as a terminator. That leaves 32 bytes of payload in the longer messages, all of which appears to be binary data.
It's the GT06 protocol and you can find it's specs here:
http://www.traccar.org/devices/
http://www.traccar.org/docs/protocol.jsp
https://dl.dropboxusercontent.com/s/sqtkulcj51zkria/GT06_GPS_Tracker_Communication_Protocol_v1.8.1.pdf
You can do it this way:
client.on('data', (buffer) => {
const decodedData = buffer.toString('utf8')
console.log(decodedData)
})
Related
Suppose I create a simple PNG with:
convert -size 1x1 canvas:red red.png
Here is a similar image (bigger size) for reference:
Then run the command identify on it. It tells me the ColorSpace of the image is sRGB but there seems to be NO indication of this inside the file. In fact running
$ hexdump -C red.png
00000000 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 |.PNG........IHDR|
00000010 00 00 00 01 00 00 00 01 01 03 00 00 00 25 db 56 |.............%.V|
00000020 ca 00 00 00 04 67 41 4d 41 00 00 b1 8f 0b fc 61 |.....gAMA......a|
00000030 05 00 00 00 20 63 48 52 4d 00 00 7a 26 00 00 80 |.... cHRM..z&...|
00000040 84 00 00 fa 00 00 00 80 e8 00 00 75 30 00 00 ea |...........u0...|
00000050 60 00 00 3a 98 00 00 17 70 9c ba 51 3c 00 00 00 |`..:....p..Q<...|
00000060 06 50 4c 54 45 ff 00 00 ff ff ff 41 1d 34 11 00 |.PLTE......A.4..|
00000070 00 00 01 62 4b 47 44 01 ff 02 2d de 00 00 00 07 |...bKGD...-.....|
00000080 74 49 4d 45 07 e5 01 0d 17 04 37 80 ef 04 02 00 |tIME......7.....|
00000090 00 00 0a 49 44 41 54 08 d7 63 60 00 00 00 02 00 |...IDAT..c`.....|
000000a0 01 e2 21 bc 33 00 00 00 25 74 45 58 74 64 61 74 |..!.3...%tEXtdat|
000000b0 65 3a 63 72 65 61 74 65 00 32 30 32 31 2d 30 31 |e:create.2021-01|
000000c0 2d 31 33 54 32 33 3a 30 34 3a 35 35 2b 30 30 3a |-13T23:04:55+00:|
000000d0 30 30 2d af d4 01 00 00 00 25 74 45 58 74 64 61 |00-......%tEXtda|
000000e0 74 65 3a 6d 6f 64 69 66 79 00 32 30 32 31 2d 30 |te:modify.2021-0|
000000f0 31 2d 31 33 54 32 33 3a 30 34 3a 35 35 2b 30 30 |1-13T23:04:55+00|
00000100 3a 30 30 5c f2 6c bd 00 00 00 00 49 45 4e 44 ae |:00\.l.....IEND.|
00000110 42 60 82 |B`.|
00000113
does not provide a clue, that I know of.
I understand that identifying the ColorSpace of an image, that does not contain that information, is a very hard problem -- see one proposed solution looking at the histogram of colors here.
So how identify, from the ImageMagick suite, determines the ColorSpace of this image?
It is common, but not standardized to assume that an image without an embedded or sidecar ICC profile or without an explicit encoding description is encoded according to IEC 61966-2-1:1999, i.e. sRGB specification.
This is just a bug in ImageMagick. You can use exiftool to check whether sRGB + intent chunk is present. In this case, no.
Gamma 2.2 is not sRGB. Thus ImageMagic is wrong here. That is a common problem on Wikipedia, all SVG images when converted to PNG have this and it destroys the colours. See: https://phabricator.wikimedia.org/T26768
We will have to reencode all images on Wikipedia, since we use ImageMagick. Sigh.
I use 'socat TCP4-LISTEN:8080,fork EXEC:./bashttpd' for http server. when try to receive image file from client socat remove some byte and corrupt my image.
correct:
01b0 0a 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 ..PNG........IHD
01c0 52 00 00 07 80 00 00 04 38 08 02 00 00 00 67 b1 R.......8.....g.
01d0 56 14 00 00 00 09 70 48 59 73 00 00 11 b0 00 00 V.
incorrect:(socat -> read line -> xxh)
00000000: 89 50 4e 47 0d 0a .PNG..
00000000: 1a 0a ..
00000000: 0d 49 48 44 52 07 80 04 38 08 02 67 b1 56 14 09 .IHDR...8..g.V
how to solve this problem?
thanks
I am trying to read files in a directory, for each file I am converting it into bytes array and appending everything in combinedBuffer array. Now I want to add all bytes buffer in combinedBuffer to finalBuffer, But I am not able to achieve this, it just iterate over the first bytes buffer in combinedBuffer rather than iterating over the entire combinedBuffer Array.
fileRead = async function(file){
const testFolder = './items/';
var dirFiles = [] // dirFiles Array
fs.readdirSync(testFolder).forEach(file => {
dirFiles.push(file)
})
dirFiles.sort(compareBasedonInt) //Natural Compare is to sort files in sampleFile1, SampleFile2.
var combinedBuffer = []
dirFiles.forEach(function(file){
var eachfileBuffer= fs.readFileSync('./items/'+file) //eachfileBuffer is the bytes buffer which we get after reading the file
combinedBuffer.push(eachfileBuffer)
})
var finalbuffer = Buffer.concat(combinedBuffer)
console.log("final buffer", finalbuffer) // finalBuffer shows just first element of combinedBuffer
console.log("combinedBuffer", combinedBuffer) // shows the array of all the bytes buffer
}
Output
combinedBuffer [ <Buffer 2f 51 40 42 0b 0a 53 0c 4a 25 05 55 7f 06 32 79 0d 50 47 0c 5d 3e 59 0b 51 54 40 5c 4a 5e 53 4f 15 05 4d 1f 41 4e 23 07 1f 52 5f 1f 52 48 14 52 4e 52 ... >,
<Buffer 19 42 01 0d 46 59 4a 58 1a 38 60 06 4a 11 2d 5c 70 65 2e 5a 17 0f 54 14 09 2a 05 14 38 34 47 0f 0e 42 5f 26 56 49 07 19 12 11 5e 4e 01 55 0b 41 26 72 ... >,
<Buffer 17 1a 11 5d 4d 19 1e 44 45 45 57 5f 05 49 19 1a 18 2a 23 27 46 00 47 45 17 45 1c 4f 5b 4f 4b 53 45 55 3e 4b 1d 08 4b 15 1a 1c 18 66 50 1a 4f 55 18 05 ... >,
<Buffer 44 0a 06 50 16 10 58 1b 15 40 07 22 58 0b 51 4e 08 07 46 6f 7c 66 12 57 4e 1d 1b 09 04 4a 40 4f 1e 11 5f 41 41 74 62 5f 76 43 5e 0d 1a 01 1a 0c 7d 44 ... >,
<Buffer 3c 09 5f 5c 6c 15 5c 02 15 5d 00 40 03 09 46 54 49 16 5c 5a ff 9f a2 45 43 00 46 46 65 1b 11 5c 5d 54 29 46 44 07 02 14 45 4b 31 6a 3a 4d 0a 58 18 47 ... >,
<Buffer 6d 4a 19 1b 85 bd ec 0c 0f 0f 4c 4a 41 08 09 1b 4f 0b 0a 1d 41 11 53 4b 1e 41 49 1e ce aa ad 11 22 1d 0f 52 46 1b 7a 0b 0e 15 1e 41 53 02 10 19 4f 01 ... >,
<Buffer 03 58 69 48 04 5f 52 02 08 56 1f 07 76 4e 1d 53 12 5e 5e 58 47 0a 5a 44 77 06 16 5e 03 4a 4d 56 56 01 1f 0b 0c 0c 40 45 0e 0c 4e 08 1b 12 1a 45 4e 50 ... >,
<Buffer 4d 5c 0c 4e 1d c2 a2 f9 1b 11 16 5f 52 00 0c 02 54 40 4d 17 0d 58 4c 1b 24 5b 12 41 4b 47 1a 0a 43 50 7c 6d 34 46 42 5a 43 42 43 38 1b 4b 53 57 5f 43 ... >,
<Buffer 49 4e 1a 13 1c 19 0b 0a 5d 19 41 46 06 14 15 51 15 15 56 57 1c 5d 09 1e 00 41 4a 6f 13 0a 60 60 64 16 4c 01 1e 49 46 0e 50 1d 51 56 5c 09 0e 37 16 12 ... >,
<Buffer 58 1d 4f 5b 4c 33 09 4f 08 19 04 1b 69 7e 2d 5d 22 95 be 8b 14 1b 76 4b 07 21 1f 4f 05 4b 30 2e 02 57 07 00 4d 5f 45 1c 08 39 37 37 41 47 25 09 05 4d ... >,
<Buffer 58 42 43 1f 5e 1e 01 17 54 04 52 1c 1a 04 42 48 52 48 3e 03 05 12 4c 16 5a 49 7e 14 49 53 1a 6f 6f 2f 46 54 54 07 00 7a 23 0a 1c 3b 56 0f 73 0a 00 0b ... >]
finalbuffer <Buffer 2f 51 40 42 0b 0a 53 0c 4a 25 05 55 7f 06 32 79 0d 50 47 0c 5d 3e 59 0b 51 54 40 5c 4a 5e 53 4f 15 05 4d 1f 41 4e 23 07 1f 52 5f 1f 52 48 14 52 4e 52 ... >
I'm trying to modify buffers, however when modifying them I wish them to be in utf-8 so I attempt to do this via myBuffer.toString('utf8') however if I make no changes and attempt to convert it back via Buffer.from(myBuffer, 'utf8'), I am presented with a completely new buffer on occasions.
These occasions seem to be when parsing an image file, instead of a html file.
My next step was to accept a bug or erroneous behaviour by comparing the two buffers using the following code:
//myBuffer is the buffer is wish to attempt to modify
let testParse = Buffer.from(myBuffer.toString('utf8'), 'utf8');
let editable = Buffer.compare(myBuffer, testParse);
console.log(myBuffer);
console.log(testParse);
console.log(editable);
The following snippet however refuses to work and editable is always -1 here is an example output:
<Buffer 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 00 00 01 10 00 00 00 5c 08 02 00 00 00 29 85 7d e1 00 00 15 31 49 44 41 54 78 01 ed 5d 05 94 db c8 b2 ... >
<Buffer ef bf bd 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 00 00 01 10 00 00 00 5c 08 02 00 00 00 29 ef bf bd 7d ef bf bd 00 00 15 31 49 44 41 54 78 01 ef ... >
-1
As you can see the buffers are completely different however returns -1
another example where the buffers are both equal:
<Buffer 3c 21 64 6f 63 74 79 70 65 20 68 74 6d 6c 3e 3c 68 74 6d 6c 20 69 74 65 6d 73 63 6f 70 65 3d 22 22 20 69 74 65 6d 74 79 70 65 3d 22 68 74 74 70 3a 2f ... >
<Buffer 3c 21 64 6f 63 74 79 70 65 20 68 74 6d 6c 3e 3c 68 74 6d 6c 20 69 74 65 6d 73 63 6f 70 65 3d 22 22 20 69 74 65 6d 74 79 70 65 3d 22 68 74 74 70 3a 2f ... >
-1
As you can see both buffers are equal and -1 is still returned.
So my question is, what am I doing wrong so that the buffers cannot be compared properly? Any suggestions/criticism are welcome.
You have to compare in the same encoding :
//:Buffer Comparison
const fs = require('fs')
fs.readFile(__dirname+"/test.jpg",(e,buffer)=>{
let testParse = Buffer.from(buffer.toString('utf8'), 'utf8');
let editable = Buffer.compare(buffer, testParse);
console.log("----: wrong method :----")
console.log(buffer);
console.log(testParse);
console.log(editable);
// You have to compare them in the same encoding :
console.log("----: right method :----")
let goodParse = Buffer.from(buffer.toString('utf8'));
let editable2 = goodParse.compare(Buffer.from(buffer.toString('utf8')));
console.log(buffer);
console.log(goodParse);
console.log(editable2);
})
As you can see, we load an image as a buffer, then it is parsed into utf8, so if you modify it, and then want to compare it to the original buffer. Since the modified was parsed to utf8 the original must also be parsed to utf8 in the moment of the comparison.
I hope you understand that explanation.
Console output:
----: wrong method :----
<Buffer ff d8 ff e0 00 10 4a 46 49 46 00 01 01 00 00 01 00 01 00 00 ff db 00 43 00 08 06 06 07 06 05 08 07 07 07 09 09 08 0a 0c 14 0d 0c 0b 0b 0c 19 12 13 0f ... >
<Buffer ef bf bd ef bf bd ef bf bd ef bf bd 00 10 4a 46 49 46 00 01 01 00 00 01 00 01 00 00 ef bf bd ef bf bd 00 43 00 08 06 06 07 06 05 08 07 07 07 09 09 08 ... >
1
----: right method :----
<Buffer ff d8 ff e0 00 10 4a 46 49 46 00 01 01 00 00 01 00 01 00 00 ff db 00 43 00 08 06 06 07 06 05 08 07 07 07 09 09 08 0a 0c 14 0d 0c 0b 0b 0c 19 12 13 0f ... >
<Buffer ef bf bd ef bf bd ef bf bd ef bf bd 00 10 4a 46 49 46 00 01 01 00 00 01 00 01 00 00 ef bf bd ef bf bd 00 43 00 08 06 06 07 06 05 08 07 07 07 09 09 08 ... >
0
Here is my CoffeeScript:
buffer = new Buffer 100
buffer[i] = i for i in [0..99]
console.log buffer
which compiles to
var buffer, i;
buffer = new Buffer(100);
for (i = 0; i < buffer.length; i++) {
buffer[i] = i;
}
console.log(buffer);
When I run it with node, I get the following output:
$ coffee exercise1
<Buffer 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f 60 61 62 63>
instead of 0 to 99. Why is that?
Ray nailed it in his comment. See the Buffer documentation; you have to specify an encoding argument (you probably want 'utf8') on a Buffer's toString.
// coffeescript
console.log buffer.toString 'utf8'
// javascript
console.log(buffer.toString('utf8'));