I have a JPEG that is un-openable in any program:
Opening in Ubuntu Image Viewer yields:
Passing the photo through convert yields similar results:
$ convert corrupt.jpg out.jpg
convert.im6: Bogus Huffman table definition `corrupt.jpg' # error/jpeg.c/JPEGErrorHandler/316.
convert.im6: no images defined `out.jpg' # error/convert.c/ConvertImageCommand/3044.
Running the photo through exiftool yields:
ExifTool Version Number : 9.46
File Name : corrupt.jpg
Directory : .
File Size : 47 kB
File Modification Date/Time : 2015:04:11 01:31:14-07:00
File Access Date/Time : 2018:05:04 10:26:04-07:00
File Inode Change Date/Time : 2018:05:04 10:26:03-07:00
File Permissions : r--------
File Type : JPEG
MIME Type : image/jpeg
Comment : Y�.�.�..2..Q.Q.
Image Width : 640
Image Height : 480
Encoding Process : Baseline DCT, Huffman coding
Bits Per Sample : 8
Color Components : 3
Y Cb Cr Sub Sampling : YCbCr4:2:2 (2 1)
Image Size : 640x480
Un-corrupted photos containing similar image contents average 45-48k, so I reckon the photo data itself is inside this JPEG somewhere.
I hosted the photo on S3. You can download it w/ wget:
wget https://s3.amazonaws.com/jordanarseno.com/corrupt.jpg
I opened the file with hexedit and found the following:
the photo contents outside of the first few hundred bytes is randomly distributed enough to suggest it contains an image. i.e. I'm not seeing consecutive streams of 0's of F's.
it does in-fact start with the FF D8 file signature, as JPEGs ought to.
the next two bytes are not FF E0 or FF E1 like the list of file signatures says should correspond to JPEGs or JFIFs. Instead it isFF FE. Which, is in the table, but is listed as:
Byte-order mark for text file encoded in little-endian 16-bit Unicode
Transfer Format
not long after the FF FE, I see bytes whose ascii representation is: &'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz. Seems rather strange for a JPEG. What is this?
likewise, the ASCII string &'()*56789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz appears about 100 bytes later.
FF D9 (the JPEG terminator string) is in the file, but characters do appear after this terminator:
FF D9 5C 72 78 E0 7C 94 CD B2 9C FF 00 C4 BF 53 C0 E7 FE 41 D3 9C FF 00 E3 95 7C F1 B6 92 5F 7A 2B EB 54 AF BF E6 30 FD A0 7F CC 3B 53 E9 FF 00 40 F9 FF 00 F8 8A 4D F7 08 30
Switching over to Windows and using JPEGsnoop yields:
JPEGsnoop 1.8.0 by Calvin Hass
http://www.impulseadventure.com/photo/
-------------------------------------
Filename: [C:\corrupt.jpg]
Filesize: [47760] Bytes
Start Offset: 0x00000000
*** Marker: SOI (xFFD8) ***
OFFSET: 0x00000000
*** Marker: COM (Comment) (xFFFE) ***
OFFSET: 0x00000002
Comment length = 36
Comment=Y.Ò................à.....2..Q.Q...
*** Marker: DQT (xFFDB) ***
Define a Quantization Table.
OFFSET: 0x00000028
Table length = 132
----
Precision=8 bits
Destination ID=0 (Luminance)
DQT, Row #0: 3 2 2 3 4 7 9 10
DQT, Row #1: 2 2 2 3 4 10 10 9
DQT, Row #2: 2 2 3 4 7 10 12 10
DQT, Row #3: 2 3 4 5 9 15 14 11
DQT, Row #4: 3 4 6 10 12 19 18 13
DQT, Row #5: 4 6 9 11 14 18 19 16
DQT, Row #6: 8 11 13 15 18 21 21 17
DQT, Row #7: 12 16 16 17 19 17 18 17
Approx quality factor = 91.45 (scaling=17.09 variance=0.95)
----
Precision=8 bits
Destination ID=1 (Chrominance)
DQT, Row #0: 3 3 4 8 17 17 17 17
DQT, Row #1: 3 4 4 11 17 17 17 17
DQT, Row #2: 4 4 10 17 17 17 17 17
DQT, Row #3: 8 11 17 17 17 17 17 17
DQT, Row #4: 17 17 17 17 17 17 17 17
DQT, Row #5: 17 17 17 17 17 17 17 17
DQT, Row #6: 17 17 17 17 17 17 17 17
DQT, Row #7: 17 17 17 17 17 17 17 17
Approx quality factor = 91.44 (scaling=17.11 variance=0.19)
*** Marker: COM (Comment) (xFFFE) ***
OFFSET: 0x000000AE
Comment length = 5
Comment=...
*** Marker: SOF0 (Baseline DCT) (xFFC0) ***
OFFSET: 0x000000B5
Frame header length = 17
Precision = 8
Number of Lines = 480
Samples per Line = 640
Image Size = 640 x 480
Raw Image Orientation = Landscape
Number of Img components = 3
Component[1]: ID=0x01, Samp Fac=0x21 (Subsamp 1 x 1), Quant Tbl Sel=0x00 (Lum: Y)
Component[2]: ID=0x02, Samp Fac=0x11 (Subsamp 2 x 1), Quant Tbl Sel=0x01 (Chrom: Cb)
Component[3]: ID=0x03, Samp Fac=0x11 (Subsamp 2 x 1), Quant Tbl Sel=0x01 (Chrom: Cr)
*** Marker: DHT (Define Huffman Table) (xFFC4) ***
OFFSET: 0x000000C8
Huffman table length = 418
----
Destination ID = 0
Class = 0 (DC / Lossless Table)
Codes of length 01 bits (000 total):
Codes of length 02 bits (001 total): 00
Codes of length 03 bits (005 total): 01 02 03 04 05
Codes of length 04 bits (001 total): 06
Codes of length 05 bits (001 total): 07
Codes of length 06 bits (001 total): 08
Codes of length 07 bits (001 total): 09
Codes of length 08 bits (001 total): 0A
Codes of length 09 bits (001 total): 0B
Codes of length 10 bits (000 total):
Codes of length 11 bits (000 total):
Codes of length 12 bits (000 total):
Codes of length 13 bits (000 total):
Codes of length 14 bits (000 total):
Codes of length 15 bits (000 total):
Codes of length 16 bits (000 total):
Total number of codes: 012
----
Destination ID = 1
Class = 0 (DC / Lossless Table)
Codes of length 01 bits (000 total):
Codes of length 02 bits (003 total): 13 0E 0F
Codes of length 03 bits (001 total): 10
Codes of length 04 bits (001 total): 11
Codes of length 05 bits (001 total): 12
Codes of length 06 bits (001 total): 12
Codes of length 07 bits (012 total): 12 0B 0D 13 15 13 11 15 10 11 12 11
Codes of length 08 bits (016 total): 01 03 03 03 04 04 04 08 04 04 08 11 0B 0A 0B 11
Codes of length 09 bits (013 total): 11 11 11 11 11 11 11 11 11 11 11 11 11
Codes of length 10 bits (011 total): 11 11 11 11 11 11 11 11 11 11 11
Codes of length 11 bits (012 total): 11 11 11 11 11 11 11 11 11 11 11 01
Codes of length 12 bits (015 total): 01 01 01 01 00 00 00 00 00 00 01 02 03 04 05
Codes of length 13 bits (012 total): 06 07 08 09 0A 0B 10 00 02 01 03 03
Codes of length 14 bits (009 total): 02 04 03 05 05 04 04 00 00
Codes of length 15 bits (010 total): 01 7D 01 02 03 00 04 11 05 12
Codes of length 16 bits (014 total): 21 31 41 06 13 51 61 07 22 71 14 32 81 91
Total number of codes: 131
----
Destination ID = 1
Class = 10 (AC Table)
ERROR: Invalid DHT Class (10). Aborting DHT Load.
ERROR: Expected marker 0xFF, got 0x73 # offset 0x0000026C. Consider using [Tools->Img Search Fwd/Rev].
*** Searching Compression Signatures ***
Signature: 01FF5BA518B453CC8F224A4C85505196
Signature (Rotated): 01D13AFD01FF0B6EC46EA4081D25BB4D
File Offset: 0 bytes
Chroma subsampling: 2x1
EXIF Make/Model: NONE
EXIF Makernotes: NONE
EXIF Software: NONE
Searching Compression Signatures: (3347 built-in, 0 user(*) )
EXIF.Make / Software EXIF.Model Quality Subsamp Match?
------------------------- ----------------------------------- ---------------- --------------
CAM:[NIKON ] [NIKON D40 ] [FINE ] Yes
Based on the analysis of compression characteristics and EXIF metadata:
ASSESSMENT: Class 1 - Image is processed/edited
This may be a new software editor for the database.
If this file is processed, and editor doesn't appear in list above,
PLEASE ADD TO DATABASE with [Tools->Add Camera to DB]
*** Additional Info ***
NOTE: Data exists after EOF, range: 0x00000000-0x0000BA90 (47760 bytes)
As a last note, the EXIF.Model identified by JPEGSnoop is incorrect. This photo would have been taken with a VC0706 UART Model: LCF - 23T 0V528
In summary: Is this JPEG recoverable?
The approach used to get this back was more luck than judgement. I think I can explain, though be aware it involves a hex editor...
The Wikipedia page for the syntax of a JPEG file explains that it is made up of a series of segments each started by a two byte marker - 0xFF and another byte to indicate the type of segment.
The hope was that it was just the Huffman table segment of the file that was wrong - as suggested by the error message. Without needing to understand what a Huffman table is, it was enough to see that the same section on Wikipedia explains it is a 0xFF 0xC4 marker for a Huffman table segment.
Further down the page, it mentions:
The JPEG standard provides general-purpose Huffman tables; encoders
may also choose to generate Huffman tables...
Opening up a few other JPEG files found what looks like a standard set of 4 consecutive Huffman table segments - each starting with that 0xFF 0xC4 marker. The sample corrupt.jpg however just had one Huffman table - from position 0x00c8 to 0x02bc below.
(Both contain that &'()*56789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz sequence you mentioned in their Huffman tables. In the corrupt file it appears twice in that single Huffman table, in the 'more conventional' JPEGs it appears in the second and fourth Huffman tables.)
From there, the fixed image is a copy and paste of the standard 4 Huffman tables, in place of that range of bytes in corrupt.jpg - now from 0x00c8 to 0x0278 in the fixed file.
Because the JPEG format is based around scanning for segments between those 0xff markers, you can just swap out the Huffman segments - there are no other pointers in the file to worry about. As you said, the rest of the file looked like a plausible JPEG.
Summary of the steps taken:
Hex search the corrupt.jpg for FF C4 and note the offset
Hex search for the next FF. If it's another FF C4 (so a second Huffman table) keep going
Delete the content from the first FF C4 (included) up to but not including the next FF
Instead replace it with the 'standard 4 Huffman tables'. These are the bytes in the last sample below, or can be copied from 0x00c8 to 0x0278 in the fixed file
Corrupt Huffman table:
0000-00d0: xx xx xx xx xx xx xx xx-ff c4 01 a2-00 00 01 05 !....... ........
0000-00e0: 01 01 01 01-01 01 00 00-00 00 00 00-00 00 01 02 ........ ........
0000-00f0: 03 04 05 06-07 08 09 0a-0b 01 00 03-01 01 01 01 ........ ........
0000-0100: 0c 10 0d 0b-0c 0f 0c 09-0a 0e 13 0e-0f 10 11 12 ........ ........
0000-0110: 12 12 0b 0d-13 15 13 11-15 10 11 12-11 01 03 03 ........ ........
0000-0120: 03 04 04 04-08 04 04 08-11 0b 0a 0b-11 11 11 11 ........ ........
0000-0130: 11 11 11 11-11 11 11 11-11 11 11 11-11 11 11 11 ........ ........
0000-0140: 11 11 11 11-11 11 11 11-11 11 11 11-11 11 11 11 ........ ........
0000-0150: 01 01 01 01-01 00 00 00-00 00 00 01-02 03 04 05 ........ ........
0000-0160: 06 07 08 09-0a 0b 10 00-02 01 03 03-02 04 03 05 ........ ........
0000-0170: 05 04 04 00-00 01 7d 01-02 03 00 04-11 05 12 21 ......}. .......!
0000-0180: 31 41 06 13-51 61 07 22-71 14 32 81-91 a1 08 23 1A..Qa." q.2....#
0000-0190: 42 b1 c1 15-52 d1 f0 24-33 62 72 82-09 0a 16 17 B...R..$ 3br.....
0000-01a0: 18 19 1a 25-26 27 28 29-2a 34 35 36-37 38 39 3a ...%&'() *456789:
0000-01b0: 43 44 45 46-47 48 49 4a-53 54 55 56-57 58 59 5a CDEFGHIJ STUVWXYZ
0000-01c0: 63 64 65 66-67 68 69 6a-73 74 75 76-77 78 79 7a cdefghij stuvwxyz
0000-01d0: 83 84 85 86-87 88 89 8a-92 93 94 95-96 97 98 99 ........ ........
0000-01e0: 9a a2 a3 a4-a5 a6 a7 a8-a9 aa b2 b3-b4 b5 b6 b7 ........ ........
0000-01f0: b8 b9 ba c2-c3 c4 c5 c6-c7 c8 c9 ca-d2 d3 d4 d5 ........ ........
0000-0200: d6 d7 d8 d9-da e1 e2 e3-e4 e5 e6 e7-e8 e9 ea f1 ........ ........
0000-0210: f2 f3 f4 f5-f6 f7 f8 f9-fa 11 00 02-01 02 04 04 ........ ........
0000-0220: 03 04 07 05-04 04 00 01-02 77 00 01-02 03 11 04 ........ .w......
0000-0230: 05 21 31 06-12 41 51 07-61 71 13 22-32 81 08 14 .!1..AQ. aq."2...
0000-0240: 42 91 a1 b1-c1 09 23 33-52 f0 15 62-72 d1 0a 16 B.....#3 R..br...
0000-0250: 24 34 e1 25-f1 17 18 19-1a 26 27 28-29 2a 35 36 $4.%.... .&'()*56
0000-0260: 37 38 39 3a-43 44 45 46-47 48 49 4a-53 54 55 56 789:CDEF GHIJSTUV
0000-0270: 57 58 59 5a-63 64 65 66-67 68 69 6a-73 74 75 76 WXYZcdef ghijstuv
0000-0280: 77 78 79 7a-82 83 84 85-86 87 88 89-8a 92 93 94 wxyz.... ........
0000-0290: 95 96 97 98-99 9a a2 a3-a4 a5 a6 a7-a8 a9 aa b2 ........ ........
0000-02a0: b3 b4 b5 b6-b7 b8 b9 ba-c2 c3 c4 c5-c6 c7 c8 c9 ........ ........
0000-02b0: ca d2 d3 d4-d5 d6 d7 d8-d9 da e2 e3-e4 e5 e6 e7 ........ ........
0000-02c0: e8 e9 ea f2-f3 f4 f5 f6-f7 f8 f9 fa-xx xx xx xx ........ ........
Then the next two bytes are ff dd for the start of the next segment:
0000-02c0: xx xx xx xx-xx xx xx xx-xx xx xx xx-ff dd 00 04 ........ ........
This was replaced with the standard 4 general-purpose Huffman tables instead - look for the ff c4 markers:
0000-00d0: xx xx xx xx xx xx xx xx-ff c4 00 1f-00 00 01 05 !....... ........
0000-00e0: 01 01 01 01-01 01 00 00-00 00 00 00-00 00 01 02 ........ ........
0000-00f0: 03 04 05 06-07 08 09 0a-0b ff c4 00-b5 10 00 02 ........ ........
0000-0100: 01 03 03 02-04 03 05 05-04 04 00 00-01 7d 01 02 ........ .....}..
0000-0110: 03 00 04 11-05 12 21 31-41 06 13 51-61 07 22 71 ......!1 A..Qa."q
0000-0120: 14 32 81 91-a1 08 23 42-b1 c1 15 52-d1 f0 24 33 .2....#B ...R..$3
0000-0130: 62 72 82 09-0a 16 17 18-19 1a 25 26-27 28 29 2a br...... ..%&'()*
0000-0140: 34 35 36 37-38 39 3a 43-44 45 46 47-48 49 4a 53 456789:C DEFGHIJS
0000-0150: 54 55 56 57-58 59 5a 63-64 65 66 67-68 69 6a 73 TUVWXYZc defghijs
0000-0160: 74 75 76 77-78 79 7a 83-84 85 86 87-88 89 8a 92 tuvwxyz. ........
0000-0170: 93 94 95 96-97 98 99 9a-a2 a3 a4 a5-a6 a7 a8 a9 ........ ........
0000-0180: aa b2 b3 b4-b5 b6 b7 b8-b9 ba c2 c3-c4 c5 c6 c7 ........ ........
0000-0190: c8 c9 ca d2-d3 d4 d5 d6-d7 d8 d9 da-e1 e2 e3 e4 ........ ........
0000-01a0: e5 e6 e7 e8-e9 ea f1 f2-f3 f4 f5 f6-f7 f8 f9 fa ........ ........
0000-01b0: ff c4 00 1f-01 00 03 01-01 01 01 01-01 01 01 01 ........ ........
0000-01c0: 00 00 00 00-00 00 01 02-03 04 05 06-07 08 09 0a ........ ........
0000-01d0: 0b ff c4 00-b5 11 00 02-01 02 04 04-03 04 07 05 ........ ........
0000-01e0: 04 04 00 01-02 77 00 01-02 03 11 04-05 21 31 06 .....w.. .....!1.
0000-01f0: 12 41 51 07-61 71 13 22-32 81 08 14-42 91 a1 b1 .AQ.aq." 2...B...
0000-0200: c1 09 23 33-52 f0 15 62-72 d1 0a 16-24 34 e1 25 ..#3R..b r...$4.%
0000-0210: f1 17 18 19-1a 26 27 28-29 2a 35 36-37 38 39 3a .....&'( )*56789:
0000-0220: 43 44 45 46-47 48 49 4a-53 54 55 56-57 58 59 5a CDEFGHIJ STUVWXYZ
0000-0230: 63 64 65 66-67 68 69 6a-73 74 75 76-77 78 79 7a cdefghij stuvwxyz
0000-0240: 82 83 84 85-86 87 88 89-8a 92 93 94-95 96 97 98 ........ ........
0000-0250: 99 9a a2 a3-a4 a5 a6 a7-a8 a9 aa b2-b3 b4 b5 b6 ........ ........
0000-0260: b7 b8 b9 ba-c2 c3 c4 c5-c6 c7 c8 c9-ca d2 d3 d4 ........ ........
0000-0270: d5 d6 d7 d8-d9 da e2 e3-e4 e5 e6 e7-e8 e9 ea f2 ........ ........
0000-0280: f3 f4 f5 f6-f7 f8 f9 fa-xx xx xx xx xx xx xx xx ........ .....(..
Related
Python Version: 3.7.2
I need to convert a string in ASCII like Øâþ ÿþ !Zk2ìm "Ï"À>q úÞ to Hexademical, which in this case would be d8 e2 02 12 02 fe 01 20 9b 10 20 20 03 ff 07 fe 20 20 21 5a 6b 32 ec 17 6d 20 0e 22 cf 22 c0 3e 71 20 02 20 03 fa de. I found several solutions for doing this on Python 2, however I can't find any way of doing this on Python 3.
To summarise: The intender behaviour is ASCII to HEX as follows:
Øâþ ÿþ !Zk2ìm "Ï"À>q úÞ TO d8 e2 02 12 02 fe 01 20 9b 10 20 20 03 ff 07 fe 20 20 21 5a 6b 32 ec 17 6d 20 0e 22 cf 22 c0 3e 71 20 02 20 03 fa de.
I've even checked on https://www.rapidtables.com/convert/number/ascii-to-hex.html and found it works, but I'm unable to implement it in Python 3.
You may use the code:
print(*[hex(ord(letter))[2:] for letter in 'Øâþ ÿþ !Zk2ìm "Ï"À>q úÞ'])
which gives the following output:
d8 e2 fe 20 10 20 20 ff fe 20 20 21 5a 6b 32 ec 6d 20 e 22 cf 22 c0 3e 71 20 20 fa de
ord() - get ascii code,
hex() - get hex from int,
[2:] - to omit 0x in every number.
EDIT
Slightly modified version (to get 0e instead of e):
string = 'Øâþ ÿþ !Zk2ìm "Ï"À>q úÞ'
print(*['{:02x}'.format(ord(letter)) for letter in string])
Use ord():
s = 'Øâþ ÿþ !Zk2ìm "Ï"À>q úÞ'
bytes = bytearray(ord(char) for char in s)
print(bytes)
Output:
bytearray(b'\xd8\xe2\xfe \x10 \xff\xfe !Zk2\xecm \x0e"\xcf"\xc0>q \xfa\xde')
That being said I can't match your output exactly because you copied and pasted a garbage char:
print(''.join(chr(char) for char in bytes)) # Øâþ ÿþ !Zk2ìm "Ï"À>q úÞ
I'm doing some EMV testing, and hoping someone can shed some light on what I'm seeing. I'm running the Mastercard M-TIP tests that apply to our solution, and I'm seeing some unexpected results. When running the M-TIP 50 tests (1, 2, 3) with the M-TIP 50 test card, I'm getting errors on the 2nd GENERATE_AC command. Our solution is technically defined as offline with online capability, but we're currently testing in an entirely offline environment.
I've spent a good amount of time reading the EMV books and scouring Google, but I still don't understand why it's returning 6985. The best guess I can come up with at this point is it has something to do with CDA (which should also be happening during this process?)
I've included the 1st and 2nd GEN_AC requests and responses to show what's happening. If someone could provide insight into what's happening or what's going wrong, I would greatly appreciate it.
1st Generate AC (ARQC)
Request : 80 AE 90 00 2F 00 00 00 00 20 00 00 00 00 00 00 00 00 80 00 80 00 08 26 17 01 03 00 B0 32 0F C0 22 00 00 00 00 00 00 00 00 00 00 44 03 02 16 11 12 60 00 80
Class :80
Ins :AE
P1 :90
P2 :00
Lc :2F
Data :00 00 00 00 20 00 00 00 00 00 00 00 00 80 00 80 00 08 26 17 01 03 00 B0 32 0F C0 22 00 00 00 00 00 00 00 00 00 00 44 03 02 16 11 12 60 00 80
Tag 9F 02: Transaction Amount : 00 00 00 00 20 00
Amount value: 20.00
Tag 9F 03: Cashback Amount : 00 00 00 00 00 00
Amount value: 0.00
Tag 95 : Terminal Verification Results (TVR) : 00 80 00 80 00
Byte 1 bit 8 = 0 Offline data authentication was performed
bit 7 = 0 SDA passed or not performed
bit 6 = 0 No ICC data missing
bit 5 = 0 Card does not appear on terminal exception file
bit 4 = 0 DDA passed or not performed
bit 3 = 0 CDA passed or not performed
bit 2 = 0 SDA not selected
bit 1 = 0 RFU
Byte 2 bit 8 = 1 ICC and terminal have different application versions
bit 7 = 0 No Expired application
bit 6 = 0 Application effective
bit 5 = 0 Requested service allowed for card product
bit 4 = 0 No New card
bit 3 = 0 RFU
bit 2 = 0 RFU
bit 1 = 0 RFU
Byte 3 bit 8 = 0 Cardholder verification was successful or not performed
bit 7 = 0 Recognised CVM
bit 6 = 0 PIN Try Limit not exceeded
bit 5 = 0 No PIN entry required (PIN pad may or may not be present or may or may not be working)
bit 4 = 0 No PIN entry required (PIN pad may or may not be present)
bit 3 = 0 No Online PIN entered
bit 2 = 0 RFU
bit 1 = 0 RFU
Byte 4 bit 8 = 1 Transaction exceeds floor limit
bit 7 = 0 Lower consecutive offline limit not exceeded
bit 6 = 0 Upper consecutive offline limit not exceeded
bit 5 = 0 Transaction not selected randomly for online processing
bit 4 = 0 Merchant did not force transaction online
bit 3 = 0 RFU
bit 2 = 0 RFU
bit 1 = 0 RFU
Byte 5 bit 8 = 0 No Default TDOL used
bit 7 = 0 Issuer authentication passed or not performed
bit 6 = 0 Script processing passed before final GENERATE AC or no script received
bit 5 = 0 Script processing passed after final GENERATE AC or no script received
bit 4 = 0 RFU
bit 3 = 0 RFU
bit 2 = 0 RFU
bit 1 = 0 RFU
Tag 5F 2A: Transaction Currency Code : 08 26
Code (num) = 08 26
Code (an) = GBP
Currency = Pound Sterling
Tag 9A : Transaction Date : 17 01 03
Year : 2017
Month: January
Day : 03
Tag 9C : Transaction Type : 00
Purchase (of goods or services)
Tag 9F 37: Unpredictable Number : B0 32 0F C0
Tag 9F 35: Terminal Type : 22
Terminal Type: 22
Attended
Merchant
Offline with online capability
Tag 9F 45: Data Authentication Code : 00 00
Tag 9F 4C: ICC Dynamic Number : 00 00 00 00 00 00 00 00
Tag 9F 34: Cardholder Verification Method (CVM) Results : 44 03 02
Byte 1 bit 8 = 0 (default value)
bit 7 = 1 (Apply succeeding CVM field if this CVM is unsuccessful)
bit 6-1= 000100 (Enciphered PIN verification performed by ICC)
Byte 2 = '03' (If terminal supports the CVM type)
Byte 3 = '02' (Successful)
Tag 9F 21: Transaction Time : 16 11 12
Time = 16:11:12
Hours (HH) = 16
Minutes (MM) = 11
Seconds (SS) = 12
Tag 9F 40: Additional Terminal Capabilities : 60 00 80
Byte 1 bit 8 = 0 Cash NOT supported
bit 7 = 1 Goods supported
bit 6 = 1 Services supported
bit 5 = 0 CashBack NOT supported
bit 4 = 0 Inquiry NOT supported
bit 3 = 0 Transfer NOT supported
bit 2 = 0 Payment NOT supported
bit 1 = 0 Administrative NOT supported
Byte 2 bit 8 = 0 CashBack Deposit NOT supported
bit 7 = 0 RFU
bit 6 = 0 RFU
bit 5 = 0 RFU
bit 4 = 0 RFU
bit 3 = 0 RFU
bit 2 = 0 RFU
bit 1 = 0 RFU
Byte 3 bit 8 = 1 Numeric keys supported
bit 7 = 0 Alphabetic and special characters keys NOT supported
bit 6 = 0 Command keys NOT supported
bit 5 = 0 Function keys NOT supported
bit 4 = 0 RFU
bit 3 = 0 RFU
bit 2 = 0 RFU
bit 1 = 0 RFU
Byte 4 bit 8 = 0 Print, attendant NOT supported
bit 7 = 0 Print, cardholder NOT supported
bit 6 = 0 Display, attendant NOT supported
bit 5 = 0 Display, cardholder NOT supported
bit 4 = 0 RFU
bit 3 = 0 RFU
bit 2 = 0 Code table 10 NOT supported
bit 1 = 0 Code table 9 NOT supported
Byte 5 bit 8 = 0 Code table 8 NOT supported
bit 7 = 0 Code table 7 NOT supported
bit 6 = 0 Code table 6 NOT supported
bit 5 = 0 Code table 5 NOT supported
bit 4 = 0 Code table 4 NOT supported
bit 3 = 0 Code table 3 NOT supported
bit 2 = 0 Code table 2 NOT supported
bit 1 = 0 Code table 1 NOT supported
masterKeyAC: 9E 15 20 43 13 F7 31 8A CB 79 B9 0B D9 86 AD 29
uniqueKeyAC: 80 32 AD CE E0 B9 40 BA FB E3 5B 5B 15 9E 8F EA
MCHIP SKD Session Key Derivation
ATC: 00 08
UN: B0 32 0F C0
Cryptogram Version No.: 10
ICC Master Key AC: 9E 15 20 43 13 F7 31 8A CB 79 B9 0B D9 86 AD 29
Derived Card Unique Key: 80 32 AD CE E0 B9 40 BA FB E3 5B 5B 15 9E 8F EA
Derived Session Key: A1 00 11 56 78 15 15 85 2B 53 76 A9 18 14 AA F2
AC calculation: 00 00 00 00 20 00 00 00 00 00 00 00 00 80 00 80 00 08 26 17 01 03 00 B0 32 0F C0 79 00 00 08 A7 40 0F 04 00 00 80
Amount Authorised : 00 00 00 00 20 00
Amount Other : 00 00 00 00 00 00
Terminal Verification Results : 00 80 00 80 00
Transaction Currency Code : 08 26
Transaction Date : 17 01 03
Transaction Type : 00
Unpredictable Number : B0 32 0F C0
Application Interchange Profile : 79 00
Application Transaction Counter : 00 08
Card Verification Results : A7 40 0F 04 00 00
AC Session Key : A1 00 11 56 78 15 15 85 2B 53 76 A9 18 14 AA F2
CDA Signature Generation
Input data: E0 B8 C8 03 72 22 60 00 80 30 00 00 00 00 00 20 00 00 00 00 00 00 00 00 80 00 80 00 08 26 17 01 03 00 B0 32 0F C0 22 00 00 00 00 00 00 00 00 00 00 44 03 02 16 11 12 60 00 80 9F 27 01 80 9F 36 02 00 08 9F 10 12 02 10 A7 40 0F 04 00 00 00 00 00 00 00 00 00 00 00 FF
ICC dynamic number: 98 43 55 5A 0A C1 C2 4A
ICC private key: 1A C2 53 A6 2F FC 28 F2 CA 67 EE 9B 2C BE 16 C2 38 FB E3 C8 8B 28 4A 81 18 44 4B 6A BD 6F 68 FD F4 70 23 62 20 D1 4A 1A 11 6F E4 A8 5C 33 FE 1E 35 CD 9A 3F 48 44 13 64 A3 E9 50 58 ED 26 35 82 D3 6E FA 8E A4 EF EE A2 42 21 C5 4C 02 FB 5D C3 AE 17 97 8B D6 CE 6B 68 A2 4B 3B 13 C8 61 3A 2E 1E 0A 53 1B A1 71 AF 7E 1E FF 44 4B FF 72 50 03 89 F6 64 2F 0F 62 E4 9A 43 0C 6D F7 0C 07 EE 0D
ICC public modulus: A0 8D F5 E5 1F E8 F5 B0 BE 6F 97 A3 0C 74 88 8D 55 E7 56 B3 42 F1 BF 06 91 99 C4 80 70 9C 75 F3 BA A0 D4 4C C4 E7 BC 9C 68 9F 5B F2 29 37 F4 B5 42 D1 9D 7B B1 98 74 5B D7 77 E2 15 8E E5 41 12 8A 15 8E 73 6A 88 4B 82 C5 21 61 6E F0 6F 8D 26 7C 07 B1 EF 79 8A B5 77 AA A3 C6 DD 89 37 C9 B2 34 4C EC AD 5A B8 D5 29 BC AC A7 F9 EC EA DE 85 99 0F 1E 04 FE AE 9F A0 33 DF 69 12 68 F9 F2 D5
Terminal unpredictable number: B0 32 0F C0
Signature: 89 A6 C6 A0 AD 68 43 14 03 EE 4E 92 4B A8 CE B0 ED D9 F2 23 9A AB C9 90 D6 67 FD D5 B4 FF FC 98 99 AB 66 A7 10 0D 5B EB EE 36 7C 36 79 2D A2 A2 92 11 A2 0C 00 71 86 4B BE 20 BA 44 57 73 E5 0C 2D FB 17 AA DE 5C 85 8B 19 66 B8 F3 40 E0 00 EB BF 10 8B 1C AE 91 BD D0 DC 0C D3 D5 40 85 42 72 B0 E2 2F 30 D5 B5 EA 61 29 C9 9F 4F 39 F3 EE BC 66 06 F7 60 11 4D D6 DB 57 CF 57 F6 C1 EF 8C 35
Signed Dynamic Authentication Data
Evaluated: 6A 05 01 26 08 98 43 55 5A 0A C1 C2 4A 80 EC 19 0A DB E4 1C 90 B0 AA 00 A2 EC B5 50 A9 54 A9 92 00 1E CA 05 21 B1 DD 13 98 ED BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB 8E 73 0F 77 06 61 76 D4 6A 68 EF 7A 9C 45 23 0C 9A 62 43 C6 BC
RecoveredDataHeader '6A'h
DynamicApplicationData_for_signature '05 01 26 08 98 43 55 5A 0A C1 C2 4A 80 EC 19 0A DB E4 1C 90 B0 AA 00 A2 EC B5 50 A9 54 A9 92 00 1E CA 05 21 B1 DD 13 98 ED BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB'h structure
SignedDataFormat '05'h
HashAlgorithmIndicator '01'h
ICCDynamicDataLength 38
ICCDynamicData '08 98 43 55 5A 0A C1 C2 4A 80 EC 19 0A DB E4 1C 90 B0 AA 00 A2 EC B5 50 A9 54 A9 92 00 1E CA 05 21 B1 DD 13 98 ED'h
PadPattern 'BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB BB'h
HashResult '8E 73 0F 77 06 61 76 D4 6A 68 EF 7A 9C 45 23 0C 9A 62 43 C6'h
RecoveredDataTrailer 'BC'h
Expected: 6A 05 01 .. .. .. .. BC
RecoveredDataHeader '6A'h
DynamicApplicationData_for_signature 05 01 .. .. .. structure
SignedDataFormat '05'h
HashAlgorithmIndicator '01'h
ICCDynamicDataLength Length(ICCDynamicData)
ICCDynamicData .. concatenation
QS{0-}
''h
PadPattern .. concatenation
QS {..}
''h
HashResult .. concatenation
QS{1-}
''h
RecoveredDataTrailer 'BC'h
Previous Transaction History
Response: 61 A5
SW1 SW2: 61 A5 (SW_OK Response bytes available(Le))
Get Response
Request : 00 C0 00 00 A5
Class :00
Ins :C0
P1 :00
P2 :00
Le :A5
Response: C0 77 81 A2 9F 27 01 80 9F 36 02 00 08 9F 4B 81 80 89 A6 C6 A0 AD 68 43 14 03 EE 4E 92 4B A8 CE B0 ED D9 F2 23 9A AB C9 90 D6 67 FD D5 B4 FF FC 98 99 AB 66 A7 10 0D 5B EB EE 36 7C 36 79 2D A2 A2 92 11 A2 0C 00 71 86 4B BE 20 BA 44 57 73 E5 0C 2D FB 17 AA DE 5C 85 8B 19 66 B8 F3 40 E0 00 EB BF 10 8B 1C AE 91 BD D0 DC 0C D3 D5 40 85 42 72 B0 E2 2F 30 D5 B5 EA 61 29 C9 9F 4F 39 F3 EE BC 66 06 F7 60 11 4D D6 DB 57 CF 57 F6 C1 EF 8C 35 9F 10 12 02 10 A7 40 0F 04 00 00 00 00 00 00 00 00 00 00 00 FF 90 00
Ack Byte : C0
Data : 77 81 A2 9F 27 01 80 9F 36 02 00 08 9F 4B 81 80 89 A6 C6 A0 AD 68 43 14 03 EE 4E 92 4B A8 CE B0 ED D9 F2 23 9A AB C9 90 D6 67 FD D5 B4 FF FC 98 99 AB 66 A7 10 0D 5B EB EE 36 7C 36 79 2D A2 A2 92 11 A2 0C 00 71 86 4B BE 20 BA 44 57 73 E5 0C 2D FB 17 AA DE 5C 85 8B 19 66 B8 F3 40 E0 00 EB BF 10 8B 1C AE 91 BD D0 DC 0C D3 D5 40 85 42 72 B0 E2 2F 30 D5 B5 EA 61 29 C9 9F 4F 39 F3 EE BC 66 06 F7 60 11 4D D6 DB 57 CF 57 F6 C1 EF 8C 35 9F 10 12 02 10 A7 40 0F 04 00 00 00 00 00 00 00 00 00 00 00 FF
Tag 77 : Response Message Template Format 2
Tag 9F 27: Cryptogram Information Data (CID) : 80
Byte 1 bit 8-7 = 10 ARQC
bit 6-5 = 00 Payment System specific cryptogram
bit 4 = 0 No advice required
bit 3-1 = 000 No information given
Tag 9F 36: Application Transaction Counter (ATC) : 00 08
Decimal value = 8
Tag 9F 4B: Signed Dynamic Application Data : 89 A6 C6 A0 AD 68 43 14 03 EE 4E 92 4B A8 CE B0 ED D9 F2 23 9A AB C9 90 D6 67 FD D5 B4 FF FC 98 99 AB 66 A7 10 0D 5B EB EE 36 7C 36 79 2D A2 A2 92 11 A2 0C 00 71 86 4B BE 20 BA 44 57 73 E5 0C 2D FB 17 AA DE 5C 85 8B 19 66 B8 F3 40 E0 00 EB BF 10 8B 1C AE 91 BD D0 DC 0C D3 D5 40 85 42 72 B0 E2 2F 30 D5 B5 EA 61 29 C9 9F 4F 39 F3 EE BC 66 06 F7 60 11 4D D6 DB 57 CF 57 F6 C1 EF 8C 35
Tag 9F 10: Issuer Application Data [M/Chip 4] : 02 10 A7 40 0F 04 00 00 00 00 00 00 00 00 00 00 00 FF
Key Derivation Index = 02
Cryptogram Version Number = 10
Card Verification Results (CVR) = A7 40 0F 04 00 00
Byte 1 bit 8-7 = 10 AC Returned in Second Generate AC: Not requested
bit 6-5 = 10 AC Returned in First Generate AC: ARQC
bit 4 = 0 RFU
bit 3 = 1 Offline PIN Verification Performed
bit 2 = 1 Offline Encrypted PIN Verification Performed
bit 1 = 1 Offline PIN Verification Successful
Byte 2 bit 8 = 0 DDA not returned
bit 7 = 1 M/Chip Select 4: Combined DDA/AC Generation Returned in First Generate AC, M/Chip Lite 4: Value not allowed
bit 6 = 0 Combined DDA/AC Generation Not Returned in Second Generate AC
bit 5 = 0 Issuer Authentication not performed
bit 4 = 0 CIAC-Default not skipped on CAT3 or not required
bit 3 = 0 RFU
bit 2 = 0 RFU
bit 1 = 0 RFU
Byte 3 bit 8-5 = 0000 Right nibble of Script Counter : '0'
bit 4-1 = 1111 Right nibble of PIN Try Counter: 'F'
Byte 4 bit 8 = 0 RFU
bit 7 = 0 Unable To Go Online Not Indicated
bit 6 = 0 Offline PIN Verification Performed
bit 5 = 0 No Failure OF Offline PIN Verification
bit 4 = 0 PTL Not Exceeded
bit 3 = 1 International Transaction
bit 2 = 0 International Transaction
bit 1 = 0 Terminal Does Not Erroneously Consider Offline PIN OK
Byte 5 bit 8 = 0 Lower Consecutive Offline Limit Not Exceeded
bit 7 = 0 Upper Consecutive Offline Limit Not Exceeded
bit 6 = 0 Lower Cumulative Offline Limit Not Exceeded
bit 5 = 0 Upper Cumulative Offline Limit Not Exceeded
bit 4 = 0 Go Online On Next Transaction Was Not Set (in this transaction or in a previous one)
bit 3 = 0 No Issuer Authentication Failed (in this transaction or in a previous one)
bit 2 = 0 No Script Received (in a previous transaction)
bit 1 = 0 No Script Failed (in a previous transaction)
Byte 6 bit 8 = 0 RFU
bit 7 = 0 RFU
bit 6 = 0 RFU
bit 5 = 0 RFU
bit 4 = 0 RFU
bit 3 = 0 RFU
bit 2 = 0 No Match found in Additional Check Table
bit 1 = 0 Match Found in Additional Check Table
DAC = 00 00
Counters = 00 00 00 00 00 00 00 FF
SW1 SW2 : 90 00 (SW_OK)
2nd Generate AC (TC)
Request : 80 AE 50 00 23 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 59 33 00 80 00 80 00 B0 32 0F C0 98 43 55 5A 0A C1 C2 4A
Class :80
Ins :AE
P1 :50
P2 :00
Lc :23
Data :00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 59 33 00 80 00 80 00 B0 32 0F C0 98 43 55 5A 0A C1 C2 4A
Tag 91 : Issuer Authentication Data [M/Chip] : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Tag 8A : Authorization Response Code : 59 33
Response Code = 59 33
Meaning = Unable to go online, Requesting for offline approval
Tag 95 : Terminal Verification Results (TVR) : 00 80 00 80 00
Byte 1 bit 8 = 0 Offline data authentication was performed
bit 7 = 0 SDA passed or not performed
bit 6 = 0 No ICC data missing
bit 5 = 0 Card does not appear on terminal exception file
bit 4 = 0 DDA passed or not performed
bit 3 = 0 CDA passed or not performed
bit 2 = 0 SDA not selected
bit 1 = 0 RFU
Byte 2 bit 8 = 1 ICC and terminal have different application versions
bit 7 = 0 No Expired application
bit 6 = 0 Application effective
bit 5 = 0 Requested service allowed for card product
bit 4 = 0 No New card
bit 3 = 0 RFU
bit 2 = 0 RFU
bit 1 = 0 RFU
Byte 3 bit 8 = 0 Cardholder verification was successful or not performed
bit 7 = 0 Recognised CVM
bit 6 = 0 PIN Try Limit not exceeded
bit 5 = 0 No PIN entry required (PIN pad may or may not be present or may or may not be working)
bit 4 = 0 No PIN entry required (PIN pad may or may not be present)
bit 3 = 0 No Online PIN entered
bit 2 = 0 RFU
bit 1 = 0 RFU
Byte 4 bit 8 = 1 Transaction exceeds floor limit
bit 7 = 0 Lower consecutive offline limit not exceeded
bit 6 = 0 Upper consecutive offline limit not exceeded
bit 5 = 0 Transaction not selected randomly for online processing
bit 4 = 0 Merchant did not force transaction online
bit 3 = 0 RFU
bit 2 = 0 RFU
bit 1 = 0 RFU
Byte 5 bit 8 = 0 No Default TDOL used
bit 7 = 0 Issuer authentication passed or not performed
bit 6 = 0 Script processing passed before final GENERATE AC or no script received
bit 5 = 0 Script processing passed after final GENERATE AC or no script received
bit 4 = 0 RFU
bit 3 = 0 RFU
bit 2 = 0 RFU
bit 1 = 0 RFU
Tag 9F 37: Unpredictable Number : B0 32 0F C0
Tag 9F 4C: ICC Dynamic Number : 98 43 55 5A 0A C1 C2 4A
Previous Transaction History
Response: 69 85
SW1 SW2: 69 85 (SW_Error Accesscondition not satisfied)
Test Completed
It seems like you did a lot of "Offline approval" without clear counters.
There are several possible reasons in your case for 6985:
1) AC Session Key Counter Limit Exceeded
Because response code in Second Generate AC is "Unable to go online, Requesting for offline approval" card goes Offline, therefore CTR accumulates and is not reset (it reset only then Online in Generate AC 2).
Try to personalize increased limits or do full Online.
2) Overflow during the amount conversion, if transaction International
(transaction currency != CRM currency Code).
Required full Online
3) Overflow during add the cumulative amount, if transaction Domestic (transaction currency == CRM currency Code).
Required full Onlinе
also, check why "Country code" is absent in Cdol1
I got some help from my test tool's support system. Their response helped explain this scenario:
Response 1:
"The transaction amount in your test cases M-TIP 50 test1, test2, and test 3 are all above the floor limit, so the transactions are supposed to go online. However, due to your test environment is offline, the transactions can not be processed online. I can see the issuer authentication data in the 2nd generate AC are all 0s (They are not returned by the issuer, but the default value set by the terminal). The card validate the authentication data to see if the data is returned by the issuer. In your case, the validation is failed and the card returns "69 85"."
Response 2:
"The MTIP50 test cases are a little special. The issuer authentication data for MTIP50 test cases is 16 bytes which is different than the normal value (10 bytes). Thus BTT has extra checks on the issuer authentication data of MTIP50. Since the issuer authentication data are all zeros in your case, the simulated card thinks they are not valid."
Resolution:
Turns out this was an issue with the testing tool I was using. They recently released a new version of their software and this no longer happens.
For example: How would one decode the following ethernet frame?
00 26 b9 e8 7e f1 00 12 f2 21 da 00 08 00 45 00 05 dc e3 cd 20 10 35 06 25 eb 0a 0a 0a 02 c0 a8 01 03 c3 9e 0f 40 00 00 10 00 00 00 14 00 70 10 00 5c 59 99 00 00 02 04 05 b4 01 03 03 06 00 00 01 98 64 34 e8 90 84 98 20 12 18 19 04 85 80 00
I know that the first 6 bytes are the MAC destination address : 00 26 b9 e8 7e f1 The next 6 bytes are the source MAC address : 00 12 f2 21 da 00 The next 2 bytes show the ethernet type : 08 00 The next 4 bytes are : 45 00... Ipv4... "5" the number of bytes in the header.. and "00" means there are no differentiated services.
What I don't know is what anything after that is or how to read it.
Anyone help?
Rearranging a bit your packet, we have:
00 26 b9 e8 7e f1 00 12 f2 21 da 00 08 00 45 00
05 dc e3 cd 20 10 35 06 25 eb 0a 0a 0a 02 c0 a8
01 03 c3 9e 0f 40 00 00 10 00 00 00 14 00 70 10
00 5c 59 99 00 00 02 04 05 b4 01 03 03 06 00 00
01 98 64 34 e8 90 84 98 20 12 18 19 04 85 80 00
If you know that the first 6 octets form the destination mac address, that means that it is an Ethernet layer 2 packet.
According to IEEE 802.3, $3.1.1:
First 6 octets are the destination mac address (00 26 b9 e8 7e f1)
Next 6 octets are the source mac address (00 12 f2 21 da 00)
Next 4 octets are, optionally the 802.1Q tag (present, 08 00 45 00)
Next 2 octets are either:
Maximum payload size - aka MTU (if <= 1500, which is the case, 05 dc is 1500)
Ethernet 2 frame (if >= 1536)
Next is the payload ranging from 46 octets (if the 802.1Q tag is absent) or 42 octets (if the 802.1Q tag is present) to up to 1500 octets (starts at e3 cd 20 10 ..., ends either at 20 12 18 19 or at 03 06 00 00, depends on the 7th item)
Last 4 octets form the CRC32 code (either 01 98 64 34 or 04 85 80 00, depending on the 7th item)
There is also 12 octets used for padding (random - not so random - bytes), that may or may not be inserted in this packet. (if inserted, the padding is e8 90 84 98 20 12 18 19 04 85 80 00)
I am trying to write a tcp server to get the data from Heacent 908 GPS tracker. After establishing the connection from the tracker I am getting the following buffer output.
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 06 64 be 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 06 64 be 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 06 64 be 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 06 64 be 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 06 64 be 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 06 64 be 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 06 64 be 0d 0a>
I am not sure how to decode this data into proper readable format.
Note: Off course I have tried to reach the manufacture but they are not responding at all.
What type of possible encoding formats are there for TCP protocol?
On next day I got data like this
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 07 75 37 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 07 75 37 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 07 75 37 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 07 75 37 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 07 75 37 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 07 75 37 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 07 75 37 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 07 75 37 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 07 75 37 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a>
<Buffer 78 78 1f 12 0e 02 14 13 01 14 c8 03 5f a6 50 07 f7 f8 c1 32 35 39 01 9a 04 0f a2 00 b0 5a 00 1a 9b 7a 0d 0a>
<Buffer 78 78 1f 12 0e 02 14 13 01 1e c8 03 5f ad bc 07 f7 f0 76 41 35 40 01 9a 04 0f a2 00 b0 5a 00 1b b6 31 0d 0a>
Something is being changed but not sure what is it...
You ask what possible encoding formats there are for TCP. That's a bit of an odd question: there are an unbounded number of encoding formats using TCP as the underlying protocol. But no matter, we can try to figure out this one!
You've posted some sample messages. Let's see if we can translate them:
byte 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
rev 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
----------------------------------------------------------
hex 78 78 0d 01 03 87 11 31 20 86 48 42 00 06 64 be 0d 0a
text x x \r -- -- -- -- 1 -- H B -- -- d -- \r \n
dec 13 1 3 17 0 6 100 13 10
be32 [218170247] [288432262] [ 419006]
----------------------------------------------------------
hex 78 78 0d 01 03 87 11 31 20 86 48 42 00 07 75 37 0d 0a
text -- u 7
dec 7 117 55
be32 [ 488759]
----------------------------------------------------------
hex 78 78 0d 01 03 87 11 31 20 86 48 42 00 08 8d c0 0d 0a
text -- -- --
dec 8 141
be32 [ 560576]
----------------------------------------------------- byte 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
hex 78 78 1f 12 0e 02 14 13 01 14 c8 03 5f a6 50 07 f7 f8 c1 32 35 39 01 9a 04 0f a2 00 b0 5a 00 1a 9b 7a 0d 0a
text -- -- -- -- -- -- -- -- -- -- _ -- P -- -- -- -- 2 5 9 -- -- -- -- -- -- -- -- -- xx -- z \r \n
----------------------------------------------------------
hex 78 78 1f 12 0e 02 14 13 01 1e c8 03 5f ad bc 07 f7 f0 76 41 35 40 01 9a 04 0f a2 00 b0 5a 00 1b b6 31 0d 0a
text -- -- -- A 5 # -- xx -- 1
Some potentially interesting facts:
Starts with "xx\r\01" which more or less seems like a possible header. But later messages start with "xx" and something else. Anyway, given that NMEA has a prefix of "GP" I wouldn't be shocked if these devices used "xx" for "something that's not NMEA."
Has "HB" in the middle, which could mean "heartbeat" since this is repeating, perhaps waiting for a reply from the server.
Ends with "\r\n" which is a common line ending (on Windows in particular), though the rest doesn't appear to be entirely textual.
The earlier messages are 18 bytes long and the later ones 36 bytes. A guess would be the short ones are status updates or heartbeats and the long ones are actual location information. 36 bytes is enough if we figure:
4 byte latitude: 24 bits if you pinch (see), 25-32 bits more likely
4 byte longitude: same as latitude
6 byte timestamp: 39 bits if using epoch time with centiseconds, 32/48/64 bits more likely
2 byte altitude: I suspect this device doesn't publish altitude at all, given some of the docs
So I think what is going on is that these messages you see are just the device "pinging" the server and waiting for a response. What sort of response? Well, you could try to brute force it, but far, far easier would be to set up a bridge in your program that takes whatever it receives from the device, sends it to the manufacturer's server, and does the same thing in reverse for the responses to the device. This way you will quickly be able to gather a corpus of valid messages which will be very helpful if we really do need to reverse engineer this thing. Or if you're lucky it will turn out to use some standard protocol like NMEA after negotiating the initial session.
Edit: now that you've given us more messages from the device, we can see that it does seem to send something else with variable content. Maybe that's the location data, but I don't have time to try to reverse engineer it right now. One idea is to physically move the unit from west to east or north to south and capture the messages it sends during that time, to try to isolate which parts of the messages are the longitude and which are the latitude (and perhaps timestamp too).
I think it's fairly clear that the first two bytes are "xx" as a header, and the last two are "\r\n" as a terminator. That leaves 32 bytes of payload in the longer messages, all of which appears to be binary data.
It's the GT06 protocol and you can find it's specs here:
http://www.traccar.org/devices/
http://www.traccar.org/docs/protocol.jsp
https://dl.dropboxusercontent.com/s/sqtkulcj51zkria/GT06_GPS_Tracker_Communication_Protocol_v1.8.1.pdf
You can do it this way:
client.on('data', (buffer) => {
const decodedData = buffer.toString('utf8')
console.log(decodedData)
})
I got some ADTS AAC raw data from somewhere(actually it is extracted from a demuxed file) and in theory it should be corrected encoded. it looks like this:
Frame1:
21 19 94 ED A1 09 45 58 09
40 02 CA AA 85 D4 E5 C5 58 A9 73 00 0C 75 1C 5D
A7 4E 52 40 90 38 71 9C 65 D5 C4 22 0B 28 7D EF
F8 42 33 15 03 BA 6C DE B1 74 B4 A1 4E 0A 21 05
15 34 6B FD D9 E7 8F BF FF 79 5C D3 7D 90 79 F6
65 57 08 3A F7 C5 14 85 5E D7 C3 7D 2A 85 E1 7A
86 BA 3A AC 13 0D AE D1 1B 65 69 B6 71 92 E5 8A
BC CB 5C 7A 6F D7 F2 2B 38 C9 0E 2A 40 2F 8E 90
9B 1F A2 3A 9C 39 A8 35 CE 69 14 CD 64 54 70 00
50 07 CE 37 83 6E F0 01 18 AA A8 49 B2 8B 8F A1
37 17 1C 06 00 00 00 06 00 72
Frames2:
21 19 95 14 C2 0A A9 61 19 8B CB 9B 56 AE A7
0A A0 34 DA EA D9 34 28 0C F8 DC 0C 30 97 12 A7
DD 3F F5 FE 7B 65 52 61 6D 7F DA BE D3 EB 30 CA
A6 94 54 8E D4 0A 32 E1 EA FD AD 02 82 B5 1E 40
4C 04 3A BE 56 21 5D 7D 5D B3 31 2A 5D AF 4E FF
A6 48 B9 42 E3 87 DE 5C 59 4B B9 BB C3 2C AD 50
6B 35 C8 24 6C 06 82 86 B2 26 17 E2 C6 DD 9A 43
53 91 D3 68 8D 67 8E 7D 0A 28 EB 7D F1 BB FC 56
5E 13 25 F9 77 E6 27 BF DA 4E 09 38 86 20 0A 00
F9 C6 F0 1D DE 00 21 05 4F 28 C0 A0 5F 0E 18 00
03 00 0E
.....
And for each following frames there is a quite strange similiar header as:
21 19 xx xx
For examples:
21 19 94 E1 ..
21 19 95 03 ..
....
So do you know what does this header mean?
This is how ADTS AAC looks like, for example for stereo:
adts_header()
channel_pair_element()
adts_header()
channel_pair_element()
adts_header()
channel_pair_element()
adts_header()
channel_pair_element()
etc...
This seems like it's not ADTS header at all. ADTS header is typicaly not used in some other container, like mp4, but is used for standalone AAC files only. ADTS header starts with 12 bit syncword 1111 1111 1111. So all ones, and this is not the case in your example.
In case muxer stripped out any header there was, you might have raw AAC which should start with single_channel_element() in case of mono or channel_pair_element() in case of stereo.
single_channel_element() starts with 3 bits 000
cannel_pair_element() starts with 3 bits 001
Your sample starts with 0010 0001 0001 1001 so it might be channel_pair_element().
You probably have stereo but without any header, like so:
channel_pair_element()
channel_pair_element()
channel_pair_element()
channel_pair_element()
etc.
You should ask the muxer to tell you the number of channels, sampling rate, etc, and you are ready to continue decoding. Muxer should grab this info from mp4 or whichever container your AAC was originaly in.
It most likely a mpeg4 latm format. if you run mediainfo tool to check, it will output as below:
$mediainfo a.aac
General
Complete name : a.aac
Format : LATM
File size : 821 KiB
Overall bit rate mode : Variable
Audio
Format : AAC
Format/Info : Advanced Audio Codec
Format profile : HE-AACv2 / HE-AAC / LC
Bit rate mode : Variable
Channel(s) : 2 channels / 1 channel / 1 channel
Channel positions : Front: L R / Front: C / Front: C
Sampling rate : 48.0 KHz / 48.0 KHz / 24.0 KHz
Compression mode : Lossy
Such format usually generated after ADTS header removed or from DTV channel. DTV data transfer use LATM format to save bandwidth, so no ADTS header there but use some codec config buffer to initialize decoder.