Read JPEG magic number with FileChannel and ByteBuffer - jpeg

I started digging into Java NIO API and as a first try I wanted to read a JPEG file magic number.
Here's the code
import java.nio.ByteBuffer;
import java.nio.CharBuffer;
import java.nio.channels.FileChannel;
import java.nio.charset.Charset;
import java.io.FileInputStream;
public class JpegMagicNumber {
public static void main(String[] args) throws Exception {
FileChannel file = new FileInputStream(args[0]).getChannel();
ByteBuffer buffer = ByteBuffer.allocate(6);
file.read(buffer);
buffer.flip();
System.out.println(Charset.defaultCharset().decode(buffer).toString());
file.close();
buffer.clear();
}
}
I expect to get the magic number chars back but all I get is garbage data into the terminal.
Am I doing something wrong ?

Short answer: There is nothing particularly defective with the code. JPEG just has 'garbage' up front.
Long answer: JPEG internally is made up of segments, one after the other. These segments start with a 0xFF byte, followed by an identifier byte, and then an optional payload/content.
Example start:
FF D8 FF E0 00 10 4A 46 49 46 00 01 01 00 00 01 00 01 00 00 FF E1
The image starts with the Start Of Image (SOI) segment, 0xFF 0xD8, which has no payload.
The next segment is 'application specific', 0xFF 0xE0. Two bytes follow with the length of the payload (these two bytes included!).
0x4A 0x46 0x49 0x46 : JFIF ← perhaps what you were looking for?
JPEG doesn't have a magic number in the sense you were perhaps looking for, like 'PK' for zip or '‰PNG' for PNG. (The closest thing is 0xFF 0xD8 0xFF for the SOI and the first byte of the next segment.)
So your code does correctly read the first six bytes of the file, decodes them into characters per your native platform, and prints them out, but a JPEG header just looks that way.

Related

bytes to string conversion with invalid characters

I need to parse UDP packets which can be invalid or contain some errors. I would like to replace invalid characters with . after a bytes to string conversion, in order to display the content of the packets.
How can I do it? This is my code:
func main() {
a := []byte{'a', 0xff, 0xaf, 'b', 0xbf}
s := string(a)
s = strings.Replace(s, string(0xFFFD), ".", 0)
fmt.Println("s: ", s) // I would like to display "a..b."
for _, r := range s {
fmt.Println("r: ", r)
}
rs := []rune(s)
fmt.Println("rs: ", rs)
}
The root problem with your approach is that the result of type converting []byte to string does not have any U+FFFDs in it: this type-conversion only copies bytes from the source to the destination, verbatim.
Just as byte slices, strings in Go are not obliged to contain UTF-8-encoded text; they can contain any data, including opaque binary data which has nothing to do with text.
But some operations on strings—namely type-converting them to []rune and iterating over them using range—do interpret strings as UTF-8-encoded text.
That is precisely where you got tripped: your range debugging loop attempted to interpret the string, and each time another attempt at decoding a properly encoded code point failed, range yielded a replacement character, U+FFFD.
To reiterate, the string obtained by the type-conversion does not contain the characters you wanted to get replaced by your regexp.
As to how to actually make a valid UTF-8-encoded string out of your data, you might employ a two-step process:
Type-convert your byte slice to a string—as you already do.
Use any means of interpreting a string as UTF-8—replacing U+FFFD which will dynamically appear during this process—as you're iterating.
Something like this:
var sb strings.Builder
for _, c := range string(b) {
if c == '\uFFFD' {
sb.WriteByte('.')
} else {
sb.WriteRune(c)
}
}
return sb.String()
A note on performance: since type-converting a []byte to string copies memory—because strings are immutable while slices are not—the first step with type-conversion might be a waste of resources for code dealing with large chunks of data and/or working in tight processing loops.
In this case, it may be worth using the DecodeRune function of the encoding/utf8 package which works on byte slices.
An example from its docs can be easily adapted to work with the loop above.
See also: Remove invalid UTF-8 characters from a string
#kostix answer is correct and explains very clearly the issue with scanning unicode runes from a string.
Just adding the following remark : if your intention is to view characters only in the ASCII range (printable characters < 127) and you don't really care about other unicode code points, you can be more blunt :
// create a byte slice with the same byte length as s
var bs = make([]byte, len(s))
// scan s byte by byte :
for i := 0; i < len(s); i++ {
switch {
case 32 <= s[i] && s[i] <= 126:
bs[i] = s[i]
// depending on your needs, you may also keep characters in the 0..31 range,
// like 'tab' (9), 'linefeed' (10) or 'carriage return' (13) :
// case s[i] == 9, s[i] == 10, s[i] == 13:
// bs[i] = s[i]
default:
bs[i] = '.'
}
}
fmt.Printf("rs: %s\n", bs)
playground
This function will give you something close to the "text" part of hexdump -C.
You may want to use strings.ToValidUTF8() for this:
ToValidUTF8 returns a copy of the string s with each run of invalid UTF-8 byte sequences replaced by the replacement string, which may be empty.
It "seemingly" does exactly what you need. Testing it:
a := []byte{'a', 0xff, 0xaf, 'b', 0xbf}
s := strings.ToValidUTF8(string(a), ".")
fmt.Println(s)
Output (try it on the Go Playground):
a.b.
I wrote "seemingly" because as you can see, there's a single dot between a and b: because there may be 2 bytes, but a single invalid sequence.
Note that you may avoid the []byte => string conversion, because there's a bytes.ToValidUTF8() equivalent that operates on and returns a []byte:
a := []byte{'a', 0xff, 0xaf, 'b', 0xbf}
a = bytes.ToValidUTF8(a, []byte{'.'})
fmt.Println(string(a))
Output will be the same. Try this one on the Go Playground.
If it bothers you that multiple (invalid sequence) bytes may be shrinked into a single dot, read on.
Also note that to inspect arbitrary byte slices that may or may not contain texts, you may simply use hex.Dump() which generates an output like this:
a := []byte{'a', 0xff, 0xaf, 'b', 0xbf}
fmt.Println(hex.Dump(a))
Output:
00000000 61 ff af 62 bf |a..b.|
There's your expected output a..b. with other (useful) data like the hex offset and hex representation of bytes.
To get a "better" picture of the output, try it with a little longer input:
a = []byte{'a', 0xff, 0xaf, 'b', 0xbf, 50: 0xff}
fmt.Println(hex.Dump(a))
00000000 61 ff af 62 bf 00 00 00 00 00 00 00 00 00 00 00 |a..b............|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 ff |...|
Try it on the Go Playground.

Why does Node.js shorten my buffer when writing to file?

I'm experiencing some weird behavior in NodeJS v14.16.1 on Windows 10.
When I append 4 bytes to a file, it's only appending 2 bytes:
const fs = require('fs');
let infoFile = fs.openSync('field.info', 'a');
fs.writeSync(infoFile, Buffer.from([0x06, 0x00, 0x04, 0x00]))
fs.closeSync(infoFile);
Output for field.info:
06 04
Expected output:
06 00 04 00
Buffer.from([0x06, 0x00, 0x04, 0x00, 0x00]) produces the expected output:
06 00 04 00 00
The .toString() on the Buffer's produce strings of the correct length, so I can only imagine it's an issue in fs.writeSync. I can't find anything in the documentation that would indicate the issue...
Edit
fs.writeSync reports 4 bytes written but only 2 exist in the file.
This is most certainly due to the editor you opening the file with, interpreting it as UTF-16. Thus, you see only two characters, because each character is representing two bytes. Showing the file properties in windows explorer clearly shows the correct file size.

Java Card have a weird response to APDU with INS = 0x70

Below, you see a simple applet that returns 0x6781 to incoming APDU commands with INS=0x70 or INS=0x71:
package testPack;
import javacard.framework.*;
public class TestApp extends Applet
{
public static void install(byte[] bArray, short bOffset, byte bLength)
{
new TestApp().register(bArray, (short) (bOffset + 1), bArray[bOffset]);
}
public void process(APDU apdu)
{
if (selectingApplet())
{
return;
}
byte[] buf = apdu.getBuffer();
switch (buf[ISO7816.OFFSET_INS])
{
case (byte)0x70:
ISOException.throwIt((short)0x6781);
break;
case (byte)0x71:
ISOException.throwIt((short)0x6781);
break;
default:
ISOException.throwIt(ISO7816.SW_INS_NOT_SUPPORTED);
}
}
}
The problem is that, I receive 0x6C01 to the APDU command with INS=0x70:
Send: 00 A4 04 00 07 01 02 03 04 05 00 00 00
Recv: 90 00
Send: 00 70 00 00 00
Recv: 6C 01
Send: 00 70 00 00 01
Recv: 01 90 00
Send: 00 71 00 00 00
Recv: 67 81
I tried two different Java Cards (One is NXP JCOP v2.4.2 r3 and another is a KONA java card) through both contact and contactless interfaces and using two different cap file generated inside two different laptops via two different IDE!!!( How Suspicious am I? :D ) But the response is equal.
I suspect to the PCSC or Card Manager for this weird response. Because in the simulator, even the process method doesn't called for this special INS value.
What's wrong with it?
INS = 70with CLA = 00 is the MANAGE CHANNEL command according to the ISO-7816 specification, as well as INS = A4 means SELECT.
If you want to use these INS codes, you must use CLA >= 0x80 in order to specify that it is your proprietary command.
I think if class represent interindustry class then only INS will work as defined in standard,Here CLA - 00 represent interindusty command therefore, card response behvaiour was like same behaviour as Manage channel command because you used INS = 70.
6.16.4 Response message (nominal case)
Table 73 - MANAGE CHANNEL response APDU
Data field Logical channel number if P1-P2='0000'
Empty if P1-P2!='0000'
SW1-SW2 Status bytes
actually your card was returning logical channel no -01 to you. ManageChannel
it seems to me that if class is propriatry. here INS will not treated as defined INS in standard. Hope CLA 80 with same INS 0x70 will give you require result.
hope it helps.
[Bit 8 set to 1 indicates the proprietary class]

What is the structure of an application protocol data unit (APDU) command and response? [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
I am trying to learn Java Card. I have just started and have not found many resources. My first question is how to understand APDU commands. (such as those defined in ISO/IEC 7816-4)
For example, if I see a byte pattern such as 10101010 how can I understand the meaning of this, and in particular determine the CLA or INS, for example?
APDU commands are a queue of binary numbers in the following form:
CLA | INS | P1 | P2 | Lc | CData | Le
The first four sections, i.e CLA , INS , P1 and P2 are mandatory in all APDU commands and each one has one byte length. These one-byte-length sections stand for Class, Instruction, Parameter1 and Parameter2 respectively.
The last three sections, i.e Lc , CData and Le are optional.Lc is the encoding of Nc, which is the encoding of the length of the CDATA field. Le is the encoding of Ne, then encoding of the maximum response data that may be send. Based on presence or absence of these sections, we have 4 case for APDU commands, as below:
Case1: CLA | INS | P1 | P2
Case2: CLA | INS | P1 | P2 | Le
Case3: CLA | INS | P1 | P2 | Lc | Data
Case4: CLA | INS | P1 | P2 | Lc | Data | Le
The length of CData is different for different commands and different applets. based on the length of CData (i.e Lc) and the length of maximum response data that may send (i.e Le), we have to type of APDU commands:
Normal/Short APDU commands, when Lc and Le are smaller than 0xFF
Extended length APDU commands, when Lc and/or Le are greater than 0xFF.
So for the length of these sections we have:
Lc : 1 byte for Short APDU commands and 3 byte (they specify this length, because its enough) for Extended APDU commands.
Data : Different lengths.
Le : Same as Lc.
How can I understand APDU commands?
Answer:
When you write an applet, you specify the response of your applet to different APDU commands that it will receive in the future. Card Manager is an applet also. The commands that it support is defined in your card's specifications/datasheet. Normally almost all cards are GlobalPlatform and ISO7816 compliant, so they must support those mandatory APDU commands that is defined in these documents. For example, as 0xA4 is defined as SELECT FILE command in ISO7816-4 standard, If you see an APDU like xx A4 xx xx is sending to Card Manager, you can conclude that it is related with SELECT FILE.
Note that you can choose one value for different functions in your different applets. For example in the following, Applet1 will return 0x6990 in the reception of 00 B0 xx xx APDU commands, while Applet2 will return 0x6991 in the reception of the same command:
Applet1:
public class SOQ extends Applet {
private SOQ() {
}
public static void install(byte bArray[], short bOffset, byte bLength)
throws ISOException {
new SOQ().register();
}
public void process(APDU arg0) throws ISOException {
byte buffer[] = arg0.getBuffer();
if(buffer[ISO7816.OFFSET_CLA] == (byte) 0x00 &&buffer[ISO7816.OFFSET_INS] == (byte) 0xB0){
ISOException.throwIt((short)0x6990);
}
}
}
Output:
OpenSC: opensc-tool.exe -s 00a404000b0102030405060708090000 -s 00B00000 -s 00B00
100
Using reader with a card: ACS CCID USB Reader 0
Sending: 00 A4 04 00 0B 01 02 03 04 05 06 07 08 09 00 00
Received (SW1=0x90, SW2=0x90)
Sending: 00 B0 00 00
Received (SW1=0x69, SW2=0x90)
Sending: 00 B0 01 00
Received (SW1=0x69, SW2=0x90)
Applet2:
public class SOQ extends Applet {
private SOQ() {
}
public static void install(byte bArray[], short bOffset, byte bLength)
throws ISOException {
new SOQ().register();
}
public void process(APDU arg0) throws ISOException {
byte buffer[] = arg0.getBuffer();
if(buffer[ISO7816.OFFSET_CLA] == (byte) 0x00 && buffer[ISO7816.OFFSET_INS] == (byte) 0xB0){
ISOException.throwIt((short)0x6991);
}
}
}
Output:
OpenSC: opensc-tool.exe -s 00a404000b0102030405060708090000 -s 00B00000 -s 00B00
100
Using reader with a card: ACS CCID USB Reader 0
Sending: 00 A4 04 00 0B 01 02 03 04 05 06 07 08 09 00 00
Received (SW1=0x90, SW2=0x00)
Sending: 00 B0 00 00
Received (SW1=0x69, SW2=0x91)
Sending: 00 B0 01 00
Received (SW1=0x69, SW2=0x91)
So the final and short answer to your question (How can I understand APDU commands?) is:
You are dealing with your applet?
You defined the supported commands and their forms, yourself!
You are dealing with another applet (Card Manager, for example)?
You need the source code of that applet or its documentation about its supported commands and their forms or the standard/specification that that applet is compliant with (Global Platform for Card Managers for example).
Note: we have almost the same for APDU responses.
I am afraid such a "complete" e-book simply does not exist. Honestly, I think it is not necessary at all. If you know the basic Java syntax, you will find JavaCard quite easy to learn (although annoying to use). All the usual difficult stuff (threading, GUI, IO, annotations, templates, databases, ...) is just missing in Javacard and the standard libraries are so limited you will be able to learn them in a few days.
There are a few nice tutorials out there:
http://www.oracle.com/technetwork/java/embedded/javacard/overview/index.html
http://javacard.vetilles.com/tutorial/
and a very good SO question:
How to get started with Java Cards?
To answer your question: JavaCard is just a language for writing smartcard applications called applets. It handles all the application logic, but it does not specify the APDU format. That is because JavaCard is not the only smartcard technology. APDU format is specifed in ISO7816 standard, which I really recommend you to read through. It is not free to download, but you can find the most important parts here:
http://www.cardwerk.com/smartcards/smartcard_standard_ISO7816-4_5_basic_organizations.aspx
You would find there, that your APDU command consists of a header:
00A404000E
and a data part:
63616C63756C61746F722E617070.
The header specifies what function should be called:
00 - class byte (CLA, 00 means "inter-industry command sent to logical channel 0")
A4 - instruction byte (INS, A4 means "SELECT applet command")
04 - parameter 1 (P1)
00 - parameter 2 (P2)
0E - length of the data part (Lc)
and the data part contains the identifier of the applet which should be selected for future usage (in your case it is an ASCII encoded string "calculator.app" btw).

Why do I get nonstandard responses from the TPM Through TBS?

I have a C++ program to do a basic TPM_GetCapabilities Through TPM Base Services and the Windows 7 SDK.
I've setup the program below
int _tmain(int argc, _TCHAR* argv[])
{
TBS_CONTEXT_PARAMS pContextParams;
TBS_HCONTEXT hContext;
TBS_RESULT rv;
pContextParams.version = TBS_CONTEXT_VERSION_ONE;
rv = Tbsi_Context_Create(&pContextParams, &hContext);
printf("\n1 RESULT : %x STATUS : %x", rv, hContext);
BYTE data[200] =
{0,0xc1, /* TPM_TAG_RQU_COMMAND */
0,0,0,18, /* blob length, bytes */
0,0,0,0x65, /* TPM_ORD_GetCapability */
0,0,0,0x06, /* TPM_CAP_VERSION */
0,0,0,0}; /* 0 bytes subcap */
BYTE buf[4000];
UINT32 len = 4000;
rv = Tbsip_Submit_Command(hContext,0,TBS_COMMAND_PRIORITY_NORMAL,data,18,buf,&len);
//CAPABILITY_RETURN* retVal = new CAPABILITY_RETURN(buf);
//printf("\n2 Response Tag: %x Output Bytes: %x",tag,);
printf("\n2 RESULT : %x STATUS : %x\n", rv, hContext);
printBuf(buf,len);
rv = Tbsip_Context_Close(hContext);
printf("\n3 RESULT : %x STATUS : %x", rv, hContext);
My Return Buffer looks like:
00:C4:00:00:00:12:00:00:00:00:00:00:00:04:01:01:00:00
According to this doc, Section 7.1 TPM_GetCapability I should get the following:
Looking at my output buffer, I am getting TPM_TAG_RSP_COMMAND,a value of 18 for my paramSize, 0 for my TPM_RESULT, 0x...04 for ordinal (Not sure what this is supposed to mean.) then 1,1,0,0 for my final bits. I'm at a loss as to how to decipher this.
The answer to your question:
You don't get a nonstandard response.
The response is perfectly fine, there is nothing nonstandard in it. It looks exactly like it is defined in the spec.
The response' content resp you get is also what is to be expected. A Standard conform TPM has to answer with 01 01 00 00 when asked for TPM_CAP_VERSION.
Why?
First of all: The line stating TPM_COMMAND_CODE ordinal is not part of the response.
It has no PARAM # and no PARAM SZ. It is only relevant for calculating the HMAC of the response.
So the response is the following:
00 C4 tag
00 00 00 12 paramSize
00 00 00 00 returnCode
00 00 00 04 respSize
01 01 00 00 resp
You asked for the capability TPM_CAP_VERSION. Here is what the spec says:
Value: 0x00000006
Capability Name: TPM_CAP_VERSION
Sub cap: Ignored
TPM_STRUCT_VER structure.
The major and minor version MUST indicate 1.1.
The firmware revision MUST indicate 0.0.
The use of this value is deprecated, new software SHOULD
use TPM_CAP_VERSION_VAL to obtain version and revision information
regarding the TPM.
So when you decode resp, which is a TPM_STRUCT_VER, you get the following:
typedef struct tdTPM_STRUCT_VER {
BYTE major; // ==> 1
BYTE minor; // ==> 1
BYTE revMajor; // ==> 0
BYTE revMinor; // ==> 0
} TPM_STRUCT_VER;
So 1.1 and 0.0, exactly according to specification.

Resources