Can anyone answer how BCD data is usually calculated for fields what have subfield values?
I don't mean in terms of code, as I have that part nailed down.
What I mean is say I have field X, which is to be sent containing data for 5 sub values. The field is BCD, but would each sub-value be converted to BCD and then appended to field X or would they be added to field X in the clear and then converted as a whole to BCD?
Can't find a clear answer anywhere... not even in the message spec I'm working off of oO
Cheers,
Mike K
You have to know the format of field X and the format of subfields.
Let me give you an example.
Assuming that you would like to transmit emv data form pos to Host using a field X.
A format for field X is described below.
Length Attribute 3 bytes LLL Length of data to follow
Subfield 1 var bytes First Additional subfield
Subfield 2 var bytes Second Additional subfield
...
Subfield n var bytes nth Additional subfield
The structure of each additional subfield is as follows
Tag Name 2 bytes
Tag Length 1 byte
Tag Value ..bytes
If a field contains subfields, then every subfield is packed or unpacked with its own format.
The subfields should then not be packed or unpacked again.
If a field contains subfields, it is unnecessary to define the format of the field body. However, the field header format (tag or length) can be defined.
The following example contains a field with three subfields
Message Structure:
<f type="VAL" name="Parent" len="21">
<f type="VAL" name="Child1" bodyPacker="BcdBodyPacker" len="6"/>
<f type="VAL" name="Child2" bodyPacker="BcdBodyPacker" len="7"/>
<f type="VAL" name="Child3" bodyPacker="BcdBodyPacker" len="8"/>
</f>
Message data:
<f name="Parent">
<f name="Child1" val="111111111111"/>
<f name="Child2" val="22222222222222"/>
<f name="Child3" val="3333333333333333"/>
</f>
Message bytes in hex:
111111111111222222222222223333333333333333
The source code of the example can be found on GitHub
The iso-8583-packer Java library was used for creation of this example. I am the author of the library.
Related
I'm importing some JSON files into my Parse.com project, and I keep getting the error "invalid key:value pair".
It states that there is an unexpected "8".
Here's an example of my JSON:
}
"Manufacturer":"Manufacturer",
"Model":"THIS IS A STRING",
"Description":"",
"ItemNumber":"Number12345",
"UPC":083456789012,
"Cost":"$0.00",
"DealerPrice":" $0.00 ",
"MSRP":" $0.00 ",
}
If I update the JSON by either removing the 0 from "UPC":083456789012, or converting it to "UPC":"083456789012", it becomes valid.
Can JSON really not accept an integer that begins with 0, or is there a way around the problem?
A leading 0 indicates an octal number in JavaScript. An octal number cannot contain an 8; therefore, that number is invalid.
Moreover, JSON doesn't (officially) support octal numbers, so formally the JSON is invalid, even if the number would not contain an 8. Some parsers do support it though, which may lead to some confusion. Other parsers will recognize it as an invalid sequence and will throw an error, although the exact explanation they give may differ.
Solution: If you have a number, don't ever store it with leading zeroes. If you have a value that needs to have a leading zero, don't treat it as a number, but as a string. Store it with quotes around it.
In this case, you've got a UPC which needs to be 12 digits long and may contain leading zeroes. I think the best way to store it is as a string.
It is debatable, though. If you treat it as a barcode, seeing the leading 0 as an integral part of it, then string makes sense. Other types of barcodes can even contain alphabetic characters.
On the other hand. A UPC is a number, and the fact that it's left-padded with zeroes to 12 digits could be seen as a display property. Actually, if you left-pad it to 13 digits by adding an extra 0, you've got an EAN code, because EAN is a superset of UPC.
If you have a monetary amount, you might display it as € 7.30, while you store it as 7.3, so it could also make sense to store a product code as a number.
But that decision is up to you. I can only advice you to use a string, which is my personal preference for these codes, and if you choose a number, then you'll have to remove the 0 to make it work.
One of the more confusing parts of JavaScript is that if a number starts with a 0 that isn't immediately followed by a ., it represents an octal, not a decimal.
JSON borrows from JavaScript syntax but avoids confusing features, so simply bans numbers with leading zeros (unless then are followed by a .) outright.
Even if this wasn't the case, there would be no reason to expect the 0 to still be in the number when it was parsed since 02 and 2 are just difference representations of the same number (if you force decimal).
If the leading zero is important to your data, then you probably have a string and not a number.
"UPC":"083456789012"
A product code is an identifier, not something you do maths with. It should be a string.
Formally, it is because JSON uses DecimalIntegerLiteral in its JSONNumber production:
JSONNumber ::
-_opt DecimalIntegerLiteral JSONFraction_opt ExponentPart_opt
And DecimalIntegerLiteral may only start with 0 if it is 0:
DecimalIntegerLiteral ::
0
NonZeroDigit DecimalDigits_opt
The rationale behind is is probably:
In the JSON Grammar - to reuse constructs from the main ECMAScript grammar.
In the main ECMAScript grammar - to make it easier to distinguish DecimalIntegerLiteral from HexIntegerLiteral and OctalIntegerLiteral. OctalIntegerLiteral in the first place.
See this productions:
HexIntegerLiteral ::
0x HexDigit
0X HexDigit
HexIntegerLiteral HexDigit
...
OctalIntegerLiteral ::
0 OctalDigit
OctalIntegerLiteral OctalDigit
The UPC should be in string format. For the future you may also get other type of UPC such as GS128 or string based product identification codes. Set your DB column to be string.
If an integer start with 0 in JavaScript it is considered to be the Octal (base 8) value of the integer instead of the decimal (base 10) value. For example:
var a = 065; //Octal Value
var b = 53; //Decimal Value
a == b; //true
I think the easiest way to send your number by JSON is send your number as string.
I am using the python3 struct module to unpack byte data I extracted from a serial com. (With help) I've figured out how to unpack most the data into human readable form. I am having difficult with the format string on a group header struct group_hdr (please see attached screenshot document). I have a byte data (b). I know the character string for "word" is "H" but it's unclear to me from the document what phd_status is. It hasn't been defined anywhere else in the Data structure document. Any ideas?. Thank you in advance.
struct group_hdr
{
union phdb_status status
word label
}
subrecord = struct.unpack_from('<??H', b)
As is explained under Status, it is a simple bitfield with a width of 32 bits. The union is probably defined elsewhere in C (or a similar language) as
union phdb_status {
unsigned int bit_0:1;
unsigned int bit_1:1;
};
The following Python code will store your values:
status, label = struct.unpack_from('<IH', b)
and you can test the individual bits of status with status & 1 and status & 2.
What is the difference between string and character class in MATLAB?
a = 'AX'; % This is a character.
b = string(a) % This is a string.
The documentation suggests:
There are two ways to represent text in MATLAB®. You can store text in character arrays. A typical use is to store short pieces of text as character vectors. And starting in Release 2016b, you can also store multiple pieces of text in string arrays. String arrays provide a set of functions for working with text as data.
This is how the two representations differ:
Element access. To represent char vectors of different length, one had to use cell arrays, e.g. ch = {'a', 'ab', 'abc'}. With strings, they can be created in actual arrays: str = [string('a'), string('ab'), string('abc')].
However, to index characters in a string array directly, the curly bracket notation has to be used:
str{3}(2) % == 'b'
Memory use. Chars use exactly two bytes per character. strings have overhead:
a = 'abc'
b = string('abc')
whos a b
returns
Name Size Bytes Class Attributes
a 1x3 6 char
b 1x1 132 string
The best place to start for understanding the difference is the documentation. The key difference, as stated there:
A character array is a sequence of characters, just as a numeric array is a sequence of numbers. A typical use is to store short pieces of text as character vectors, such as c = 'Hello World';.
A string array is a container for pieces of text. String arrays provide a set of functions for working with text as data. To convert text to string arrays, use the string function.
Here are a few more key points about their differences:
They are different classes (i.e. types): char versus string. As such they will have different sets of methods defined for each. Think about what sort of operations you want to do on your text, then choose the one that best supports those.
Since a string is a container class, be mindful of how its size differs from an equivalent character array representation. Using your example:
>> a = 'AX'; % This is a character.
>> b = string(a) % This is a string.
>> whos
Name Size Bytes Class Attributes
a 1x2 4 char
b 1x1 134 string
Notice that the string container lists its size as 1x1 (and takes up more bytes in memory) while the character array is, as its name implies, a 1x2 array of characters.
They can't always be used interchangeably, and you may need to convert between the two for certain operations. For example, string objects can't be used as dynamic field names for structure indexing:
>> s = struct('a', 1);
>> name = string('a');
>> s.(name)
Argument to dynamic structure reference must evaluate to a valid field name.
>> s.(char(name))
ans =
1
Strings do have a bit of overhead, but still increase by 2 bytes per character. After every 8 characters it increases the size of the variable. The red line is y=2x+127.
figure is created using:
v=[];N=100;
for ct = 1:N
s=char(randi([0 255],[1,ct]));
s=string(s);
a=whos('s');v(ct)=a.bytes;
end
figure(1);clf
plot(v)
xlabel('# characters')
ylabel('# bytes')
p=polyfit(1:N,v,1);
hold on
plot([0,N],[127,2*N+127],'r')
hold off
One important practical thing to note is, that strings and chars behave differently when interacting with square brackets. This can be especially confusing when coming from python. consider following example:
>>['asdf' '123']
ans =
'asdf123'
>> ["asdf" "123"]
ans =
1×2 string array
"asdf" "123"
For example, if I define a photo like this:
$cat 30.proto
message hello
{
required int32 f1=1;
required int32 f2=2;
optional int32 f3=3;
}
Then I would double if protobuf can handle things like this:
I declare 3 objects, each don't have f3 field.
Write to output
Then, in reader side, how does reader know that these 6 values should belong to 3 objects(each 2 fields), or belong to 2 objects(each 3 fields)?
In another word, how does the "require"/"optional" reflected inside encoded bytes? If not reflected in the byte stream, then how does protobuf determine the start of a new offset? We know protobuf don't have "delimiter" bits.
I had a simple quick test on this:
$cat 30.cpp
#include "30.pb.h"
#include<fstream>
using namespace std;
int main()
{
fstream f("./log30.data",ios::binary|ios::out);
hello p1,p2,p3,p4,p5;
p1.set_f1(1);
p1.set_f2(2);
p2.set_f1(3);
p2.set_f2(4);
p3.set_f1(5);
p3.set_f2(6);
p1.SerializeToOstream(&f);
p2.SerializeToOstream(&f);
p3.SerializeToOstream(&f);
p4.set_f1(7);
p4.set_f2(8);
p4.set_f3(9);
p5.set_f1(0xa);
p5.set_f2(0xb);
p5.set_f3(0xc);
p4.SerializeToOstream(&f);
p5.SerializeToOstream(&f);
return 0;
}
$g++ 30.cpp 30.pb.cc -lprotobuf && ./a.out && xxd log30.data
00000000: 0801 1002 0803 1004 0805 1006 0807 1008 ................
00000010: 1809 080a 100b 180c ........
I just guessed if the byte stream always starts with the smallest tag number, and increases as it dumps the byte stream: when a smaller tag number is met, it thinks that's the start of a new object. Just my humble guess.
Need your explanations!
(3) Then, in reader side, how does reader know that these 6 values
should belong to 3 objects(each 2 fields), or belong to 2 objects(each
3 fields)?
In another word, how does the "require"/"optional" reflected inside
encoded bytes? If not reflected in the byte stream, then how does
protobuf determine the start of a new offset? We know protobuf don't
have "delimiter" bits.
Protobuf doesn't. It's up to you, the programmer, to split the messages before you feed them to protobuf.
For example, run this program:
#include "30.pb.h"
#include <fstream>
#include <iostream>
using namespace std;
int main()
{
fstream f("./log30.data",ios::binary|ios::out);
hello p1,p2,p3,p4,p5;
p1.set_f1(1);
p1.set_f2(2);
p2.set_f1(3);
p2.set_f2(4);
p3.set_f1(5);
p3.set_f2(6);
p1.SerializeToOstream(&f);
p2.SerializeToOstream(&f);
p3.SerializeToOstream(&f);
p4.set_f1(7);
p4.set_f2(8);
p4.set_f3(9);
p5.set_f1(0xa);
p5.set_f2(0xb);
p5.set_f3(0xc);
p4.SerializeToOstream(&f);
p5.SerializeToOstream(&f);
f.close();
f.open("./log30.data", ios::binary|ios::in);
hello hin;
hin.ParseFromIstream(&f);
cout << "f1: " << hin.f1() << ", f2: " << hin.f2() << ", f3: " << hin.f3() << "\n";
return 0;
}
You should see only the values of your last serialized hello object, as protobuf reads the whole stream and overwrites older values with newer ones.
Form the documentation
As you know, a protocol buffer message is a series of key-value pairs. The binary version of a message just uses the field's number as the key – the name and declared type for each field can only be determined on the decoding end by referencing the message type's definition (i.e. the .proto file).
When a message is encoded, the keys and values are concatenated into a byte stream. When the message is being decoded, the parser needs to be able to skip fields that it doesn't recognize. This way, new fields can be added to a message without breaking old programs that do not know about them. To this end, the "key" for each pair in a wire-format message is actually two values – the field number from your .proto file, plus a wire type that provides just enough information to find the length of the following value.
...
If a proto2 message definition has repeated elements (without the [packed=true] option), the encoded message has zero or more key-value pairs with the same tag number.
So optional elements could not be placed into output stream. While required must be included. Schema must be known for both serialization and deserialization (in contrast to Avro where schema must be embedded with data) so validation of required/optional field happens after deserialization when parser check if all required fields has values.
My data looks like this
-3442.77 -16749.64 893.08 -3442.77 -16749.64 1487.35 -3231.45 -16622.36 902.29
.....
159*2539.87 10*0.00 162*2539.87 10*0.00
which means I start with either 7 or 8 reals per line and then (towards the end) have 159 values of 2539.87 followed by 10 values of 0 followed by 162 of 2539.87 etc. This seems to be a space-saving method as previous versions of this file format were regular 6 reals per line.
I am already reading the data into a string because of not knowing whether there are 7 or 8 numbers per line. I can therefore easily spot lines that contain *. But what then? I suppose I have to identify the location of each * and then identify the integer number before and real value after before assigning to an array. Am I missing anything?
Read the line. Split it into tokens delimited by whitespace(s). Replace the * in tokens that have it with space. Then read from the string one or two values, depending on wheather there was an asterisk or not. Sample code follows:
REAL, DIMENSION(big) :: data
CHARACTER(LEN=40) :: token
INTEGER :: iptr, count, idx
REAL :: val
iptr = 1
DO WHILE (there_are_tokens_left)
... ! Get the next token into "token"
idx = INDEX(token, "*")
IF (idx == 0) THEN
READ(token, *) val
count = 1
ELSE
! Replace "*" with space and read two values from the string
token(idx:idx) = " "
READ(token, *) count, val
END IF
data(iptr:iptr+count-1) = val ! Add "val" "count" times to the list of values
iptr = iptr + count
END DO
Here I have arbitrarily set the length of the token to be 40 characters. Adjust it according to what you expect to find in your input files.
BTW, for the sake of completeness, this method of compressing something by replacing repeating values with value/repetition-count pairs is called run-length encoding (RLE).
Your input data may have been written in a form suitable for list directed input (where the format specification in the READ statement is simply ''*''). List directed input supports the r*c form that you see, where r is a repeat count and c is the constant to be repeated.
If the total number of input items is known in advance (perhaps it is fixed for that program, perhaps it is defined by earlier entries in the file) then reading the file is as simple as:
REAL :: data(size_of_data)
READ (unit, *) data
For example, for the last line shown in your example on its own ''size_of_data'' would need to be 341, from 159+10+162+10.
With list directed input the data can span across multiple records (multiple lines) - you don't need to know how many items are on each line in advance - just how many appear in the next "block" of data.
List directed input has a few other "features" like this, which is why it is generally not a good idea to use it to parse "arbitrary" input that hasn't been written with it in mind - use an explicit format specification instead (which may require creating the format specification on the fly to match the width of the input field if that is not know ahead of time).
If you don't know (or cannot calculate) the number of items in advance of the READ statement then you will need to do the parsing of the line yourself.