I am using the python3 struct module to unpack byte data I extracted from a serial com. (With help) I've figured out how to unpack most the data into human readable form. I am having difficult with the format string on a group header struct group_hdr (please see attached screenshot document). I have a byte data (b). I know the character string for "word" is "H" but it's unclear to me from the document what phd_status is. It hasn't been defined anywhere else in the Data structure document. Any ideas?. Thank you in advance.
struct group_hdr
{
union phdb_status status
word label
}
subrecord = struct.unpack_from('<??H', b)
As is explained under Status, it is a simple bitfield with a width of 32 bits. The union is probably defined elsewhere in C (or a similar language) as
union phdb_status {
unsigned int bit_0:1;
unsigned int bit_1:1;
};
The following Python code will store your values:
status, label = struct.unpack_from('<IH', b)
and you can test the individual bits of status with status & 1 and status & 2.
Related
I want to read a tuple from my Excel file for my OPL model in CPLEX, which has the following format:
tuple A {
string a;
int b;
{string} c;
}
The set {string} c of each tuple element is of variable length.
How can I read this via SheetConnection or is this not possible?
I've tried to read it via named fields in Excel, but that didn't work.
I have also read on the CPLEX documentation, that it is only possible to read tuples of fixed width, but my data is not of fixed width.
if you know the number of tuples, you could read 2 arrays with SheetRead
One for
string a;
int b;
The second for
{string} c;
For that array you could use the max number of strings.
And then in the .mod you turn those 2 arrays into a tuple set.
Iam working in python 3.6
I receive from serial comunication an string '3F8E353F'. This is a float number 1.111. How can I convert this string to a float number?
Thank you
Ah yes. Since this is 32-bits, unpack it into an int first then:
x='3F8E353F'
struct.unpack('f',struct.pack('i',int(x,16)))
On my system this gives:
>>> x='3F8E353F'
>>> struct.unpack('f',struct.pack('i',int(x,16)))
(1.1109999418258667,)
>>>
Very close to the expected value. However, this can give 'backwards' results based on the 'endianness' of bytes in your system. Some systems store their bytes least significant byte first, others most significant byte first. See this reference page for the descriptors to format based on byte order.
I used struct.unpack('f',struct.pack('i',int(x,16))) to convert Hex value to float but for negative value I am getting below error
struct.error: argument out of range
To resolve this I used below code which converts Hex (0xc395aa3d) to float value (-299.33). It works for both positive as well for negative value.
x = 0xc395aa3d
struct.unpack('f', struct.pack('I', int(x,16) ))
Another way is to use bytes.fromhex()
import struct
hexstring = '3F8E353F'
struct.unpack('!f', bytes.fromhex(hexstring))[0]
#answer:1.1109999418258667
Note: The form '!' is available for those poor souls who claim they can’t remember whether network byte order is big-endian or little-endian (from struct docs).
For example, if I define a photo like this:
$cat 30.proto
message hello
{
required int32 f1=1;
required int32 f2=2;
optional int32 f3=3;
}
Then I would double if protobuf can handle things like this:
I declare 3 objects, each don't have f3 field.
Write to output
Then, in reader side, how does reader know that these 6 values should belong to 3 objects(each 2 fields), or belong to 2 objects(each 3 fields)?
In another word, how does the "require"/"optional" reflected inside encoded bytes? If not reflected in the byte stream, then how does protobuf determine the start of a new offset? We know protobuf don't have "delimiter" bits.
I had a simple quick test on this:
$cat 30.cpp
#include "30.pb.h"
#include<fstream>
using namespace std;
int main()
{
fstream f("./log30.data",ios::binary|ios::out);
hello p1,p2,p3,p4,p5;
p1.set_f1(1);
p1.set_f2(2);
p2.set_f1(3);
p2.set_f2(4);
p3.set_f1(5);
p3.set_f2(6);
p1.SerializeToOstream(&f);
p2.SerializeToOstream(&f);
p3.SerializeToOstream(&f);
p4.set_f1(7);
p4.set_f2(8);
p4.set_f3(9);
p5.set_f1(0xa);
p5.set_f2(0xb);
p5.set_f3(0xc);
p4.SerializeToOstream(&f);
p5.SerializeToOstream(&f);
return 0;
}
$g++ 30.cpp 30.pb.cc -lprotobuf && ./a.out && xxd log30.data
00000000: 0801 1002 0803 1004 0805 1006 0807 1008 ................
00000010: 1809 080a 100b 180c ........
I just guessed if the byte stream always starts with the smallest tag number, and increases as it dumps the byte stream: when a smaller tag number is met, it thinks that's the start of a new object. Just my humble guess.
Need your explanations!
(3) Then, in reader side, how does reader know that these 6 values
should belong to 3 objects(each 2 fields), or belong to 2 objects(each
3 fields)?
In another word, how does the "require"/"optional" reflected inside
encoded bytes? If not reflected in the byte stream, then how does
protobuf determine the start of a new offset? We know protobuf don't
have "delimiter" bits.
Protobuf doesn't. It's up to you, the programmer, to split the messages before you feed them to protobuf.
For example, run this program:
#include "30.pb.h"
#include <fstream>
#include <iostream>
using namespace std;
int main()
{
fstream f("./log30.data",ios::binary|ios::out);
hello p1,p2,p3,p4,p5;
p1.set_f1(1);
p1.set_f2(2);
p2.set_f1(3);
p2.set_f2(4);
p3.set_f1(5);
p3.set_f2(6);
p1.SerializeToOstream(&f);
p2.SerializeToOstream(&f);
p3.SerializeToOstream(&f);
p4.set_f1(7);
p4.set_f2(8);
p4.set_f3(9);
p5.set_f1(0xa);
p5.set_f2(0xb);
p5.set_f3(0xc);
p4.SerializeToOstream(&f);
p5.SerializeToOstream(&f);
f.close();
f.open("./log30.data", ios::binary|ios::in);
hello hin;
hin.ParseFromIstream(&f);
cout << "f1: " << hin.f1() << ", f2: " << hin.f2() << ", f3: " << hin.f3() << "\n";
return 0;
}
You should see only the values of your last serialized hello object, as protobuf reads the whole stream and overwrites older values with newer ones.
Form the documentation
As you know, a protocol buffer message is a series of key-value pairs. The binary version of a message just uses the field's number as the key – the name and declared type for each field can only be determined on the decoding end by referencing the message type's definition (i.e. the .proto file).
When a message is encoded, the keys and values are concatenated into a byte stream. When the message is being decoded, the parser needs to be able to skip fields that it doesn't recognize. This way, new fields can be added to a message without breaking old programs that do not know about them. To this end, the "key" for each pair in a wire-format message is actually two values – the field number from your .proto file, plus a wire type that provides just enough information to find the length of the following value.
...
If a proto2 message definition has repeated elements (without the [packed=true] option), the encoded message has zero or more key-value pairs with the same tag number.
So optional elements could not be placed into output stream. While required must be included. Schema must be known for both serialization and deserialization (in contrast to Avro where schema must be embedded with data) so validation of required/optional field happens after deserialization when parser check if all required fields has values.
I am getting input from the user, however when I try to compare it later on to a string literal it does not work. That is just a test though.
I would like to set it up so that when a blank line is entered (just hitting the enter/return key) the program exits. I don't understand why the strings are not comparing because when I print it, it comes out identical.
in := bufio.NewReader(os.Stdin);
input, err := in.ReadBytes('\n');
if err != nil {
fmt.Println("Error: ", err)
}
if string(input) == "example" {
os.Exit(0)
}
string vs []byte
string definition:
string is the set of all strings of 8-bit bytes, conventionally but not necessarily representing UTF-8-encoded text. A string may be empty, but not nil. Values of string type are immutable.
byte definition:
byte is an alias for uint8 and is equivalent to uint8 in all ways. It is used, by convention, to distinguish byte values from 8-bit unsigned integer values.
What does it mean?
[]byte is a byte slice. slice can be empty.
string elements are unicode characters, which can have more then 1 byte.
string elements keep a meaning of data (encoding), []bytes not.
equality operator is defined for string type but not for slice type.
As you see they are two different types with different properties.
There is a great blog post explaining different string related types [1]
Regards the issue you have in your code snippet.
Bear in mind that in.ReadBytes(char) returns a byte slice with char inclusively. So in your code input ends with '\n'. If you want your code to work in desired way then try this:
if string(input) == "example\n" { // or "example\r\n" when on windows
os.Exit(0)
}
Also make sure that your terminal code page is the same as your .go source file. Be aware about different end-line styles (Windows uses "\r\n"), Standard go compiler uses utf8 internally.
[1] Comparison of Go data types for string processing.
I have seen code like this:
struct failed_login_res {
string errorMsg<>;
unsigned int error;
};
What does the <> at the end mean? How is it different from normal declaration like string errorMsg?
Correction: this is for RPC stub, not C++ and I can confirm that it does compile. The question is then still valid.
From a quick googling, I came across this PDF.
Section 6.9 is as follows:
Strings: C has no built-in string type, but instead uses the null-terminated “char *” convention. In XDR language, strings are declared using the “string” keyword, and compiled into “char *”s in the output header file. The maximum size contained in the angle brackets specifies the maximum number of characters allowed in the strings (not counting the NULL character). The maximum size may be left off, indicating a string of arbitrary length.
Examples:
string name<32>; --> char *name;
string longname<>; --> char *longname;