I am trying to dissect signed short (16-bit) values for an application layer protocol.
I know it is possible with unsigned binary fields by passing a negative size like in this example:
fields_desc = [ BitField("Name", None, -16) ]
However, all the signed fields seem to be lacking this low endian feature. For example, this is the only use case I could find:
fields_desc = [ SignedShortField("Name", None) ]
Documentation:
https://scapy.readthedocs.io/en/latest/api/scapy.fields.html
Related
privateEncrypt() says "..., the padding property can be passed. Otherwise, this function uses RSA_PKCS1_PADDING." sign() says "padding Optional padding value for RSA, one of the following:
crypto.constants.RSA_PKCS1_PADDING (default)".
So naive expectation would be that both returns the same buffer as the padding scheme and hash function used are identical. (I suppose that that privateEncrypt() uses the signing variant of the scheme, when publicEncrypt() uses encryption variant; please, count this as part of the question, as I could find this one in docs, and sunk mapping OpenSSL manuals to node:crypto, your expertise is helpful!)
But they don't. Maybe I read the docs incorrectly, maybe it's common knowledge I'm missing, or maybe it's something else. Please explain the differences between them in this sense, or correct the snippet so it would be visually clear.
// this snippet is solely for discussion purpose and shouldn't be used in production
import {
generateKeyPairSync, createSign, privateEncrypt, createVerify, createHash
} from "node:crypto";
const keyPair = generateKeyPairSync("rsa", {modulusLength: 1024});
const encrypted = privateEncrypt(
keyPair.privateKey,
createHash('sha256').update("stack overflow").digest()
);
// console.debug(encrypted);
const signed = createSign("SHA256").update("stack overflow").end().sign(keyPair.privateKey);
// console.debug(signed);
// console.debug(createVerify("SHA256").update("stack overflow").end().verify(
// keyPair.publicKey, signed
// )); // "true"
console.assert(!Buffer.compare(encrypted, signed)); // "Assertion failed"
Sign#sign() is used for signing and applies the RSASSA-PKCS1-v1_5 variant for PKCS#1v1.5 padding.
RSASSA-PKCS1-v1_5 is described in RFC 8017, Section 8.2. This uses the EMSA-PKCS1-v1_5 encoding described in Section 9.2, which pads the message as follows:
EM = 0x00 || 0x01 || PS || 0x00 || T
where T is the DER encoding of the DigstInfo value (containing the digest OID and the message hash) and PS is a sequence of as many 0xFF bytes until the length of the key/modulus is reached.
While sign() implicitly determines T from the message, crypto.privateEncrypt() uses the message directly as T. Thus, for privateEncrypt() to work like sign(), T must be explicitly determined from the message and passed instead of the message. Since the posted code already hashes, only the digest-specific byte sequence containing the OID (see RFC8017, p. 47) needs to be prepended.
const digestInfo = Buffer.from('3031300d060960864801650304020105000420','hex');
const dataHashed = crypto.createHash('sha256').update("stack overflow").digest();
const dataToSign = Buffer.concat([digestInfo, dataHashed]);
const encrypted = crypto.privateEncrypt(keyPair.privateKey, dataToSign);
Because RSASSA-PKCS1-v1_5 is deterministic, both approaches provide the same signature (for an identical key).
Why are there both methods?
Sign#sign() is used for regular signing, crypto.privateEncrypt() is meant for a low level signig, e.g. when not the message itself but only the hashed data is available.
Since v12.0.0 there is also crypto.sign() which performs a high level signing like Sign#sign(). This new variant also supports newer algorithms like Ed25519:
const signedv12 = crypto.sign('sha256', 'stack overflow', keyPair.privateKey);
In contrast, crypto.publicEncrypt() performs an encryption and uses the RSAES-PKCS1-v1_5 variant for PKCS#1v1.5 padding.
Note that 1024 bit keys are insecure nowadays. The key size should be at least 2048 bits.
I've started taking a look at Nim for hobby game modding purposes.
Intro
Yet, I found it difficult to work with Nim compared to C when it comes to machine-specific low-level memory layout and would like to know if Nim actually has better support here.
I need to control byte order and be able to de/serialize arbitrary Plain-Old-Datatype objects to binary custom file formats. I didn't directly find a Nim library which allows flexible storage options like representing enum and pointers with Big-Endian 32-bit. Or maybe I just don't know how to use the feature.
std/marshal : just JSON, i.e. no efficient, flexible nor binary format but cross-compatible
nim-serialization : seems like being made for human readable formats
nesm : flexible cross-compatibility? (It has some options and has a good interface)
flatty : no flexible cross-compatibility, no byte order?
msgpack4nim : no flexible cross-compatibility, byte order?
bingo : ?
Flexible cross-compatibility means, it must be able to de/serialize fields independently of Nim's ABI but with customization options.
Maybe "Kaitai Struct" is more what I look for, a file parser with experimental Nim support.
TL;DR
As a workaround for a serialization library I tried myself at a recursive "member fields reverser" that makes use of std/endians which is almost sufficient.
But I didn't succeed with implementing byte reversal of arbitrarily long objects in Nim. Not practically relevant but I still wonder if Nim has a solution.
I found reverse() and reversed() from std/algorithm but I need a byte array to reverse it and turn it back into the original object type. In C++ there would be reinterprete_cast, in C there is void*-cast, in D there is a void[] cast (D allows defining array slices from pointers) but I couldn't get it working with Nim.
I tried cast[ptr array[value.sizeof, byte]](unsafeAddr value)[] but I can't assign it to a new variable. Maybe there was a different problem.
How to "byte reverse" arbitrary long Plain-Old-Datatype objects?
How to serialize to binary files with byte order, member field size, pointer as file "offset - start offset"? Are there bitfield options in Nim?
It is indeed possible to use algorithm.reverse and the appropriate cast invocation to reverse bytes in-place:
import std/[algorithm,strutils,strformat]
type
LittleEnd{.packed.} = object
a: int8
b: int16
c: int32
BigEnd{.packed.} = object
c: int32
b: int16
a: int8
## just so we can see what's going on:
proc `$`(b: LittleEnd):string = &"(a:0x{b.a.toHex}, b:0x{b.b.toHex}, c:0x{b.c.toHex})"
proc `$`(l:BigEnd):string = &"(c:0x{l.c.toHex}, b:0x{l.b.toHex}, a:0x{l.a.toHex})"
var lit = LittleEnd(a: 0x12, b:0x3456, c: 0x789a_bcde)
echo lit # (a:0x12, b:0x3456, c:0x789ABCDE)
var big:BigEnd
copyMem(big.addr,lit.addr,sizeof(lit))
# here's the reinterpret_cast you were looking for:
cast[var array[sizeof(big),byte]](big.addr).reverse
echo big # (c:0xDEBC9A78, b:0x5634, a:0x12)
for C-style bitfields there is also the {.bitsize.} pragma
but using it causes Nim to lose sizeof information, and of course bitfields wont be reversed within bytes
import std/[algorithm,strutils,strformat]
type
LittleNib{.packed.} = object
a{.bitsize: 4}: int8
b{.bitsize: 12}: int16
c{.bitsize: 20}: int32
d{.bitsize: 28}: int32
BigNib{.packed.} = object
d{.bitsize: 28}: int32
c{.bitsize: 20}: int32
b{.bitsize: 12}: int16
a{.bitsize: 4}: int8
const nibsize = 8
proc `$`(b: LittleNib):string = &"(a:0x{b.a.toHex(1)}, b:0x{b.b.toHex(3)}, c:0x{b.c.toHex(5)}, d:0x{b.d.toHex(7)})"
proc `$`(l:BigNib):string = &"(d:0x{l.d.toHex(7)}, c:0x{l.c.toHex(5)}, b:0x{l.b.toHex(3)}, a:0x{l.a.toHex(1)})"
var lit = LitNib(a: 0x1,b:0x234, c:0x56789, d: 0x0abcdef)
echo lit # (a:0x1, b:0x234, c:0x56789, d:0x0ABCDEF)
var big:BigNib
copyMem(big.addr,lit.addr,nibsize)
cast[var array[nibsize,byte]](big.addr).reverse
echo big # (d:0x5DEBC0A, c:0x8967F, b:0x123, a:0x4)
It's less than optimal to copy the bytes over, then rearrange them with reverse, anyway, so you might just want to copy the bytes over in a loop. Here's a proc that can swap the endianness of any object, (including ones for which sizeof is not known at compiletime):
template asBytes[T](x:var T):ptr UncheckedArray[byte] =
cast[ptr UncheckedArray[byte]](x.addr)
proc swapEndian[T,U](src:var T,dst:var U) =
assert sizeof(src) == sizeof(dst)
let len = sizeof(src)
for i in 0..<len:
dst.asBytes[len - i - 1] = src.asBytes[i]
Bit fields are supported in Nim as a set of enums:
type
MyFlag* {.size: sizeof(cint).} = enum
A
B
C
D
MyFlags = set[MyFlag]
proc toNum(f: MyFlags): int = cast[cint](f)
proc toFlags(v: int): MyFlags = cast[MyFlags](v)
assert toNum({}) == 0
assert toNum({A}) == 1
assert toNum({D}) == 8
assert toNum({A, C}) == 5
assert toFlags(0) == {}
assert toFlags(7) == {A, B, C}
For arbitrary bit operations you have the bitops module, and for endianness conversions you have the endians module. But you already know about the endians module, so it's not clear what problem you are trying to solve with the so called byte reversal. Usually you have an integer, so you first convert the integer to byte endian format, for instance, then save that. And when you read back, convert from byte endian format and you have the int. The endianness procs should be dealing with reversal or not of bytes, so why do you need to do one yourself? In any case, you can follow the source hyperlink of the documentation and see how the endian procs are implemented. This can give you an idea of how to cast values in case you need to do some yourself.
Since you know C maybe the last resort would be to write a few serialization functions and call them from Nim, or directly embed them using the emit pragma. However this looks like the least cross platform and pain free option.
Can't answer anything about generic data structure serialization libraries. I stray away from them because they tend to require hand holding imposing certain limitations on your code and depending on the feature set, a simple refactoring (changing field order in your POD) may destroy the binary compatibility of the generated output without you noticing it until runtime. So you end up spending additional time writing unit tests to verify that the black box you brought in to save you some time behaves as you want (and keeps doing so across refactorings and version upgrades!).
I am interested in passing binary data between python, numpy, and cython using the buffer protocol. Looking at PEP 3118, there appear to be some additions to the struct string-syntax that add support for useful features such as named fields and nested structs.
However, it appears that support for the full range of buffer syntax is limited in all of those three places. For example, say I have the following cython struct:
ctypedef packed struct ImageComp:
uint32_t width
uint32_t height
uint8_t *pixels
#Here is the appropriate struct format string representation
IMAGE_FORMAT = b'T{L:width:L:height:&B:pixels:}'
Attempting to extract the PEP-3118 compliant bytes string as follows
cdef void *image_temp = malloc(sizeof(ImageComp))
IMAGE_SIZE = sizeof(ImageComp)
IMAGE_FORMAT = (<ImageComp[:1]>image_temp)._format
IMAGE_DTYPE = np.asarray(<ImageComp[:1]>image_temp).dtype
free(image_temp)
Fails with this error message:
Invalid base type for memoryview slice: ImageComp
since typed memoryviews cannot be created if they contain pointers.
Similarly, creating a view.array using my custom string or using the python struct module's calcsize function will give a warning like struct.error: bad char in struct format.
I can manually create and fill a Py_buffer object as described here, but attempting to convert this to a numpy array with np.asarray yields ValueError: 'T{L:width:L:height:&B:pixels:}' is not a valid PEP 3118 buffer format string.
With all of this in mind, I have the following questions:
Is there any module in the standard python library that takes advantage of the complete PEP 3118 specification?
Is this struct format syntax defined formally anywhere (i.e. with a PEG grammar)?
Is there a way to force cython or numpy to automatically generate a valid format string if it contains pointers?
I am developing a bluetooth app that connects to a fitness watch. This is my first time working with bluetooth. I managed to connect my app with the device using the brilliant FlutterBlue library.
However I am unable to make sense of the results I get from reading. This is how I read the characteristics:
_readCharacteristic(BluetoothCharacteristic c) async {
var results = await widget.device.readCharacteristic(c);
print("${results.toList()}");
//setState(() {});
}
This is the result:
[7, 133, 0, 0, 1, 0, 0, 124, 92, 1]
I have no idea what these numbers mean or what I am supposed to do with them.
From the documentation:
var characteristics = service.characteristics;
for(BluetoothCharacteristic c in characteristics) {
List<int> value = await device.readCharacteristic(c);
print(value);
}
// Writes to a characteristic
await device.writeCharacteristic(c, [0x12, 0x34])
We can see that the library works on List(int) types, and that it sends/receives "32 bit" values.
It is most likely pairs of bytes being sent over, so 16 bit values represented as numbers in the list; these are characters. This means that you can send over characters with their utf8 representation.
In the above example, the characteristic is writing 0x12, then 0x34. In the link to the ascii character table, this translates to "(Device Control 2) (4)".
It is your job to decode them into characters, (UTF8) and also to encode them when sending them back to the watch. This is required by the watch's software, which can respond to certain characteristic writes depending on the received value.
You probably will have to do some digging into the documentation/bluetooth spec of the watch you are using.
Check out the UTF8Decoder class of dart:convert lib. It should help you translate it into human-readable text. If not, you will have to do some digging.
String decoded = UTF8Decoder().convert(value) // value == List<int>, Uint8List, etc.
print(decoded)
I want to use the scheme “Hess - Efficient identity based signature schemes based on pairings.” of Charm-Crypto.
I need to calculate the bit-length of group elements (eg. S2 in the siganture).
As I understand it from this related question, serializing the elements gives me a Base64 encoded string. From that and this other question about Base64, I concluded that I need to compute the following:
signature = {'S1' : S1, 'S2' : S2}
S2_serial = group.serialize(signature['S2'])
sigLenInBase64 = len(S2_serial)
sigLenInByte = (sigLenInBase64 *3)/4
sigLenInBit = sigLenInByte * 8
As S2 is a group element of G1, I expect the size to be the same as that of the underlying curve (so 512Bit for 'SS512', 224Bit for 'MNT224', etc). However, the size differ by 28 Bit for 'SS512' and 'MNT224' (with 540 and 252 Bit, respectively) and by 21 Bit for 'MNT159'. Why is that? How can I predict for a given curve by how much it will be off?
My current guess is, that I'm not taking into account some additional info (like a byte for the sign).
Update:
Using the accepted answer, I now can correctly compute the size like this:
def compute_bitsize(base64_input):
b_padded = base64_input.split(str.encode(":"))[1]
pad_size = b_padded.count(str.encode("="))
b_len_without_pad = len(b_padded)-4
byte_len = (b_len_without_pad *3)/4 +(3-pad_size)-1
bit_len = byte_len * 8
return bit_len
Take the example serialization for 'SS512' from the question you link to. The serialization string is:
1:Nri028S0SySkDoN0JH1Lu6HQo/Jkhq7DCZHI1MUrHOuIgCONonN14GzmhwopYQOxnjOysclhYNOnsvCLjVDLggE=
Looking at the line:
PyObject *result = PyBytes_FromFormat("%d:%s", element->element_type, (const char *) base64_data_buf);
from the source code for charm we see that the part of the string you are interested in is
Nri028S0SySkDoN0JH1Lu6HQo/Jkhq7DCZHI1MUrHOuIgCONonN14GzmhwopYQOxnjOysclhYNOnsvCLjVDLggE=
Decoding this as described on Wikipedia under 'Decoding Base64 with padding' gives us 65 bytes where one byte is a "byte for the sign of the element" according to the question you link to.