I am a newbie here.
I am looking to know about any tool/quick way to convert a 24bit PCM raw(headerless) file, having 3 byte PCM samples,
into a 32 bit PCM raw file which has 4 bytes per sample, with the MSByte of the 4 byte data as sign/zero extension of the 3byte sample.
Apart from the 24bit raw file, I have its corresponding WAVE file as well if it helps.
When tried in audacity, although it converted 24 bit to 32bit, it did not sign/zero extend, but it left shifted by 8, the 24 bit sample. So in effect the 24 bit sample was sitting in the left aligned 24 bits of the 32 bit , which is not what was desired.
Thanks.
I'm going to assume you meant shifted left by 8 instead of shifted right by eight.
In this case the notion of sign extension is unnecessary. Imagine you have a negative 24-bit value 0x800000. Then the left shifted version would be 0x80000000. No sign extension but it still has the correct negative sign.
In summary I think audacity is doing exactly as it should, which is to simply shift the bits up. Unless for some reason your data is unsigned which would be exceptionally unusual.
Upon more search was pointed a way to do this is using sox - on linux.
sox -t s24 --endian little input.pcm -t s32 output.pcm vol 0.00390625
It worked fine.
the vol 0.00390625 is to reduce the volume by 48dB because conversion of raw PCM sample from 24bit to 32bit, by default left shifts by 8 bit, but I want it to be down-shifted back by 8 bits which is reduction in volume by 48dB
For writing my own ogg-container-class (not using libogg), I try to understand the needed header format. According to the spec, at byte 27 of the stream (starting to count at 0) starts the "segment_table (containing packet lacing values)". This is the red marked byte 13. Concerning the Opus-data that I want to include, the Opus data must start with OpusHead (4F 70 75 73) on its beginning. Why doesn't it start on position 27 where the red 13 is placed? A 13 is a "device control 3" symbol that neither occurs in the Ogg spec, nor in the Opus spec.
EDIT: I found this link that describes the spec a little. There it becomes clear (which it is not from the first link imho) that the 13 (byte 27) is the size of the following segment.
That appears to be a single byte giving the length of the following segment_table data. So there is 13(hex) bytes (16 decimal) bytes of segment_table data.
RFC 3533 is a more verbose description of the format header.
Byte 26 says how many bytes the segment table occupies, so you read that, add 27, and that tells you where the first packet starts (or continues).
The segment table tells you the length(s) of the encapsulated packet(s). Basically you read through the table, adding together the values in each successive byte. If the value you just added is < 255 then that marks a packet boundary, so record the current value of the accumulator, reset it to zero, then continue until you reach the end of the table.
In your example, the segment table size in byte 26 is 1, so the data starts at 27+1 or byte 28, which is the start of the 'OpusHead' string. The value in the 1 byte segment table is 0x13, so the packet is 19 bytes long. 28+19 is 47 (or 0x2f) which is the start of the 'OggS' capture pattern at the start of the next header.
This slightly complicated algorithm is designed to store framing data for many small packets with bounded overhead while still allowing arbitrarily large packets. Note also that packets can be continued between pages, spanning 2 or more segment tables.
How can I increase the depth of an extent tree? How much space should be allocated? To be more precise I only need the depth to be more than 1.
In ext4, an inode uses 60 bytes to address 4 extents by default (12bytes for the header + 4*12bytes for the extents). Every extent can hold data up to 128MiB (by default).
If a file is larger than 4*128MiB (> 512MiB) ext4 is building an extent tree for that. So you'll reach depth = 1 with that.
Now the question is when will you reach depth=2. I tried with a file of ~700MiB and checked the extent depth which was 1. Then I opened the fs-block indexed by the extent and analysed the header.
0A F3 06 00 54 01 00 00 00 00 00 00 00
It has the typical magic number of f30A and the next two bytes should indicate the number of extents. (in this case=6 which makes sense because 700/128 = 6).
The next 2 bytes 0154 should indicate the maximum number of extents (according to the header information) which comes up to 340 in my case.
-> so why 340? If we look at the actual fs-block size of ext4 the default is 4096 bytes.
Each index consists of 12 bytes. 340*12 = 4080 + 12 for the header = 4092 so it is the max number of extent-info that can fit into this fs-block.
If I interpret this correctly, you can save up to 340 other pointers/leaves/indexes in this fs-block. Each of these 340 refer to an extent (which can hold up to 128MiB)
-> so it comes to: 340*128MiB ~ 43,5 GiB
Additionally each inode can hold up to 4 extents. So I guess this number needs to be multiplied by 4 even though.
-> so I think your file should be > 4*340*128MiB ~ 174GiB to reach depth=2
I have not yet tested this hypothesis, but I will try it soon. Meanwhile if someone can proove me wrong, I'm happy as well. :)
I have a string of decimal digits like:
965854242113548732659745896523654789653244879653245794444524
length : 60 character
I want to send it to a function, but first I want reduce the length of it as much as possible. How can I do that?
I think about convert it to base-34, that will be 1RG7EEWTN7NW60EWIWMASEWWMEOSWC2SS8482WQE. That is 40 characters in length. Can I reduce it more some way?
Your number fits into 70 bits - for such a small payload compression seems nonsensical. Assuming that the server API supports arbitrary binary data, I would simply encode the value in binary and prefix it with the number of bytes needed.
1 byte length information - for 854657986453156789675, the example you gave initially, this would be 9
9 bytes of binary payload
→ 10 bytes of data transferred for your example.
Your example in hex:
09 2e 54 c3 1e 81 cf 05 fd ab
With the length given in bytes, this of course supports only decimals up to 255 bytes length, but I suppose this is sufficient. If your transport protocol has a built in concept of length of a packet, you could even skip the initial length byte.
Important: ensure that all sides use the same endianness. As you are transmitting your data over the network, network byte order (big endian) would be natural.
If you want to transmit very large numbers, keep in mind that you can use any compression algorithm you like on the binary representation of your data. However, your payload must be significantly larger in order to make compression feasible - for example, using zLib compression for the above 9 byte payload results in an 18 byte payload due to the overhead for the zLib datastructures.
If (and only if) you cannot use arbitrary bytes for your payload, you can encode your data (possibly after compression). Most modern libraries have built in support for Base64, so this would be a natual way of representing the data.
I need to include basic file-sending and file-receiving routines in my program, and it needs to be through the ZMODEM protocol. The problem is that I'm having trouble understanding the spec.
For reference, here is the specification.
The spec doesn't define the various constants, so here's a header file from Google.
It seems to me like there are a lot of important things left undefined in that document:
It constantly refers to ZDLE-encoding, but what is it? When exactly do I use it, and when don't I use it?
After a ZFILE data frame, the file's metadata (filename, modify date, size, etc.) are transferred. This is followed by a ZCRCW block and then a block whose type is undefined according to the spec. The ZCRCW block allegedly contains a 16-bit CRC, but the spec doesn't define on what data the CRC is computed.
It doesn't define the CRC polynomial it uses. I found out by chance that the CRC32 poly is the standard CRC32, but I've had no such luck with the CRC16 poly. Nevermind, I found it through trial and error. The CRC16 poly is 0x1021.
I've looked around for reference code, but all I can find are unreadable, undocumented C files from the early 90s. I've also found this set of documents from the MSDN, but it's painfully vague and contradictory to tests that I've run: http://msdn.microsoft.com/en-us/library/ms817878.aspx (you may need to view that through Google's cache)
To illustrate my difficulties, here is a simple example. I've created a plaintext file on the server containing "Hello world!", and it's called helloworld.txt.
I initiate the transfer from the server with the following command:
sx --zmodem helloworld.txt
This prompts the server to send the following ZRQINIT frame:
2A 2A 18 42 30 30 30 30 30 30 30 30 30 30 30 30 **.B000000000000
30 30 0D 8A 11 00.Š.
Three issues with this:
Are the padding bytes (0x2A) arbitrary? Why are there two here, but in other instances there's only one, and sometimes none?
The spec doesn't mention the [CR] [LF] [XON] at the end, but the MSDN article does. Why is it there?
Why does the [LF] have bit 0x80 set?
After this, the client needs to send a ZRINIT frame. I got this from the MSDN article:
2A 2A 18 42 30 31 30 30 30 30 30 30 32 33 62 65 **.B0100000023be
35 30 0D 8A 50.Š
In addition to the [LF] 0x80 flag issue, I have two more issues:
Why isn't [XON] included this time?
Is the CRC calculated on the binary data or the ASCII hex data? If it's on the binary data I get 0x197C, and if it's on the ASCII hex data I get 0xF775; neither of these are what's actually in the frame (0xBE50). (Solved; it follows whichever mode you're using. If you're in BIN or BIN32 mode, it's the CRC of the binary data. If you're in ASCII hex mode, it's the CRC of what's represented by the ASCII hex characters.)
The server responds with a ZFILE frame:
2A 18 43 04 00 00 00 00 DD 51 A2 33 *.C.....ÝQ¢3
OK. This one makes sense. If I calculate the CRC32 of [04 00 00 00 00], I indeed get 0x33A251DD. But now we don't have ANY [CR] [LF] [XON] at the end. Why is this?
Immediately after this frame, the server also sends the file's metadata:
68 65 6C 6C 6F 77 6F 72 6C 64 2E 74 78 74 00 31 helloworld.txt.1
33 20 32 34 30 20 31 30 30 36 34 34 20 30 20 31 3 240 100644 0 1
20 31 33 00 18 6B 18 50 D3 0F F1 11 13..k.PÓ.ñ.
This doesn't even have a header, it just jumps straight to the data. OK, I can live with that. However:
We have our first mysterious ZCRCW frame: [18 6B]. How long is this frame? Where is the CRC data, and is it CRC16 or CRC32? It's not defined anywhere in the spec.
The MSDN article specifies that the [18 6B] should be followed by [00], but it isn't.
Then we have a frame with an undefined type: [18 50 D3 0F F1 11]. Is this a separate frame or is it part of ZCRCW?
The client needs to respond with a ZRPOS frame, again taken from the MSDN article:
2A 2A 18 42 30 39 30 30 30 30 30 30 30 30 61 38 **.B0900000000a8
37 63 0D 8A 7c.Š
Same issues as with the ZRINIT frame: the CRC is wrong, the [LF] has bit 0x80 set, and there's no [XON].
The server responds with a ZDATA frame:
2A 18 43 0A 00 00 00 00 BC EF 92 8C *.C.....¼ï’Œ
Same issues as ZFILE: the CRC is all fine, but where's the [CR] [LF] [XON]?
After this, the server sends the file's payload. Since this is a short example, it fits in one block (max size is 1024):
48 65 6C 6C 6F 20 77 6F 72 6C 64 21 0A Hello world!.
From what the article seems to mention, payloads are escaped with [ZDLE]. So how do I transmit a payload byte that happens to match the value of [ZDLE]? Are there any other values like this?
The server ends with these frames:
18 68 05 DE 02 18 D0 .h.Þ..Ð
2A 18 43 0B 0D 00 00 00 D1 1E 98 43 *.C.....Ñ.˜C
I'm completely lost on the first one. The second makes as much sense as the ZRINIT and ZDATA frames.
My buddy wonders if you are implementing a time
machine.
I don't know that I can answer all of your questions -- I've never
actually had to implement zmodem myself -- but here are few answers:
From what the article seems to mention, payloads are escaped with
[ZDLE]. So how do I transmit a payload byte that happens to match the
value of [ZDLE]? Are there any other values like this?
This is explicitly addressed in the document you linked to at the
beginning of your questions, which says:
The ZDLE character is special. ZDLE represents a control sequence
of some sort. If a ZDLE character appears in binary data, it is
prefixed with ZDLE, then sent as ZDLEE.
It constantly refers to ZDLE-encoding, but what is it? When exactly
do I use it, and when don't I use it?
In the Old Days, certain "control characters" were used to control the
communication channel (hence the name). For example, sending XON/XOFF
characters might pause the transmission. ZDLE is used to escape
characters that may be problematic. According to the spec, these are
the characters that are escaped by default:
ZMODEM software escapes ZDLE, 020, 0220, 021, 0221, 023, and 0223.
If preceded by 0100 or 0300 (#), 015 and 0215 are also escaped to
protect the Telenet command escape CR-#-CR. The receiver ignores
021, 0221, 023, and 0223 characters in the data stream.
I've looked around for reference code, but all I can find are
unreadable, undocumented C files from the early 90s.
Does this include the code for the lrzsz package? This is still
widely available on most Linux distributions (and surprisingly handy
for transmitting files over an established ssh connection).
There are a number of other implementations out there, including
several in software listed on freecode, including qodem,
syncterm, MBSE, and others. I believe the syncterm
implementation is written as library that may be reasonable easy
to use from your own code (but I'm not certain).
You may find additional code if you poke around older collections of
MS-DOS software.
I can't blame you. The user manual is not organized in a user friendly way
Are the padding bytes (0x2A) arbitrary?
No, from page 14,15:
A binary header begins with the sequence ZPAD, ZDLE, ZBIN.
A hex header begins with the sequence ZPAD, ZPAD, ZDLE, ZHEX.
.
The spec doesn't mention the [CR] [LF] [XON] at the end, but the MSDN article does. Why is it there?
Page 15
* * ZDLE B TYPE F3/P0 F2/P1 F1/P2 F0/P3 CRC-1 CRC-2 CR LF XON
.
Why does the [LF] have bit 0x80 set?
Not sure. From Tera term I got both control characters XORed with 0x80 (8D 8A 11)
We have our first mysterious ZCRCW frame: [18 6B]. How long is this frame? Where is the CRC data, and is it CRC16 or CRC32? It's not defined anywhere in the spec.
The ZCRCW is not a header or a frame type, it's more like a footer that tells the receiver what to expect next. In this case it's the footer of the data subpacket containing the file name. It's going to be a 32 bit checksum because you're using a "C" type binary header.
ZDLE C TYPE F3/P0 F2/P1 F1/P2 F0/P3 CRC-1 CRC-2 CRC-3 CRC-4
.
Then we have a frame with an undefined type: [18 50 D3 0F F1 11]. Is this a separate frame or is it part of ZCRCW?
That's the CRC for the ZCRCW data subpacket. It's 5 bytes because the first one is 0x10, a control character that needs to be ZDLE escaped. I'm not sure what 0x11 is.
and there's no [XON].
XON is just for Hex headers. You don't use it for a binary header.
ZDLE A TYPE F3/P0 F2/P1 F1/P2 F0/P3 CRC-1 CRC-2
.
So how do I transmit a payload byte that happens to match the value of [ZDLE]?
18 58 (AKA ZDLEE)
18 68 05 DE 02 18 D0
That's the footer of the data subframe. The next 5 bytes are the CRC (last byte is ZDLE encoded)
The ZDLE + ZBIN (0x18 0x41) means the frame is CRC-CCITT(XMODEM 16) with Binary Data.
ZDLE + ZHEX (0x18 0x42) means CRC-CCITT(XMODEM 16) with HEX data.
The HEX data is tricky, since at first some people don't understand it. Every two bytes, the ASCII chars represent one byte in binary. For example, 0x30 0x45 0x41 0x32 means 0x30 0x45, 0x41 0x32, or in ASCII 0 E A 2, then 0x0E, 0xA2. They expand the binary two nibbles to a ASCII representation. I found in several dataloggers that some devices use lower case to represent A~F (a~f) in HEX, it doesn't matter, but on those, you will not find 0x30 0x45 0x41 0x32 (0EA2) but 0x30 0x65 0x61 0x32 (0ea2), it doesn't change a thing, just make it a little bit confuse at first.
And yes, the CRC16 for ZBIN or ZHEX is CRC-CCITT(XMODEM).
The ZDLE ZBIN32 (0x18 0x43) or ZDLE ZBINR32 (0x18 0x44) use CRC-32 calculation.
Noticing that the ZDLE and the following byte are excluded in the CRC calculation.
I am digging into the ZMODEM since I need to "talk" with some Elevators Door Boards, to program a new set of parameters at once, instead using their software to change attribute by attribute. This "talk" could be on the bench instead sitting over the elevator car roof with a notebook. Those boards talk ZMODEM, but as I don't have the packets format they expect, the board still rejecting my file transfer. The boards send 0x2a 0x2A 0x18 0x42 0x30 0x31 0x30 (x6) + crc, the Tera Terminal transfering the file in ZMODEM send to the board 0x2A 0x2A 0x18 0x42 0x30 0x30 ... + CRC, I don't know why this 00 or 01 after the 0x4B means. The PC send this and the filename and some file attributes. The board after few seconds answer with "No File received"...