I have a zip file over 15gb with an error. I need to modify a bash code to handle the error (not to correct it just to "catch it"), the thing is that the size of the file makes me go very slow when I want to test something, I need to recreate the same error but with a file of like 1gb so I can do testing faster, I have tried everything that came to my mind but without success.
This is the error:
$ zip -T AUDIOS.zip
Could not find:AUDIOS.z01
Hit c (change path to where this split file is)
q (abort archive - quit)
or ENTER(Try reading this split again)
The error is obvious but I think is the result of a corrupted file because the zip file is not really a multipart zip so I assume there are some errors in the headers of the file or something like that that makes the zip command think that it is a multipart file.
Note:If I use 7z to unzip the file everything goes well, it says that there are error headers but the file uncompress the right way anyways.
I wrote the last two paragraphs just to give some context, what I really need is to recreate the error in a smaller zip file so when I execute zip -T zipfile it returns the same as the big file.
The link posted by #tink does match the error you are seeing. The unknown is whether the corruption in your zip file matches the one in the link. There may be multiple reason that can trigger that error message. Not sure if that is important to you.
If you want to verify that the details in the link match with your zip file get a copy of zipdetails from here and run this
perl zipdetails -v AUDIOS.zip
It will provide a detailed output of the zip file. For this error you need to post the data at the end of the zip file that looks like this. I've highlighted the part in the link that is the cause of the problem.
00BF 0004 50 4B 06 06 ZIP64 END CENTRAL DIR 06064B50
RECORD
00C3 0008 2C 00 00 00 Size of record 000000000000002C
00 00 00 00
00CB 0001 1E Created Zip Spec 1E '3.0'
00CC 0001 03 Created OS 03 'Unix'
00CD 0001 2D Extract Zip Spec 2D '4.5'
00CE 0001 00 Extract OS 00 'MS-DOS'
00CF 0004 00 00 00 00 Number of this disk 00000000
00D3 0004 00 00 00 00 Central Dir Disk no 00000000
00D7 0008 01 00 00 00 Entries in this disk 0000000000000001
00 00 00 00
00DF 0008 01 00 00 00 Total Entries 0000000000000001
00 00 00 00
00E7 0008 5A 00 00 00 Size of Central Dir 000000000000005A
00 00 00 00
00EF 0008 65 00 00 00 Offset to Central dir 0000000000000065
00 00 00 00
00F7 0004 50 4B 06 07 ZIP64 END CENTRAL DIR 07064B50
LOCATOR
00FB 0004 00 00 00 00 Central Dir Disk no 00000000
00FF 0008 BF 00 00 00 Offset to Central dir 00000000000000BF
00 00 00 00
0107 0004 00 00 00 00 Total no of Disks 00000000 <==== This value should be 1
010B 0004 50 4B 05 06 END CENTRAL HEADER 06054B50
010F 0002 00 00 Number of this disk 0000
0111 0002 00 00 Central Dir Disk no 0000
0113 0002 01 00 Entries in this disk 0001
0115 0002 01 00 Total Entries 0001
0117 0004 5A 00 00 00 Size of Central Dir 0000005A
011B 0004 FF FF FF FF Offset to Central Dir FFFFFFFF
011F 0002 00 00 Comment Length 0000
There is a test file zero.zip that I use in the test harness for zipdetails to check this particular error. If this is indeed the same as you corruption you can just use zero.zip for your testing.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed last year.
Improve this question
I have generated an AES-256 ciphertext in hex (cipher.hex) that I am trying to decrypt:
53 9B 33 3B 39 70 6D 14 90 28 CF E1 D9 D4 A4 07
with a corresponding 256-bit key in hex (key.hex):
80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01
I started by attempting to decode both into base64 using the following commands:
base64 cipher.hex > input;
base64 key.hex > key;
and lastly, passed them into openssl as seen below:
openssl aes-256-cbc -d -a -pass file:key -in input -out out
at which point I get back "bad magic number" from openssl with no guidance on what went wrong.
Is there something wrong with the procedure I'm following? I have also tried converting the hex values into binary... really not sure what input to give openssl from hex, let alone if base64 decoding a newline-wrapped hexidecimal is valid.
You're trying to use CBC and password-based encryption on something that is encrypted with ECB and a raw key. You're also base64 encoding the hex characters, not the actual bytes. You're encoding <ASCII for 5><ASCII for 3><ASCII for space>... You want to encode 0x53, 0x9B, 0x33, etc. Base64 isn't needed here in any case.
First, convert your hex input to raw data (not base64). I like xxd for this:
echo "53 9B 33 3B 39 70 6D 14 90 28 CF E1 D9 D4 A4 07" | xxd -r -p > input
Next, you need your key in the hex format openssl likes, no spaces or newlines:
KEY=$(echo "80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01" | tr -d '[ \n]'
echo $KEY
# prints 8000000000000000000000000000000000000000000000000000000000000001
And now, you can decrypt the data (aes-256-ecb with a hex key):
openssl aes-256-ecb -d -in input -K $KEY | xxd
This prints:
00000000: 8070 6050 4030 2010 0807 0605 0403 02 .p`P#0 ........
Which is too orderly to be wrong. (0x80, 0x70, 0x60, 0x50, ...)
If this were CBC, there would be an IV, and in most cases a padding block. That said, for a single block with no padding and a NULL IV (which you should never use), CBC is equivalent to ECB, so it is possible to decrypt this as CBC and get the same result:
openssl aes-256-cbc -iv 00000000000000000000000000000000 -d -in input -K $KEY | xxd
The "bad magic number" error is because you used the -pass option, which employs OpenSSL's password key derivation algorithm. Files that have been encrypted that way start with the ASCII sequence Salted__. That's missing, so the input is rejected. (It wouldn't work anyway, since this was encrypted with a raw key.)
This is a great solution as far as it goes:
How to use bluetoothctl like hcitool lescan to report repeated proximity beacons
I want to take it one step further though: instead of printing the beacon detections to the terminal, I want to pipe each detected beacon's MAC address to a command that publishes it to a Mosquitto broker. This would be useful for adding the beacons to HomeAssistant, for example, for presence detection of a key fob attached to a person's house keys. I managed to get this idea to work by modifying the last two lines of the script above but it only publishes the first detection ... then no more. I guess somehow the scan is being turned off or aborted by trying to run the mosquitto_pub command. (something to do with interrupting the thread or subshells????)
Here are the last lines of the script with the MAC addresses of my TWO TEST BEACONS obfuscated.
You can see the full script that I modified at the link above.
) | sed --unbuffered --quiet --expression 's/^.*Device //p' \
| grep --line-buffered -E 'FF:FF::::|FF:GG::::' \
| stdbuf -oL cut -c 1-17 \
| { read topic; mosquitto_pub -d -u username -P password -m ON -t monitor/$topic; }
Note:
| stdbuf -oL cut -c 1-17 \ trims the output down to just the MAC address - e.g. FF:GG::::
The last line publishes the ON command to an unique state topic ending in the beacon's MAC address e.g. 'monitor/FF:GG::::'.
If I remove that last line the detections are added as they occur to the terminal console in an ever growing list as time goes by. But once I add the last line it only works once, then goes dead.
How can the script be made to respond to each detection, not just the first one?
And why exactly does it only respond to the first detection then hang?
The command line tools of BlueZ are not designed for this purpose, so it makes them very difficult to pipe scan results reliably. In my experience, processes die or hang, and you end up with broken pipes.
The C APIs of BlueZ are much more stable. I wrote a simple command line program in C that you could use instead of bluetoothctl for this purpose:
https://gist.github.com/davidgyoung/0a18028b4338ff6cb201fba274502662
That program must be compiled with cc scanner.c -lbluetooth -o scanner, after which you can start the scanner with just scanner. The output of the program will be something like this:
B8:27:EB:1F:93:4D -68 02 01 06 11 06 82 75 25 D9 37 9D D7 8F 5F 4A F4 20 00 00 75 30
71:5C:23:9D:BC:7F -68 02 01 1A 02 0A 0C 0B FF 4C 00 10 06 03 1A 3B D4 B2 EB
B8:27:EB:1F:93:4D -68 02 01 06 11 06 82 75 25 D9 37 9D D7 8F 5F 4A F4 20 00 00 75 30
4A:53:7F:64:71:EC -91 03 03 9F FE 17 16 9F FE 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
69:0D:FF:7B:75:65 -73 02 01 1A 02 0A 0C 0A FF 4C 00 10 05 03 1C 27 BB 63
61:39:71:E9:1D:C9 -93 02 01 1A 02 0A 18 0A FF 4C 00 10 05 01 18 3B 24 12
B8:27:EB:1F:93:4D -68 02 01 06 11 06 82 75 25 D9 37 9D D7 8F 5F 4A F4 20 00 00 75 30
Each line above shows the MAC address of the detected bluetooth device, followed by the signal strength RSSI, then the hex bytes of the advertisement.
I use this program to pipe into other shell scripts, and have found it to be reliable over many days. Feel free to use it if helpful.
I am writing a custom service over BLE to transmit custom characteristics. It has it's own UUID such as '8E400001-B5A3-F393-E0A9-E50E24DCCA9E'.
I have found that a particular android app (BLE client) I wish to support requires that a custom service UUID is included in the advertising packet.
However our peripheral is running via bluez 5.43 on debian and I cannot figure out how to put this UUID in the advertising packet.
I think it will be something like:
hcitool -i hci0 cmd 0x08 0x0008 .... ?
I verified the following commands to be working (BlueZ ver 5.43, Linux 4.9.0-2-amd64 (x86_64)):
hcitool -i hci0 cmd 0x08 0x0008 12 11 07 9E CA DC 24 0E E5 A9 E0 93 F3 A3 B5 01 00 40 8E 00 00 00 00 00 00 00 00 00 00 00 00 00
Explanation:
I assume everything until 0x0008 is clear, if not let me know or look at the description of the LE Set Advertising Data Command in the spec. 0x12 is Advertising_Data_Length, i.e., the number of useful bytes that comes after (until the padding bytes). Now you have to insert the Advertising_Data, which is formatted as in Vol 3 Part C, Section 11, Fig. 11.1 of the spec. So, 0x11 is the length of the AD Structure. Then, 0x07 is the AD Type "Complete List of 128-bit Service Class UUIDs" (see here). Finally comes the UUID. Notice that everything is little endian.
Then send the LE Set Advertising Parameters Commands, for example:
hcitool -i hci0 cmd 0x08 0x0006 00 08 00 08 00 00 00 00 00 00 00 00 00 07 00
Then enable advertising by sending the LE Advertising Enable command:
hcitool -i hci0 cmd 0x08 0x000A 01
See btmon output here.
There is a well known blog post going around on how to set a usb bluetooth 4 dongle to be an iBeacon. It boils down to this magical command:
sudo hcitool -i hci0 cmd 0x08 0x0008 1e 02 01 1a 1a ff 4c 00 02 15 e2 c5 6d b5 df fb 48 d2 b0 60 d0 f5 a7 10 96 e0 00 00 00 00 c5 00 00 00 00 00 00 00 00 00 00 00 00 00
The issue with this example is that it is so opaque it's hard to use it in any more general format. I've been able to break it apart a bit:
sudo hcitool -i hci0 cmd
sends an hci command to the hci0 device
0x08 0x0008
is just magic to set the ad package, other stackoverflow commands have said "just use it, don't ask
1e
is the length of the ENTIRE following data packet in bytes
02 01 1a 1a
Are flags to set up the ad packet (details on request)
ff 4c 00 ...
is the 'company specific data' that encodes the iBeacon info
What I've tried to do is replace the "FF ..." bytes with the opcodes for setting the NAME parameter "04 09 41 42 43" (which should set it to ABC) but that doesn't work.
I'm surprised the hcitool doesn't give us some examples on how to set the ad packet as this would be very useful in setting all sorts of other params (like TEMP or POWER). Has anyone else had any experience in using hcitool to set things like NAME?
Late reply, but somebody might find this useful. I found it as I was looking around for solutions myself when using hcitool.
If you use hcitool cmd --help it will tell you something like this cmd <ogf> <ocf> .... It helps to look at the Bluetooth Core Specification to find out what 0x08 and 0x0008 would be for OGF and OCF. Specifically Vol. 2, Part E, 7.8
For the LE Controller Commands, the OGF code is defined as 0x08
and for the OCF of 0x0008
Advertising_Data_Length, Advertising_Data
So basically, with 0x08 0x0008 you say you are setting (in the LE Controller) the length of the data that is sent. As for the name, since the length of the BLE advertisement packet is 31 bytes (1E), you need to send the whole 31 bytes. So if you only have ABC as the name, setting 04 09 41 42 43 is correct, but that's only five bytes. For 31 you need to add 00 26 times. Just be careful you don't add too much or too little.
Also, I wasn't under the impression that BLE ad. packets are of fixed 31 byte size, but they are at least for hcitool. It doesn't work when you specifically set the outgoing size to something smaller than 1E.
No. None of this answers is correct and clean.
1) BLE have a separate command set. "LE Set Advertising Data" command must be used (see 7.8.7 vol 2 part E).
2) LE Set Advertising Data is much complex than what explained above. There are at least 2 1-octet length fields, packet must be 32 bytes length, zero padded, but the first length byte must not be 0x1e (31) but the length of the significant used part, containing one or more headers. Each header still contains a length, one AS Type byte (0x09 for set local name) and the name.
SET LE LOCAL NAME:
hciconfig hci0 down
hciconfig hci0 up
hcitool -i hci0 cmd 0x08 0x0008 0c 0b 09 6c 69 6e 6b 6d 6f 74 69 6f 6e 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
hciconfig hci0 leadv 0
0x0c : length of the following group of headers
0x0b : length of this header
0x09 : AD Type for complete name
rest 0x0a length is the name
Check out this answer to a similar question. It basically describes how you can download the giant Bluetooth Core Spec document, and read through all the commands that it offers you. You can use the hcitool command to execute any of these commands if you can just figure out the right format (and figure out what the commands actually do!)
Major caveat: I have not tried setting the name myself, but glancing at the spec, it looks like this is described on page 482 of the spec in section "7.3.11 Write Local Name Command". According to this the command consists of:
OCF: 0x0013
Name (up to 248 bytes)
So I would try a command like this:
hcitool -i hci0 cmd 0x08 0x0013 41 42 43
One other tip: When you issue commands like this, try running hcidump & so the command executes in the background. Then, you can enter experimental hcitool commands (or even hciconfig commands) and see annotated details about what (human readable) commands executed and what errors occurred, if any.
Using the above tip, you can also try executing hciconfig name abc to set the local name using that command line tool, while you are executing a hcidump & in the background. This should show you the proper hcitool command values to use.
It would appear you need to use two commands rather than one. The iBeacon data is contained in the "LE Set Advertising Data" data and has been mentioned elsewhere in this post. To see a BLE friendly name you can use an additional command "LE Set Scan Response Data", this requires the receiver to scan your device (rather than passively read the advertising packet). So you can combine Angelo's example with this one to set the device as an iBeacon and set the "friendly name" which is the Scan Response data
sudo hcitool -i hci0 cmd 0x08 0x0008 1E 02 01 1A 1A FF 4C 00 02 15 E2 0A 39 F4 73 F5 4B C4 A1 2F 17 D1 AD 07 A9 61 00 00 00 00 C8 00
sudo hcitool -i hci0 cmd 0x08 0x0009 0c 0b 09 6c 69 6e 6b 6d 6f 74 69 6f 6e 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
This worked for me using an Ubuntu box with a BLE dongle and then scanning for the beacon using this android BLE scanning app
I am trying to write a program to dump option/expansion ROMs on linux. I already have the necessary PCI port IO to get an expansion ROM's base address out of the PCI configuration data at offset 0x30 and make it enabled, but when I try to access the base in memory I get a segfault. So I am trying to understand what is happening when you do the "echo 1 > rom" from a linux command line, since after that the rom seems plenty accessible (see here for a little more context: http://etherboot.org/wiki/romdumping)
E.g. let's say I do the following:
lspci
01:00.0 VGA compatible controller: ATI Technologies Inc RV370 5B60 [Radeon X300 (PCIE)]
cd /sys/bus/pci/devices/0000:01:00.0
lspci -x -v -s 01:00.0
01:00.0 VGA compatible controller: ATI Technologies Inc RV370 5B60 [Radeon X300 (PCIE)]
Subsystem: ATI Technologies Inc Device 0402
Flags: bus master, fast devsel, latency 0, IRQ 27
Memory at d0000000 (32-bit, prefetchable) [size=128M]
I/O ports at dc00 [size=256]
Memory at dfde0000 (32-bit, non-prefetchable) [size=64K]
Expansion ROM at dfe00000 [disabled] [size=128K]
Capabilities: [50] Power Management version 2
Capabilities: [58] Express Endpoint, MSI 00
Capabilities: [80] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable+
Capabilities: [100] Advanced Error Reporting <?>
Kernel driver in use: radeon
Kernel modules: radeonfb, radeon
00: 02 10 60 5b 07 04 10 00 00 00 00 03 10 00 80 00
10: 08 00 00 d0 01 dc 00 00 00 00 de df 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 02 10 02 04
30: 00 00 e0 df 50 00 00 00 00 00 00 00 0b 01 00 00
Note the line that says "Expansion ROM at dfe00000 [disabled] [size=128K]"
Now per the PCI spec I can see that the bottom bit should be set to 1 to enable the expansion ROM, so I did a read and write back of 0xdfe00001, and then I get
01:00.0 VGA compatible controller: ATI Technologies Inc RV370 5B60 [Radeon X300 (PCIE)]
Subsystem: ATI Technologies Inc Device 0402
Flags: bus master, fast devsel, latency 0, IRQ 27
Memory at d0000000 (32-bit, prefetchable) [size=128M]
I/O ports at dc00 [size=256]
Memory at dfde0000 (32-bit, non-prefetchable) [size=64K]
Expansion ROM at dfe00000 [size=128K]
Capabilities: [50] Power Management version 2
Capabilities: [58] Express Endpoint, MSI 00
Capabilities: [80] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable+
Capabilities: [100] Advanced Error Reporting <?>
Kernel driver in use: radeon
Kernel modules: radeonfb, radeon
00: 02 10 60 5b 07 04 10 00 00 00 00 03 10 00 80 00
10: 08 00 00 d0 01 dc 00 00 00 00 de df 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 02 10 02 04
30: 01 00 e0 df 50 00 00 00 00 00 00 00 0b 01 00 00
(note the 0x01 at offset 0x30 and the line now saying "Expansion ROM at dfe00000 [size=128K]").
But I can't access 0xdfe00000. Meanwhile, when O do the "echo 1 > rom" in the command line, it does NOT change the line to remove the "disabled", and really it doesn't make any change at all to the output of lspci. So what is the "echo 1 > rom" doing that I'm not, which makes it subsequently possible to do the "dd if=rom of=/tmp/rom"?
Much appreciated
I've tried to get the exROM contents from an NVidia VGA board (NVS310) - from the Linux command line :-)
This is what I did, approximately:
From lspci -vvn, I found out that my board is at 0000:02:00.0 and its exROM is at 0xf7000000 size 512 kB (and its address decoder is disabled in the exROM BAR = via bit 0 in PCI config space reg #c0 == offset 0x30).
I then used setpci to toggle that "exROM BAR enable bit" in the PCI config space. Notice the notation value:mask in the "write" form of setpci:
setpci --dumpregs
setpci -s 0000:02:00.0 ROM_ADDRESS
setpci -s 0000:02:00.0 ROM_ADDRESS=00000001:00000001
setpci -s 0000:02:00.0 ROM_ADDRESS
lspci -vvn -s 0000:02:00.0
dd if=/dev/mem of=vgabios.bin bs=64k skip=63232 count=8
Which resulted in some semi-plausible looking binary data in a file 512 kB long.
Access via /sys/bus/pci/devices/0000:02:00.0/rom did not work, even after I wrote a "1" into that pseudo-file.
rom is a special file usually read-protected unless '1' is written to it, from sysfs-pci.txt:
The 'rom' file is special in that it provides read-only access to the device's
ROM file, if available. It's disabled by default, however, so applications
should write the string "1" to the file to enable it before attempting a read
call, and disable it following the access by writing "0" to the file. Note
that the device must be enabled for a rom read to return data successfully.
In the event a driver is not bound to the device, it can be enabled using the
'enable' file, documented above.
So it looks like 'echo 1 > rom' really only enables reading the rom through sysfs.
Maybe you can't access the rom through its bar because it's not mapped to dfe00000 by default? Not sure, worth checking though.
sudo echo 1 >rom for me (a zsh user) outputs permission denied: rom and the same seems to be the case when using bash, when using su and then typing echo 1 >rom works. If you are scripting you may need to thus do sudo su -c 'echo 1 >rom' or pkexec --keep-cwd su -c 'echo 1 >rom'. Perhaps it can also depend on your system (security) configuration. All I can say in that regard is that I use arch linux, without mitigations and linux-zen 5.19