I am experiencing some issues when using mmls command after having created an image with dcfldd/guymager in some particular situations. Usually this approach seems to be working fine to create physical images of devices, but with some USBs (working fine and undamaged) I manage to create the .dd disk image file, but then it won't be opened by mmls, nor fsstat.
fls does open the file system structure, but it seems like it won't show me any unallocated files just as if this was a logical image.
This is the command run to create a disk image using dcfldd:
sudo dcfldd if=/dev/sda hash=sha256 hashlog=usb.sha256hash of=./usb.dd bs=512 conv=noerror,sync,notrunc
Also, this is the output of usb.info, generated by guymager:
GUYMAGER ACQUISITION INFO FILE
==============================
Guymager
========
Version : 0.8.13-1
Version timestamp : 2022-05-11-00.00.00 UTC
Compiled with : gcc 12.1.1 20220507 (Red Hat 12.1.1-1)
libewf version : 20140812 (not used as Guymager is configured to use its own EWF module)
libguytools version: 2.0.2
Host name : lucafedora
Domain name : (none)
System : Linux lucafedora 6.1.7-100.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Jan 18 18:37:43 UTC 2023 x86_64
Device information
==================
Command executed: bash -c "search="`basename /dev/sda`: H..t P.......d A..a de.....d" && dmesg | grep -A3 "$search" || echo "No kernel HPA messages for /dev/sda""
Information returned:
----------------------------------------------------------------------------------------------------
No kernel HPA messages for /dev/sda
Command executed: bash -c "smartctl -s on /dev/sda ; smartctl -a /dev/sda"
Information returned:
----------------------------------------------------------------------------------------------------
/usr/bin/bash: line 1: smartctl: command not found
/usr/bin/bash: line 1: smartctl: command not found
Command executed: bash -c "hdparm -I /dev/sda"
Information returned:
----------------------------------------------------------------------------------------------------
/usr/bin/bash: line 1: hdparm: command not found
Command executed: bash -c "CIDFILE=/sys/block/$(basename /dev/sda)/device/cid; echo -n "CID: " ; if [ -e $CIDFILE ] ; then cat $CIDFILE ; else echo "not available" ; fi "
Information returned:
----------------------------------------------------------------------------------------------------
CID: not available
Hidden areas: unknown
Acquisition
===========
Linux device : /dev/sda
Device size : 8053063680 (8.1GB)
Format : Linux dd raw image - file extension is .dd
Image path and file name: /home/HOMEDIR/case_usb/usb.dd
Info path and file name: /home/HOMEDIR/case_usb/usb.info
Hash calculation : SHA-256
Source verification : on
Image verification : on
No bad sectors encountered during acquisition.
No bad sectors encountered during verification.
State: Finished successfully
MD5 hash : --
MD5 hash verified source : --
MD5 hash verified image : --
SHA1 hash : --
SHA1 hash verified source : --
SHA1 hash verified image : --
SHA256 hash : 7285a8b0a2b472a8f120c4ca4308a94a3aaa3e308a1dd86e3670041b07c27e76
SHA256 hash verified source: 7285a8b0a2b472a8f120c4ca4308a94a3aaa3e308a1dd86e3670041b07c27e76
SHA256 hash verified image : 7285a8b0a2b472a8f120c4ca4308a94a3aaa3e308a1dd86e3670041b07c27e76
Source verification OK. The device delivered the same data during acquisition and verification.
Image verification OK. The image contains exactely the data that was written.
Acquisition started : 2023-01-28 12:27:07 (ISO format YYYY-MM-DD HH:MM:SS)
Verification started: 2023-01-28 12:30:11
Ended : 2023-01-28 12:35:24 (0 hours, 8 minutes and 16 seconds)
Acquisition speed : 41.97 MByte/s (0 hours, 3 minutes and 3 seconds)
Verification speed : 24.62 MByte/s (0 hours, 5 minutes and 12 seconds)
Generated image files and their MD5 hashes
==========================================
No MD5 hashes available (configuration parameter CalcImageFileMD5 is off)
MD5 Image file
n/a usb.dd
Worth to mention that when mmls is run against usb.dd it produces no output whatsoever. I have to forcefully add -v option for it to spit out this kind of information:
tsk_img_open: Type: 0 NumImg: 1 Img1: usb.dd
aff_open: Error determining type of file: usb.dd
aff_open: Success
Error opening vmdk file
Error checking file signature for vhd file
tsk_img_findFiles: usb.dd found
tsk_img_findFiles: 1 total segments found
raw_open: segment: 0 size: 8053063680 max offset: 8053063680 path: usb.dd
dos_load_prim: Table Sector: 0
raw_read: byte offset: 0 len: 65536
raw_read: found in image 0 relative offset: 0 len: 65536
raw_read_segment: opening file into slot 0: usb.dd
dos_load_prim_table: Testing FAT/NTFS conditions
dos_load_prim_table: MSDOS OEM name exists
bsd_load_table: Table Sector: 1
gpt_load_table: Sector: 1
gpt_open: Trying other sector sizes
gpt_open: Trying sector size: 512
gpt_load_table: Sector: 1
gpt_open: Trying sector size: 1024
gpt_load_table: Sector: 1
gpt_open: Trying sector size: 2048
gpt_load_table: Sector: 1
gpt_open: Trying sector size: 4096
gpt_load_table: Sector: 1
gpt_open: Trying sector size: 8192
gpt_load_table: Sector: 1
gpt_open: Trying secondary table
gpt_load_table: Sector: 15728639
raw_read: byte offset: 8053063168 len: 512
raw_read: found in image 0 relative offset: 8053063168 len: 512
gpt_open: Trying secondary table sector size: 512
gpt_load_table: Sector: 15728639
gpt_open: Trying secondary table sector size: 1024
gpt_load_table: Sector: 7864319
raw_read: byte offset: 8053062656 len: 1024
raw_read: found in image 0 relative offset: 8053062656 len: 1024
gpt_open: Trying secondary table sector size: 2048
gpt_load_table: Sector: 3932159
raw_read: byte offset: 8053061632 len: 2048
raw_read: found in image 0 relative offset: 8053061632 len: 2048
gpt_open: Trying secondary table sector size: 4096
gpt_load_table: Sector: 1966079
raw_read: byte offset: 8053059584 len: 4096
raw_read: found in image 0 relative offset: 8053059584 len: 4096
gpt_open: Trying secondary table sector size: 8192
gpt_load_table: Sector: 983039
raw_read: byte offset: 8053055488 len: 8192
raw_read: found in image 0 relative offset: 8053055488 len: 8192
sun_load_table: Trying sector: 0
sun_load_table: Trying sector: 1
mac_load_table: Sector: 1
mac_load: Missing initial magic value
mac_open: Trying 4096-byte sector size instead of 512-byte
mac_load_table: Sector: 1
mac_load: Missing initial magic value
I'm trying to use sfdisk to create an image file inside a docker container and I can use below command without any problem:
root#c8e9be2eb26f:/# sfdisk bbb_image.img << EOF
> 1M,48M,0xE,*
> ,,,-
> EOF
Checking that no-one is using this disk right now ... OK
Disk bbb_image.img: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
>>> Created a new DOS disklabel with disk identifier 0x34b8e793.
bbb_image.img1: Created a new partition 1 of type 'W95 FAT16 (LBA)' and of size 48 MiB.
bbb_image.img2: Created a new partition 2 of type 'Linux' and of size 975 MiB.
bbb_image.img3: Done.
New situation:
Disklabel type: dos
Disk identifier: 0x34b8e793
Device Boot Start End Sectors Size Id Type
bbb_image.img1 * 2048 100351 98304 48M e W95 FAT16 (LBA)
bbb_image.img2 100352 2097151 1996800 975M 83 Linux
The partition table has been altered.
Syncing disks.
Now in my Dockerfile this seems doesn't work and reproduce incomplete results:
RUN sfdisk bbb_image.img << "EOF\n\
1M,48M,0xE,*\n\
,,,-\n\
EOF"
And reproduce this in the console which is wrong:
Step 4/4 : RUN sfdisk bbb_image.img << "EOF\n1M,48M,0xE,*\n,,,-\nEOF\n"
---> Running in afc86ffef92a
Checking that no-one is using this disk right now ... OK
Disk bbb_image.img: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
>>> Done.
New situation:
ERROR: Service 'test' failed to build: The command '/bin/sh -c sfdisk bbb_image.img << "EOF\n1M,48M,0xE,*\n,,,-\nEOF\n"' returned a non-zero code: 1
I'm not sure how to handle the EOF in the Dockerfile.
Maybe try this one:
RUN sfdisk bbb_image.img << "\n1M,48M,0xE,*\n,,,-\n"
I have this SSD that was in another Ovirt cluster that I destroyed, and now I'm trying to re-purpose this disk but can't seem to clear it. I have tried wipefs and dd but it still shows up in fdisk -l. How can I clear all lvms from this disk?
sudo fdisk -l
Disk /dev/sda: 3.5 TiB, 3840755982336 bytes, 7501476528 sectors
Disk model: SAMSUNG MZ7LH3T8
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/mmcblk0: 58.25 GiB, 62537072640 bytes, 122142720 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 61C206C7-69A3-4C51-9215-1C0B45E32661
Device Start End Sectors Size Type
/dev/mmcblk0p1 2048 788479 786432 384M Linux filesystem
/dev/mmcblk0p2 788480 1048575 260096 127M EFI System
/dev/mmcblk0p3 1048576 1050623 2048 1M BIOS boot
/dev/mmcblk0p4 1050624 122142686 121092063 57.8G Linux filesystem
[core#h2-master-01 ~]$ sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 3.5T 0 disk
ââgluster_vg_sdc-gluster_thinpool_gluster_vg_sdc_tmeta 253:0 0 15.8G 0 lvm
â ââgluster_vg_sdc-gluster_thinpool_gluster_vg_sdc-tpool 253:2 0 3.5T 0 lvm
â ââgluster_vg_sdc-gluster_thinpool_gluster_vg_sdc 253:3 0 3.5T 1 lvm
â ââgluster_vg_sdc-gluster_lv_data 253:4 0 3.4T 0 lvm
ââgluster_vg_sdc-gluster_thinpool_gluster_vg_sdc_tdata 253:1 0 3.5T 0 lvm
ââgluster_vg_sdc-gluster_thinpool_gluster_vg_sdc-tpool 253:2 0 3.5T 0 lvm
ââgluster_vg_sdc-gluster_thinpool_gluster_vg_sdc 253:3 0 3.5T 1 lvm
ââgluster_vg_sdc-gluster_lv_data 253:4 0 3.4T 0 lvm
mmcblk0 179:0 0 58.2G 0 disk
ââmmcblk0p1 179:1 0 384M 0 part /boot
ââmmcblk0p2 179:2 0 127M 0 part /boot/efi
ââmmcblk0p3 179:3 0 1M 0 part
ââmmcblk0p4 179:4 0 57.8G 0 part /sysroot
mmcblk0boot0 179:8 0 4M 1 disk
mmcblk0boot1 179:16 0 4M 1 disk
[core#h2-master-01 ~]$ sudo lvdisplay
[core#h2-master-01 ~]$ sudo vgdisplay
[core#h2-master-01 ~]$ sudo pvdisplay
wipefs -a /dev/sda with a reboot seemed looks to have cleared the disk. Thanks all!
So I tried this:
root#kali:~/Desktop/fmk# binwalk upgrade-2.4.0.bin
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
512 0x200 LZMA compressed data, properties: 0x6D, dictionary size: 8388608 bytes, uncompressed size: 2805816 bytes
927576 0xE2758 Squashfs filesystem, little endian, version 4.0, compression:xz, size: 12316692 bytes, 2963 inodes, blocksize: 262144 bytes, created: 2015-08-04 02:40:49
And then I used the following dd:
sudo dd if=upgrade-2.4.0.bin of=pineapple.squashfs bs=1 count=12316692
And I can't unsquashfs pineapple.squashfs.
Can't find a SQUASHFS superblock on pineapple.squashfs
You have to set the offset where the squashfs is
Usage: dd [OPERAND]...
or: dd OPTION
Copy a file, converting and formatting according to the operands.
bs=BYTES read and write up to BYTES bytes at a time
cbs=BYTES convert BYTES bytes at a time
conv=CONVS convert the file as per the comma separated symbol list
count=N copy only N input blocks
ibs=BYTES read up to BYTES bytes at a time (default: 512)
if=FILE read from FILE instead of stdin
iflag=FLAGS read as per the comma separated symbol list
obs=BYTES write BYTES bytes at a time (default: 512)
of=FILE write to FILE instead of stdout
oflag=FLAGS write as per the comma separated symbol list
seek=N skip N obs-sized blocks at start of output
skip=N skip N ibs-sized blocks at start of input
status=LEVEL The LEVEL of information to print to stderr;
'none' suppresses everything but error messages,
'noxfer' suppresses the final transfer statistics,
'progress' shows periodic transfer statistics
...
So, to extract the filesystem
dd if=upgrade-2.4.0.bin of=pineapple.squashfs bs=1 skip=927576
I did it with:
binwalk -Me upgrade-2.4.0.bin
I have an 8 core CPU with 8GB of RAM, and I'm creating a batch file to automate 7-zip CLI in exhausting most parameters and variables to compress the same set of files with the ultimate goal of finding the strongest combination of parameters and variables that result in the smallest archive size possible.
This is very time consuming by nature especially when the set of files to be processed is in gigabytes. I need a way not only to automate but to speed up this whole process.
7-zip works with different compression algorithms, some are single-threaded only, and some are multi-threaded, some do not require much amount of memory, and some require huge amounts of it and could even surpass the 8GB barrier. I've already successfully created an automated batch that works in sequence which exclude combinations requiring more than 8GB of memory.
I've split the different compression algorithms in several batches to simplify the whole process. For example, compression in PPMd as a 7z archive uses 1-thread and up to 1024MB. This is my current batch:
#echo off
echo mem=1m 2m 3m 4m 6m 8m 12m 16m 24m 32m 48m 64m 96m 128m 192m 256m 384m 512m 768m 1024m
echo o=2 3 4 5 6 7 8 10 12 14 16 20 24 28 32
echo s=off 1m 2m 4m 8m 16m 32m 64m 128m 256m 512m 1g 2g 4g 8g 16g 32g 64g on
echo x=1 3 5 7 9
for %%x IN (9) DO for %%d IN (1024m 768m 512m 384m 256m 192m 128m 96m 64m 48m 32m 24m 16m 12m 8m 6m 4m 3m 2m 1m) DO for %%w IN (32 28 24 20 16 14 12 10 8 7 6 5 4 3 2) DO for %%s IN (on) DO 7z.exe a teste.resultado\%%xx.ppmd.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -m0=PPMd:mem=%%d:o=%%w -ms=%%s
exit
x, s, o and mem are parameters, and what's after each of them are the variables which 7z.exe will work with. x and s in this case are of no concern, they mean compression strength and solid block size for the archive.
That batch will work fine, but is limited to running only 1 instance of 7z.exe at a time and now I'm looking for a way to make it run more 7z.exe instances in parallel but without exceeding 8GB of RAM or 8 threads at once, whichever comes first, before proceeding to do the next ones in the sequence.
How can I improve this? I have some ideas but I don't know how to make them work in a batch. I was thinking of 2 other variables that won't interact with the 7z processes but would control when the next 7z instance would start. One variable would keep track of how many threads are currently in use and another would track how much memory are in use. Could that work?
Edit:
Sorry, I need to add details, I'm new to this posting style. Following this answer - https://stackoverflow.com/a/19481253/2896127 - I mentioned 8 batches were created and that 7z.PPMd batch was one of them. Maybe listing all the batches and how 7z deals with the parameters will give a better insight on the whole issue. I'll start with the simple ones:
7z.PPMd - 1 fully utilized thread and dictionary dependant 32m-1055m memory usage per instance.
7z.BZip2 - 8 fully utilized threads and fixed 109m memory usage per instance.
zip.Bzip2 - 8 partially utilized threads and fixed 336m memory usage per instance.
zip.Deflate - 8 partially utilized threads and fixed 260m memory usage per instance.
zip.PPMd - 8 partially utilized threads and dictionary dependant 280m-2320m memory usage per instance.
What I mean with partially utilized threads is that, while I assign 8 threads to be used by each 7.exe instance, the algorithm can do variable CPU usage at a randomly fashion, out of my control, unpredictable, but the limitation is set there - no more than 8 threads. In the case of 8 fully utilized threads, it means that on my 8 core CPU, each instance is utilizing 100% of CPU.
The most complex ones - 7z.LZMA, 7z.LZMA2, zip.LZMA - will need to be explained in detail but I am running short on time now. I'll be back to edit the LZMA part whenever I have more free time.
Thanks again.
EDIT: Adding in LZMA part.
7z.LZMA - each instance is n-threaded, ranging from 1 to 2:
1 fully utilized thread, dictionary dependant, 64k to 512m:
64k dictionary uses 32m memory
...
512m dictionary uses 5407m memory
excluded range: 768m to 1024m (above the limit of 8192m memory available)
2 partially utilized threads, dictionary dependant, 64k to 512m:
64k dictionary uses 38m memory
...
512m dictionary uses 5413m memory
excluded range: 768m to 1024m (above the limit of 8192m memory available)
7z.LZMA2 - each instance is n-threaded, ranging from 1 to 8:
1 fully utilized thread, dictionary dependant, 64k to 512m:
64k dictionary uses 32m memory
...
512m dictionary uses 5407m memory
excluded range: 768m to 1024m (above the limit of 8192m memory available)
2 or 3 partially utilized threads, dictionary dependant, 64k to 512m:
64k dictionary uses 38m memory
...
512m dictionary uses 5413m memory
excluded range: 768m to 1024m (above the limit of 8192m memory available)
4 or 5 partially utilized threads, dictionary dependant, 64k to 256m:
64k dictionary uses 51m memory
...
256m dictionary uses 5677m memory
excluded range: 384m to 1024m (above the limit of 8192m memory available)
6 or 7 partially utilized threads, dictionary dependant, 64k to 192m:
64k dictionary uses 62m memory
...
192m dictionary uses 6965m memory
excluded range: 256m to 1024m (above the limit of 8192m memory available)
8 partially utilized threads, dictionary dependant, 64k to 128m:
64k dictionary uses 72m memory
...
128m dictionary uses 6717m memory
excluded range: 192m to 1024m (above the limit of 8192m memory available)
zip.LZMA - each instance is n-threaded, ranging from 1 to 8:
1 fully utilized thread, dictionary dependant, 64k to 512m:
64k dictionary uses 3m memory
...
512m dictionary uses 5378m memory
excluded range: 768m to 1024m (above the limit of 8192m memory available)
2 or 3 partially utilized threads, dictionary dependant, 64k to 512m:
64k dictionary uses 9m memory
...
512m dictionary uses 5384m memory
excluded range: 768m to 1024m (above the limit of 8192m memory available)
4 or 5 partially utilized threads, dictionary dependant, 64k to 256m:
64k dictionary uses 82m memory
...
256m dictionary uses 5456m memory
excluded range: 384m to 1024m (above the limit of 8192m memory available)
6 or 7 partially utilized threads, dictionary dependant, 64k to 256m:
64k dictionary uses 123m memory
...
256m dictionary uses 8184m (very close to the limit though, I may consider excluding it)
excluded range: 384m to 1024m (above the limit of 8192m memory available)
8 partially utilized threads, dictionary dependant, 64k to 128m:
64k dictionary uses 164m memory
...
128m dictionary uses 5536m memory
excluded range: 192m to 1024m (above the limit of 8192m memory available)
I'm trying to understand the behaviour of the commands with nul in them. I don't quite understand what's happening during that part, what those symbols ^ > ^&1 "" are meant to say.
2>nul del %lock%!nextProc!
%= Redirect the lock handle to the lock file. The CMD process will =%
%= maintain an exclusive lock on the lock file until the process ends. =%
start /b "" cmd /c %lockHandle%^>"%lock%!nextProc!" 2^>^&1 !cpu%%N! !cmd!
)
set "launch="
Then later on, at the :wait code:
) 9>>"%lock%%%N"
) 2>nul
if %endCount% lss %startCount% (
1>nul 2>nul ping /n 2 ::1
goto :wait
)
2>nul del %lock%*
EDIT 2 (29-10-2013): Adding the current point of the situation.
After trial and error research, complemented with step by step notes of what's happening, I was able to understand the behaviour above. I simplified the line with start command to this:
start /b /low cmd /c !cmd!>"%lock%!nextProc!"
Though it works, I still don't understand the meaning of 1^>"filename" 2^>^&1 'command'. I know it is related to writing text in the filename what would otherwise be displayed to me. In this case, it would show all of 7z.exe text but written in the file. Until 7z.exe instance finishes its job, nothing is written in the file, but the file already exists, yet at the same time doesn't exist. When 7z.exe actually finishes, the file is finalized and this time it exists for the next part of the script.
Now I can understand the processing behaviour of the suggested script and I'm complementing it with something of my own - I am trying to implement all batches into "one batch do it all" script. In the simplified version, this is it:
echo 8 threads - maxproc=1
for %%x IN (9) DO for %%t IN (8) DO for %%d IN (900k) DO for %%s IN (on) DO 7z.exe a teste.resultado\%%xx.bzip2.%%tt.%%dd.%%ss.7z .\teste.original\* -mx=%%x -ms=%%s -m0=BZip2:d=%%d:mt=%%t
for %%x IN (9) DO for %%t IN (8) DO for %%d IN (900k) DO 7z.exe a teste.resultado\%%xx.bzip2.%%tt.%%dd.zip .\teste.original\* -mx=%%x -mm=BZip2:d=%%d -mmt=%%t
for %%x IN (9) DO for %%t IN (8) DO for %%w IN (257 256 192 128 96 64 48 32 24 16 12 8) DO 7z.exe a teste.resultado\%%xx.deflate64.%%tt.%%ww.zip .\teste.original\* -mx=%%x -mm=deflate64:fb=%%w -mmt=%%t
for %%x IN (9) DO for %%t IN (8) DO for %%w IN (258 256 192 128 96 64 48 32 24 16 12 8) DO 7z.exe a teste.resultado\%%xx.deflate.%%tt.%%ww.zip .\teste.original\* -mx=%%x -mm=deflate:fb=%%w -mmt=%%t
for %%x IN (9) DO for %%t IN (8) DO for %%d IN (256m 128m 64m 32m 16m 8m 4m 2m 1m) DO for %%w IN (16 15 14 13 12 11 10 9 8 7 6 5 4 3 2) DO 7z.exe a teste.resultado\%%xx.ppmd.%%tt.%%dd.%%ww.zip .\teste.original\* -mx=%%x -mm=PPMd:mem=%%d:o=%%w -mmt=%%t
echo 4 threads - maxproc=2
for %%x IN (9) DO for %%t IN (4) DO for %%d IN (256m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO for %%s IN (on) DO 7z.exe a teste.resultado\%%xx.lzma2.%%tt.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -ms=%%s -m0=lzma2:d=%%d:fb=%%w -mmt=%%t
echo 2 threads - maxproc=4
for %%x IN (9) DO for %%t IN (2) DO for %%d IN (512m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO for %%s IN (on) DO 7z.exe a teste.resultado\%%xx.lzma.%%tt.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -ms=%%s -m0=LZMA:d=%%d:fb=%%w -mmt=%%t
for %%x IN (9) DO for %%t IN (2) DO for %%d IN (512m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO for %%s IN (on) DO 7z.exe a teste.resultado\%%xx.lzma2.%%tt.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -ms=%%s -m0=lzma2:d=%%d:fb=%%w -mmt=%%t
for %%x IN (9) DO for %%t IN (2) DO for %%d IN (512m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO 7z.exe a teste.resultado\%%xx.lzma.%%tt.%%dd.%%ww.zip .\teste.original\* -mx=%%x -mm=lzma:d=%%d:fb=%%w -mmt=%%t
echo 1 threads - maxproc=8
for %%x IN (9) DO for %%t IN (1) DO for %%d IN (512m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO for %%s IN (on) DO 7z.exe a teste.resultado\%%xx.lzma.%%tt.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -ms=%%s -m0=LZMA:d=%%d:fb=%%w -mmt=%%t
for %%x IN (9) DO for %%t IN (1) DO for %%d IN (512m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO for %%s IN (on) DO 7z.exe a teste.resultado\%%xx.lzma2.%%tt.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -ms=%%s -m0=lzma2:d=%%d:fb=%%w -mmt=%%t
for %%x IN (9) DO for %%d IN (1024m 768m 512m 384m 256m 192m 128m 96m 64m 48m 32m 24m 16m 12m 8m 6m 4m 3m 2m 1m) DO for %%w IN (32 28 24 20 16 14 12 10 8 7 6 5 4 3 2) DO for %%s IN (on) DO 7z.exe a teste.resultado\%%xx.ppmd.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -m0=PPMd:mem=%%d:o=%%w -ms=%%s
for %%x IN (9) DO for %%t IN (1) DO for %%d IN (512m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO 7z.exe a teste.resultado\%%xx.lzma.%%tt.%%dd.%%ww.zip .\teste.original\* -mx=%%x -mm=lzma:d=%%d:fb=%%w -mmt=%%t
In short, I want to process all that in the most efficient manner possible. Doing it by deciding how many processes can run at a time would be a way, but then there's also the memory required for each process, so that the sum of all required memory by those processes won't exceed 8192 MB. I got this part working.
#echo off
setlocal enableDelayedExpansion
set "maxMem=8192"
set "maxThreads=8"
:cycle1
set "cycleCount=4"
set "cycleThreads=1"
set "maxProc="
set /a "maxProc=maxThreads/cycleThreads"
set "cycleFor1=for %%x IN (9) DO for %%t IN (1) DO for %%d IN (512m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO for %%s IN (on) DO ("
set "cycleFor2=for %%x IN (9) DO for %%t IN (1) DO for %%d IN (512m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO for %%s IN (on) DO ("
set "cycleFor3=for %%x IN (9) DO for %%d IN (1024m 768m 512m 384m 256m 192m 128m 96m 64m 48m 32m 24m 16m 12m 8m 6m 4m 3m 2m 1m) DO for %%w IN (32 28 24 20 16 14 12 10 8 7 6 5 4 3 2) DO for %%s IN (on) DO ("
set "cycleFor4=for %%x IN (9) DO for %%t IN (1) DO for %%d IN (512m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO ("
set "cycleCmd1=7z.exe a teste.resultado\%%xx.lzma.%%tt.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -ms=%%s -m0=LZMA:d=%%d:fb=%%w -mmt=%%t"
set "cycleCmd2=7z.exe a teste.resultado\%%xx.lzma2.%%tt.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -ms=%%s -m0=lzma2:d=%%d:fb=%%w -mmt=%%t"
set "cycleCmd3=7z.exe a teste.resultado\%%xx.ppmd.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -m0=PPMd:mem=%%d:o=%%w -ms=%%s"
set "cycleCmd4=7z.exe a teste.resultado\%%xx.lzma.%%tt.%%dd.%%ww.zip .\teste.original\* -mx=%%x -mm=lzma:d=%%d:fb=%%w -mmt=%%t"
set "tempMem1=5407"
set "tempMem2=5407"
set "tempMem3=1055"
set "tempMem4=5378"
rem set "tempMem1=5407"
rem set "tempMem2=5407"
rem set "tempMem3=1055 799 543 415 287 223 159 127 95 79 63 55 47 43 39 37 35 34 33 32"
rem set "tempMem4=5378"
set "memSum=0"
if not defined memRem set "memRem=!maxMem!"
for /l %%N in (1 1 %cycleCount%) DO (set "tempProc%%N=")
for /l %%N in (1 1 %cycleCount%) DO (
set memRem
set /a "tempProc%%N=%memRem%/tempMem%%N"
set /a "memSum+=tempMem%%N"
set /a "memRem-=tempMem%%N"
set /a "maxProc=!tempProc%%N!"
call :executeCycle
set /a "memRem+=tempMem%%N"
set /a "memSum-=tempMem%%N"
set /a "maxProc-=!tempProc%%!
)
goto :fim
:executeCycle
set "lock=lock_%random%_"
set /a "startCount=0, endCount=0"
for /l %%N in (1 1 %maxProc%) DO set "endProc%%N="
set launch=1
for %%x IN (9) DO for %%t IN (1) DO for %%d IN (512m) DO for %%w IN (273 256 192 128 96 64 48 32 24 16 12 8) DO for %%s IN (on) DO (
set "cmd=7z.exe a teste.resultado\%%xx.lzma.%%tt.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -ms=%%s -m0=LZMA:d=%%d:fb=%%w -mmt=%%t"
if !startCount! lss %maxProc% (
set /a "startCount+=1, nextProc=startCount"
) else (
call :wait
)
set cmd!nextProc!=!cmd!
echo !time! - proc!nextProc!: starting !cmd!
2>nul del %lock%!nextProc!
start /b /low cmd /c !cmd!>"%lock%!nextProc!"
)
set "launch="
:wait
for /l %%N in (1 1 %startCount%) do (
if not defined endProc%%N if exist "%lock%%%N" (
echo !time! - proc%%N: finished !cmd%%N!
if defined launch (
set nextProc=%%N
exit /b
)
set /a "endCount+=1, endProc%%N=1"
) 9>>"%lock%%%N"
) 2>nul
if %endCount% lss %startCount% (
1>nul 2>nul ping /n 2 ::1
goto :wait
)
2>nul del %lock%*
echo ===
echo Thats all folks!
exit /b
:fim
pause
I have trouble with cycleFor1 and cycleCmd1 located in :cycle1 part - they should be replacing the for line and the first cmd variable inside the :executeCycle, to make it work as I intend to. How do I do that?
The other issue I have is about tempMem3. I have logged all memory required when the command cycleCmd3 would be running. It is dictionary dependant. tempMem3 and cycleCmd3 are related like this:
for %%d IN (1024m 768m 512m 384m 256m 192m 128m 96m 64m 48m 32m 24m 16m 12m 8m 6m 4m 3m 2m 1m) DO
set "tempMem3=1055 799 543 415 287 223 159 127 95 79 63 55 47 43 39 37 35 34 33 32"
So 1024m would use 1055, 768m would use 799, and so on till 1m using 32. I don't know how to translate that into the script.
Any help is appreciated.
I've already posted a robust batch solution that limits the number of parallel processes at Parallel execution of shell processes. That script uses a list of commands that is embedded within the script. Follow the link to see how it works.
I modified that script to generate the commands using FOR loops as per your question. I also set the limit to 8 simultaneous processes.
Your maximum memory is 1g, and you never have more than 8 processes, so I don't see how you could ever exceed 8g. If you increase the max memory per processes, then you do have to worry about total memory. you will have to add additional logic to keep track of how much memory is being used, and which cpu IDs are available. Note that batch numbers are limited to ~2g, so I recommend computing memory used in megabytes.
By default, the script hides the output of the commands. If you want to see the output, then run it with the /O option.
#echo off
setlocal enableDelayedExpansion
:: Display the output of each process if the /O option is used
:: else ignore the output of each process
if /i "%~1" equ "/O" (
set "lockHandle=1"
set "showOutput=1"
) else (
set "lockHandle=1^>nul 9"
set "showOutput="
)
:: Define the maximum number of parallel processes to run.
:: Each process number can optionally be assigned to a particular server
:: and/or cpu via psexec specs (untested).
set "maxProc=8"
:: Optional - Define CPU targets in terms of PSEXEC specs
:: (everything but the command)
::
:: If a cpu is not defined for a proc, then it will be run on the local machine.
:: I haven't tested this feature, but it seems like it should work.
::
:: set cpu1=psexec \\server1 ...
:: set cpu2=psexec \\server1 ...
:: set cpu3=psexec \\server2 ...
:: etc.
:: For this demo force all cpu specs to undefined (local machine)
for /l %%N in (1 1 %maxProc%) do set "cpu%%N="
:: Get a unique base lock name for this particular instantiation.
:: Incorporate a timestamp from WMIC if possible, but don't fail if
:: WMIC not available. Also incorporate a random number.
set "lock="
for /f "skip=1 delims=-+ " %%T in ('2^>nul wmic os get localdatetime') do (
set "lock=%%T"
goto :break
)
:break
set "lock=%temp%\lock%lock%_%random%_"
:: Initialize the counters
set /a "startCount=0, endCount=0"
:: Clear any existing end flags
for /l %%N in (1 1 %maxProc%) do set "endProc%%N="
:: Launch the commands in a loop
set launch=1
echo mem=1m 2m 3m 4m 6m 8m 12m 16m 24m 32m 48m 64m 96m 128m 192m 256m 384m 512m 768m 1024m
echo o=2 3 4 5 6 7 8 10 12 14 16 20 24 28 32
echo s=off 1m 2m 4m 8m 16m 32m 64m 128m 256m 512m 1g 2g 4g 8g 16g 32g 64g on
echo x=1 3 5 7 9
for %%x IN (9) DO for %%d IN (1024m 768m 512m 384m 256m 192m 128m 96m 64m 48m 32m 24m 16m 12m 8m 6m 4m 3m 2m 1m) DO (
set "cmd=7z.exe a teste.resultado\%%xx.ppmd.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -m0=PPMd:mem=%%d:o=%%w -ms=%%s"
if !startCount! lss %maxProc% (
set /a "startCount+=1, nextProc=startCount"
) else (
call :wait
)
set cmd!nextProc!=!cmd!
if defined showOutput echo -------------------------------------------------------------------------------
echo !time! - proc!nextProc!: starting !cmd!
2>nul del %lock%!nextProc!
%= Redirect the lock handle to the lock file. The CMD process will =%
%= maintain an exclusive lock on the lock file until the process ends. =%
start /b "" cmd /c %lockHandle%^>"%lock%!nextProc!" 2^>^&1 !cpu%%N! !cmd!
)
set "launch="
:wait
:: Wait for procs to finish in a loop
:: If still launching then return as soon as a proc ends
:: else wait for all procs to finish
:: redirect stderr to null to suppress any error message if redirection
:: within the loop fails.
for /l %%N in (1 1 %startCount%) do (
%= Redirect an unused file handle to the lock file. If the process is =%
%= still running then redirection will fail and the IF body will not run =%
if not defined endProc%%N if exist "%lock%%%N" (
%= Made it inside the IF body so the process must have finished =%
if defined showOutput echo ===============================================================================
echo !time! - proc%%N: finished !cmd%%N!
if defined showOutput type "%lock%%%N"
if defined launch (
set nextProc=%%N
exit /b
)
set /a "endCount+=1, endProc%%N=1"
) 9>>"%lock%%%N"
) 2>nul
if %endCount% lss %startCount% (
1>nul 2>nul ping /n 2 ::1
goto :wait
)
2>nul del %lock%*
if defined showOutput echo ===============================================================================
echo Thats all folks!
To execute at the same time no more than 8 instances of 7z.exe process you could do this:
#Echo OFF & Setlocal EnableDelayedExpansion
Set /A "pCount=0" & REm Process count
For
...
) DO (
Set /A "pCount+=1"
If !pCount! LEQ 8 (
Start /B 7z.exe a teste.resultado\%%xx.ppmd.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -m0=PPMd:mem=%%d:o=%%w -ms=%%s
)
)
...
If do you want to run each process in a new parallel CMD window then you would replace the Start /B line on my code for this else:
CMD /C "Start /w 7z.exe a teste.resultado\%%xx.ppmd.%%dd.%%ww.%%ss.7z .\teste.original\* -mx=%%x -m0=PPMd:mem=%%d:o=%%w -ms=%%s"