When I run Inno Setup on a large set of files (>2GB), it takes a long time to run. I believe it is spending its time in the compression, which should be CPU bound, but it is only using a couple of CPUs. Is there a way to spread this across (many) more cores?
Specifically, I'm working with this boost-release repository, which has an Inno Setup script that includes:
[Setup]
....
Compression=lzma2/ultra64
....
[Files]
Source: "boost_1.69.0/*"; DestDir: "{app}"; Flags: ignoreversion recursesubdirs ignoreversion
....
Calling Compil32.exe boost_installer.iss takes approximately 25 minutes on a machine with 16 cores and 32GB of RAM (Azure F16s v2).
The set of files is approximately 2.5GB with 2 GB of that being a set of about 300 compiled libraries. The remaining 500MB is 60,000 source files.
So to get to the bottom of this, I created a test project that went through all sorts of permutations of various Inno Setup configuration options.
The ones I found useful (and gave me a 40% improvement in speed!) are:
SolidCompression=yes
LZMAUseSeparateProcess=yes
LZMANumBlockThreads=6
Without SolidCompression, the LZMANumBlockThreads doesn't have much impact. But together, I saw a more typically parallelize-able problem, where more threads gave faster results (to a point).
If you find this interesting, I'd recommend the writeup I did on it, it has a lot of data to back it up.
Try setting LZMANumBlockThreads directive (the default value is 1):
When compressing a large amount of data, the LZMA2 compressor has the ability to divide the data into "blocks" and compress two or more of these blocks in parallel through the use of additional threads (provided sufficient processor power is available). This directive specifies the number of threads to use -- that is, the maximum number of blocks that the LZMA2 compressor may compress in parallel.
Compression=zip
SolidCompression=yes
LZMAUseSeparateProcess=yes
LZMANumBlockThreads=6
use zip compression for 2x faster installations.
speed tested. approved. use zip.
Related
I've lots of files to compress (22K # 31.5 GB) and then unpack (install) them via Inno Installer. My current [Setup] config is:
LZMANumBlockThreads=6
LZMADictionarySize=1048576 // doesn't affect the installation performance
LZMANumFastBytes=273 // doesn't affect the installation performance
CompressionThreads=auto
DisableReadyPage=True
InternalCompressLevel=max
SolidCompression=no // 'yes' causes out of memory error due to overloading 16 GB of RAM
Compression=lzma2/ultra64
The problem exists in the installation performance. I see that only 3-4 of 48 available CPU threads are fully loaded and the unpacking process seems to be much slower than it can be on my 2 * Xeons.
NVMe SSD uses only 2-4% of its performance.
My colleagues with different PC configurations noticed the same problem.
How to bypass this bottleneck and say to Inno to use every available CPU thread and unpack my files faster?
P.S. tried different configs described there: https://github.com/teeks99/inno-test but didn't get any appliable result. As I can see, I need to do something with islzma64.exe but it doesn't have any sources - even there https://github.com/jrsoftware/issrc/tree/main/Files it is the solid executable file without any documentation.
Thank you.
try to add
[setup]
LZMAUseSeparateProcess=yes
Description of LZMAUseSeparateProcess
also, when you are using solidcompression=no, then 1 file = 1 solidblock ~= dict size, so you have ~22k of solidblocks, decomprassion from 1 (few) solid block(s) should be faster than from thousands. I think, solid compression and LZMADictionarySize=1048576 is the best variation, also lzma2 have a limits of allocating memory (its about 'out of memory error due to overloading 16 GB of RAM'), not sure how much in modification inside inno
I am writing a large file to disk from a user-mode application. In parallel to it, I am writing one or more smaller files. The large file won't be read back anytime soon, but the small files could be. I have enough RAM for the application + smaller files, but not enough for the large file. Can I tell the OS not to keep parts of the large file in cache after they are written to disk so that more cache is available for smaller files? I still want writes to the large file be fast enough.
Can I tell the OS not to keep parts of the large file in cache ?
Yes, you probably want to use some system call like posix_fadvise(2) or madvise(2). In weird cases, you might use readahead(2) or userfaultfd(2) or Linux-specific flags to mmap(2). Or very cleverly handle SIGSEGV (see signal(7), signal-safety(7) and eventfd(2) and signalfd(2)) You'll need to write your C program doing that.
But I am not sure that it is worth your development efforts. In many cases, the behavior of a recent Linux kernel is good enough.
See also proc(5) and linuxatemyram.com
You many want to read the GC handbook. It is relevant to your concerns
Conbsider studying for inspiration the source code of existing open-source software such as GCC, Qt, RefPerSys, PostGreSQL, GNU Bash, etc...
Most of the time, it is simply not worth the effort to explicitly code something to manage your page cache.
I guess that mount(2) options in your /etc/fstab file (see fstab(5)...) are in practice more important. Or changing or tuning your file system (e.g. ext4(5), xfs(5)..). Or read(2)-ing in large pieces (1Mbytes).
Play with dd(1) to measure. See also time(7)
Most applications are not disk-bound, and for those who are disk bound, renting more disk space is cheaper that adding and debugging extra code.
don't forget to benchmark, e.g. using strace(1) and time(1)
PS. Don't forget your developer costs. They often are a lot above the price of a RAM module (or of some faster SSD disk).
I learned about DiskSpanning and that it is required for larger file sizes. I however just want to compile my files into a single installation file. What I got was an application installer and 9 bin files of 2Gb each.
Is it possible to have all this contained into a single installer?
I am attempting to take several game pak files and create an installer with them.
[Setup]
Compression=lzma
SolidCompression=yes
WizardStyle=modern
DiskSpanning=yes
SlicesPerDisk=1
DiskSliceSize=max
It is not possible.
The documentation says:
Note that it is required that you set this directive to yes if the compressed size of your installation exceeds 2,100,000,000 bytes, even if you don't intend to place the installation onto multiple disks.
2 GB is also the maximum size you can get per file.
Valid values: 262144 through 2100000000, or max
If you're worried that someone might miss copying some files, you might try a self extracting ZIP archive, e.g. using 7zip SFX Builder.
I have an application that automatically updates itself each time it is run. It does this by comparing the "modified" timestamp on each file in the program directory, copying files that are newer from a network file share.
I packaged this program with Inno Setup Compiler and ran the resulting installer on a test system. All of the installed files have timestamps one second older compared to the files I used when compiling the installer. This causes every file in the program directory to be recopied from the file share on first run. This is an issue for our remote users that have limited bandwidth.
I've tried this numerous times, with and without the "TimeStampsInUTC" directive set in my Inno Setup script and on four different test systems. The timestamps are consistently one second off.
Could this be a problem in Inno Setup Compiler? Are there any suggestions to work around this issue? I'm hoping to use this installer at two new offices soon.
There's TimeStampRounding directive, which is by default set to 2:
By default, time stamps on files referenced by non external [Files] section entries are rounded down to the nearest 2-second boundary. FAT partitions have only a 2-second time stamp resolution, so this ensures that time stamps are set the same way on both FAT and NTFS partitions.
The rounding can be altered or disabled by setting the TimeStampRounding directive. Setting it to 0 will disable the rounding. Setting it to a number between 1 and 60 will cause time stamps to be rounded down to the nearest TimeStampRounding-second boundary.
(emphasis mine)
I am working to reduce the build time of a large Visual C++ 2008 application. One of the worst bottlenecks appears to be the generation of the PDB file: during the linking stage, mspdbsrv.exe quickly consumes available RAM, and the build machine begins to page constantly.
My current theory is that our PDB files are simply too large. However, I've been unable to find any information on what the "normal" size of a PDB file is. I've taking some rough measurements of one of the DLLs in our application, as follows:
CPP files: 34.5 MB, 900k lines
Header files: 21 MB, 400k lines
Compiled DLL: 33 MB (compiled for debug, not release)
PDB: 187 MB
So, the PDB file is roughly 570% the size of the DLL. Can someone with experience with large Visual C++ applications tell me whether these ratios and sizes make sense? Or is there a sign here that we are doing something wrong?
(The largest PDB file in our application is currently 271 MB, for a 47.5 MB DLL. Source code size is harder to measure for that one, though.)
Thanks!
Yes, .pdb files can be very large - even of the sizes you mention. Since a .pdb file contains data to map source lines to machine code and you compile a lot of code there's a lot of data in the .pdb file and you likely can't do anything with that directly.
One thing you could try is to split your program into smaller parts - DLLs. Each DLL will have its own independent .pdb. However I seriously doubt it will decrease the build time.
Do you really need full debug information at all time? You can create a configuration with less debug info in it.
But as sharptooth already said, it is time to refactor and split your program in small more maintainable parts. This won't only reduce build time.