Building x64 NSIS (using VS2012) - visual-studio-2012

Question

I am in the process of creating a pure x64 version of my application. In order to do that, I also need an x64 installer. I've read online that the NSIS code does support x64 but since they don't distribute an x64 build that I need to build from source (including all plugins/etc).
I've been able to build NSIS (v3.0.4) from source for x86 using Python 2.7/SCons 3.1.1/VS 2012/Zlib 1.2.7.
scons ZLIB_W32=C:\Source\zlib-1.2.7
But when I add the TARGET_ARCH=amd64 to the scons command,
scons ZLIB_W32=C:\Source\zlib-1.2.7 TARGET_ARCH=amd64
It doesn't work. Initially it built but didn't link because zlib was still x86.
C:\Source\zlib-1.2.7\lib\zdll.lib : warning LNK4272: library machine type 'x86' conflicts with target machine type 'x64'
However, after rebuilding zlib1.dll (using VS2012) to be x64 (dumpbin confirms this)
I now get an error that it can't find zlib
scons ZLIB_W32=C:\Source\zlib-1.2.7 TARGET_ARCH=amd64
scons: Reading SConscript files ...
WARNING: VER_PACKED not set, defaulting to 0x03003666!
Delete("nsis-18-Nov-2019.cvs")
Delete(".instdist")
Delete(".test")
Using Microsoft tools configuration (14.2)
Checking for memset requirement... (cached) yes
Checking for memcpy requirement... (cached) yes
Checking for C library gdi32... (cached) yes
Checking for C library user32... (cached) yes
Checking for C library pthread... (cached) no
Checking for C library iconv... (cached) no
Checking for C library shlwapi... (cached) yes
Checking for C library oleaut32... (cached) yes
Checking for C library version... (cached) yes
Checking for C library zdll... no
Checking for C library z... no
zlib (win32) is missing!
Note that I have made sure that the directory structure matches in the x64 build so that the following files exist:
C:\Source\zlib-1.2.7\zlib1.dll
C:\Source\zlib-1.2.7\lib\zlib1.lib
C:\Source\zlib-1.2.7\include\zconf.h
C:\Source\zlib-1.2.7\include\zlib.h
It did occur to me that I'm telling scons to look for x86 zlib (hence the W32 in ZLIB_W32) but I didn't see an option for telling scons to look for x64 zlib in the -h output.
What am I missing?
-UPDATE 1-
I'm making progress but not out of the woods yet. I've identified several issues with my build. #1, I wasn't actually using VS2012 like I thought (I have several versions installed). See above where the scons output says 14.0 (VS2019). Oops. Unfortunately, simply adding a MSVC_VERSION=11.0 to my command line didn't fix it. It seems that the nsis project isn't passing this along to scons. The only way I could figure out how to do this was to modify the nsis SConstruct file from:
######################################################################
####### Build Environment ###
######################################################################
path = ARGUMENTS.get('PATH', '')
toolset = ARGUMENTS.get('TOOLSET', '')
arch = ARGUMENTS.get('TARGET_ARCH', 'x86')
if toolset and path:
defenv = Environment(TARGET_ARCH = arch, ENV = {'PATH' : path}, TOOLS = toolset.split(',') + ['zip'])
else:
if path:
defenv = Environment(TARGET_ARCH = arch, ENV = {'PATH' : path})
if toolset:
defenv = Environment(TARGET_ARCH = arch, TOOLS = toolset.split(',') + ['zip'])
if not toolset and not path:
defenv = Environment(TARGET_ARCH = arch)
Export('defenv')
to:
######################################################################
####### Build Environment ###
######################################################################
path = ARGUMENTS.get('PATH', '')
toolset = ARGUMENTS.get('TOOLSET', '')
arch = ARGUMENTS.get('TARGET_ARCH', 'x86')
vs_version = ARGUMENTS.get('MSVC_VERSION', '')
if toolset and path:
defenv = Environment(TARGET_ARCH = arch, ENV = {'PATH' : path}, TOOLS = toolset.split(',') + ['zip'], MSVC_VERSION = vs_version)
else:
if path:
defenv = Environment(TARGET_ARCH = arch, ENV = {'PATH' : path}, MSVC_VERSION = vs_version)
if toolset:
defenv = Environment(TARGET_ARCH = arch, TOOLS = toolset.split(',') + ['zip'], MSVC_VERSION = vs_version)
if not toolset and not path:
defenv = Environment(TARGET_ARCH = arch, MSVC_VERSION = vs_version)
Export('defenv')
Now my build output correctly identifies as VS2012:
C:\Source\nsis\nsis-code-r7069-NSIS-tags-v304>scons ZLIB_W32=C:\Source\zlib MSVC_VERSION=11.0
scons: Reading SConscript files ...
WARNING: VER_PACKED not set, defaulting to 0x03003666!
Delete("nsis-19-Nov-2019.cvs")
Delete(".instdist")
Delete(".test")
Using Microsoft tools configuration (11.0)
Checking for memset requirement... (cached) yes
<snip>
It seems like there should be a better way to allow this but I'm not familiar enough (yet) with scons or nsis's usage of it to submit a PR to fix this. Or maybe it already exists and I'm just not smart enough to find it (yet).
Now, on to building x64 nsis. Big thanks to #Anders for narrowing down which files are needed a bit and #bdbaddog for suggesting to look into the scons config.log where I found that I had named the zlib.dll lib file incorrectly when I build it for x64. After changing the output lib to
C:\Source\zlib-1.2.7\lib\zdll.lib
It at least tries to build but fails at linking...
-edit to update 1- all of the linking errors that were here previously were a bad rabbit hole I went down yesterday. Not sure if it was caused by scons leaving some temp files somewhere or VS doing it or me just being stupid but today when I started trying to track down the linking errors I couldn't reproduce them (there was a machine reboot yesterday and who knows what might have been locked/cached until then).
-UPDATE 2-
This update is removing all the mis-leading crap from update 1. Current situation is x64 NSIS build (using both my zlib-1.2.7-amd64 build AND the official NSIS zlib-1.2.8-amd64 pre-built binaries on the NSIS wiki) is failing at linking with the same error:
System.obj : error LNK2019: unresolved external symbol CallProc2 referenced in function Call
build\urelease\System\amd64-unicode\System.dll : fatal error LNK1120: 1 unresolved externals
scons: *** [build\urelease\System\amd64-unicode\System.dll] Error 1120
scons: building terminated because of errors.
So far I'm not having any luck figuring out what CallProc2 is but I think this means zlib is no longer at fault here.
-UPDATE 3-
Thanks to #Anders for the direction here since I was really stumped on the Callproc issue, the CallProc2 method is isolated to the system plugin and is caused by it not building correctly. There is some discussion in the comments below trying to find the cause but for now, I'm just excluding that plugin to get to a system that works before coming back to this.
Right now, the scons build completes and produces x64 binaries.
C:\Source\nsis\build\urelease\makensisw\>dumpbin /HEADERS ./makensisw.exe
Microsoft (R) COFF/PE Dumper Version 11.00.61030.0
Copyright (C) Microsoft Corporation. All rights reserved.
Dump of file ./makensisw.exe
PE signature found
File Type: EXECUTABLE IMAGE
FILE HEADER VALUES
8664 machine (x64)
However, the installer build now results in the following error:
scons dist-installer ZLIB_W32=C:\Source\zlib-1.2.8-x64 MSVC_VERSION=11.0
link /nologo /map /subsystem:console,5.01 /OUT:build\urelease\VPatch\Source\GenPat\GenPat.exe /LIBPATH:C:\Source\zlib-1.2.8-x64\lib zdll.lib build\urelease\VPatch\Source\GenPat\adler32.obj build\urelease\VPatch\Source\GenPat\Checksums.obj build\urelease\VPatch\Source\GenPat\ChunkedFile.obj build\urelease\VPatch\Source\GenPat\FileFormat1.obj build\urelease\VPatch\Source\GenPat\GlobalTypes.obj build\urelease\VPatch\Source\GenPat\main.obj build\urelease\VPatch\Source\GenPat\md5.obj build\urelease\VPatch\Source\GenPat\PatchGenerator.obj build\urelease\VPatch\Source\GenPat\POSIXUtil.obj
adler32.obj : error LNK2019: unresolved external symbol _adler32 referenced in function "unsigned long __cdecl Checksum::adler32(unsigned long,unsigned char const *,unsigned int)" (?adler32#Checksum##YAKKPBEI#Z)
build\urelease\VPatch\Source\GenPat\GenPat.exe : fatal error LNK1120: 1 unresolved externals
scons: *** [build\urelease\VPatch\Source\GenPat\GenPat.exe] Error 1120
Unfortunately, without the installer, I'm kinda lost again. The x64 binaries run but when trying to compile any script (I believe), I get the following error (note that I copied makensis and zlib1.dll into the same directory for this):
C:\Source>makensis.exe ./myapp.nsi
Error: reading stub "C:\Stubs\zlib-amd64-unicode"
Error initalizing CEXEBuild: error setting default stub
-UPDATE 4-
I have finally gotten a build to work successfully after ignoring the system plugin. This works both with compiling zlib from source for x64 or with the prebuilt version from the NSIS wiki (linked in #Anders answer below):
scons ZLIB_W32=C:\Source\zlib-1.2.8-x64 TARGET_ARCH=amd64 MSVC_VERSION=11.0 SKIPPLUGINS=System
and I was able to deploy to my dev system using (from an admin enabled prompt since I was installing to Program Files):
scons PREFIX="C:\Program Files (x86)\NSIS" install ZLIB_W32=C:\Source\zlib-1.2.8-x64 TARGET_ARCH=amd64 SKIPPLUGINS=System
Unfortunately, you cannot build the NSIS installer from here (dist-install) because the NSIS installer itself apparently depends on usage of the System plugin. So this scenario isn't sufficient for being able to prep a build machine unless you are going to build everything from scratch on that machine.
Also, I'm not out of the woods yet because I also use the System plugin for my installer so I need to figure out why that isn't working.
However, I was able to compile and run a pure x64 bare minimum installer package with the above setup:
# name the installer
OutFile "Installer.exe"
# default section start; every NSIS script has at least one section.
Section
# default section end
SectionEnd
The above was straight out of the NSIS docs.

Answer 1

My environment:
cl.exe:
Microsoft (R) C/C++ Optimizing Compiler Version 19.24.28316 for x64
nsis version: 3.06.1
zlib: Zlib-1.2.8-win64-AMD64
I modified SCons/Tool/masm.py:
C:\miniconda3\Lib\site-packages\SCons-4.0.1-py3.7.egg\SCons\Tool>diff -c masm.old.py masm.py
*** masm.old.py Fri Oct 09 22:02:45 2020
--- masm.py Fri Oct 09 21:58:06 2020
***************
*** 61,66 ****
--- 61,68 ----
shared_obj.add_emitter(suffix, SCons.Defaults.SharedObjectEmitter)
env['AS'] = 'ml'
+ if env.get('TARGET_ARCH')=='amd64':
+ env['AS'] = 'ml64'
env['ASFLAGS'] = SCons.Util.CLVar('/nologo')
env['ASPPFLAGS'] = '$ASFLAGS'
env['ASCOM'] = '$AS $ASFLAGS /c /Fo$TARGET $SOURCES'
Also modified nsis-3.06.1-src\Contrib\System\SConscript as follows:
C:\dev\nsis-3.06.1-src\Contrib\System>diff -c SConscript.old SConscript
*** SConscript.old Fri Oct 09 22:06:31 2020
--- SConscript Fri Oct 09 19:17:40 2020
***************
*** 4,9 ****
--- 4,10 ----
Source/Buffers.c
Source/Plugin.c
Source/System.c
+ Source/Call-amd64.S
""")
libs = Split("""
Then C:\dev\nsis-3.06.1-src>scons TARGET_ARCH=amd64 ended successfully.

Answer 2

I have
\zlib1.dll
\include\zconf.h
\include\zlib.h
\lib\libzdll.a
\lib\zdll.lib
\lib\zlib.lib
You can get pre-built zlib for NSIS here.
In the past I have actually used Process Monitor to figure out which file SCons is looking for, that sadly seems to be the fastest way to find out.
Regarding CallProc2. Add "System" to SKIPPLUGINS to get past this issue to make sure everything else compiles. CallProc2 is implemented in the amd64 .S file and used by system.c. You need the 64-bit Microsoft assembler detected correctly to build the .S file.
If you only need to compile NSIS once then you can cheat; Open a Visual Studio command prompt and execute ml64.exe /c Call-amd64.S and then copy the .obj file to the same build directory as system.obj and run Scons again.
To do it properly you need to investigate why Scons is not compiling the .S file. Could be a configuration issue or a bug in the related Sconscript file.