Debugging Inno Setup installer that respawns itself - inno-setup

Question

As it can be seen from this question we start a new instance of Inno Setup:
Instance := ShellExecute(0, '', ExpandConstant('{srcexe}'), Params, '', SW_SHOW);
where
function ShellExecute(hwnd: HWND; lpOperation: string; lpFile: string;
lpParameters: string; lpDirectory: string; nShowCmd: Integer): THandle;
external 'ShellExecuteW#shell32.dll stdcall';
All the code from this question's answer I moved to the VCL_Styles.iss file and included it into my main script.
The problem is that after I've passed the ShellExecute call and terminate by the debugger afterwards one instance of Inno Setup keeps running (so I have to kill the process using Windows Task Manager) and I get the following messages in the Debug Output:
*** Terminating process
*** Removing left-over temporary directory: C:\Users\JCONST~1\AppData\Local\Temp\is-PV9OS.tmp
*** Setup is still running; can't get exit code
instead of exit code 6 which according to the documentation is returned when:
The Setup process was forcefully terminated by the debugger (Run |
Terminate was used in the Compiler IDE).
I'm not sure which instance of Inno Setup is still running and how can I stop it?
Here's the contents of the VCL.Styles that I include into my main script so I get the aforementioned error:
[Setup]
ShowLanguageDialog=no
[Code]
function ShellExecute(hwnd: HWND; lpOperation: string; lpFile: string;
lpParameters: string; lpDirectory: string; nShowCmd: Integer): THandle;
external 'ShellExecuteW#shell32.dll stdcall';
<event('InitializeSetup')>
function MyInitializeSetup2: Boolean;
var
Instance: THandle;
I: Integer;
S, Params, Language: String;
begin
Result := True;
for I := 1 to ParamCount do
begin
S := ParamStr(I);
if CompareText(Copy(S, 1, 5), '/SL5=') <> 0 then
begin
Params := Params + AddQuotes(S) + ' ';
end;
end;
Params := Params + '/LANG=en';
Language := ExpandConstant('{param:LANG}');
if Language = '' then
begin
Instance := ShellExecute(0, '', ExpandConstant('{srcexe}'), Params, '', SW_SHOW);
if Instance <= 32 then
begin
S := 'Running installer with the selected language failed. Code: %d';
MsgBox(Format(S, [Instance]), mbError, MB_OK);
end;
Result := False;
Exit;
end;
end;

Answer 1

When the debugger steps over the ShellExecute and the new instance of the installer process is started, the IDE debugger seems to pick that process and restarts the debugging. I assume this is not intended behaviour, or at least not a well-tested one. The Terminate function then probably tries to close/communicate with to the old process (which has terminated on its own meanwhile – due to its InitializeSetup returning False after the ShellExecute).
Martijn Laan (the current maintainer of Inno Setup) stated that Inno Setup is not designed to respawn itself. Actually Inno Setup own Exec API explicitly prevents respawning the installer. Bypassing this restriction by using WinAPI ShellExecute instead introduces the problem described in the question. It's not a surprise that the debugger cannot handle this situation.

Answer 2

Looks like an Inno Setup's IDE bug may have caused that problem.
Here's the report link:
https://groups.google.com/g/innosetup/c/pDSbgD8nbxI/m/0lvTsslOAwAJ