ADO Release pipeline AzurePowershell task fails on Connect-AzAccount - azure

Question

I have a very annoying issue with one of our servers running our self-hosted agents.
We have a Release pipeline containing Azure Powershell tasks (Task version 5.*). The 'Use PowerShell Core' is switched on.
The scripts run perfectly on 'server_1', but on 'server_2' the task fails during initialisation / Connect-AzAccount. Please note that the logging below is thus the initialisation of the Azure Powershell task, and not our own script.
Log 'server_1':
2022-10-04T13:20:47.8917865Z ##[command]Import-Module -Name C:\Program Files\WindowsPowerShell\Modules\Az.Accounts\1.6.6\Az.Accounts.psd1 -Global
2022-10-04T13:20:48.8314051Z ##[command]Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
2022-10-04T13:20:50.2298081Z ##[command]Clear-AzContext -Scope Process
2022-10-04T13:20:50.3313243Z ##[command]Connect-AzAccount -ServicePrincipal -Tenant xxxxxxxxxxxxxxxxxx -Credential System.Management.Automation.PSCredential -Environment AzureCloud #processScope
2022-10-04T13:20:51.1463087Z ##[command] Set-AzContext -SubscriptionId yyyyyyyyyyyyyyyyyyyy -TenantId xxxxxxxxxxxxxxxxxx
2022-10-04T13:20:52.4233903Z ## Az module initialization Complete
2022-10-04T13:20:52.4241534Z ## Beginning Script Execution
Log 'server_2':
2022-10-04T12:54:16.3415330Z ##[command]Import-Module -Name C:\Program Files\WindowsPowerShell\Modules\Az.Accounts\1.6.6\Az.Accounts.psd1 -Global
2022-10-04T12:54:17.0951601Z ##[command]Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
2022-10-04T12:54:17.4023735Z ##[command]Clear-AzContext -Scope Process
2022-10-04T12:54:17.4838081Z ##[command]Connect-AzAccount -ServicePrincipal -Tenant xxxxxxxxxxxxxxxxxx -Credential System.Management.Automation.PSCredential -Environment AzureCloud #processScope
2022-10-04T12:54:17.6673926Z ##[error]Unable to read beyond the end of the stream.
2022-10-04T12:54:17.6860076Z ##[error]Initializing Az module failed: For troubleshooting, refer: https://aka.ms/azurepowershelltroubleshooting
2022-10-04T12:54:17.7781536Z ##[error]There was an error with the service principal used for the deployment.
The versions of Azure Powershell modules Az, and Az.Accounts are the same on both servers.
POwershell 5 and Powershell 7 has been installed on both servers.
'server_1' OS is Windows Server 2016 Datacenter
'server_2' OS is Windows Server 2012 R2 Datacenter
When I run a Powershell 5 console on the servers and use Connect-AzAccount using a service principle I also see the same behaviour (server_1 successful, server_2 unsuccessful).
The details of the -debug are:
DEBUG: AzureQoSEvent: CommandName - Connect-AzAccount; IsSuccess - False; Duration - 00:00:47.2917018; Exception - System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
at System.IO.__Error.EndOfFile()
at System.IO.MemoryStream.InternalReadInt32()
at Microsoft.IdentityModel.Clients.ActiveDirectory.TokenCache.Deserialize(Byte[] state)
at Microsoft.Azure.Commands.Common.Authentication.Core.ProtectedFileTokenCache.ReadFileIntoCache(String cacheFileName)
at Microsoft.IdentityModel.Clients.ActiveDirectory.TokenCache.OnBeforeAccess(TokenCacheNotificationArgs args)
at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Flows.AcquireTokenHandlerBase.<RunAsync>d__57.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.<AcquireTokenForClientCommonAsync>d__33.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.<AcquireTokenAsync>d__56.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Commands.Common.Authentication.ServicePrincipalTokenProvider.AcquireTokenWithCertificate(AdalConfiguration config, String appId, String thumbprint)
at Microsoft.Azure.Commands.Common.Authentication.ServicePrincipalTokenProvider.GetAccessTokenWithCertificate(AdalConfiguration config, String clientId, String certificateThumbprint, String credentialType)
at Microsoft.Azure.Commands.Common.Authentication.Factories.AuthenticationFactory.Authenticate(IAzureAccount account, IAzureEnvironment environment, String tenant, SecureString password, String promptBehavior, Action`1 promptAction,
IAzureTokenCache tokenCache, String resourceId)
at Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient.AcquireAccessToken(IAzureAccount account, IAzureEnvironment environment, String tenantId, SecureString password, String promptBehavior, Action`1 promptAction)
at Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient.Login(IAzureAccount account, IAzureEnvironment environment, String tenantId, String subscriptionId, String subscriptionName, SecureString password, Boolean
skipValidation, Action`1 promptAction, String name, Boolean shouldPopulateContextList)
at Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand.<>c__DisplayClass92_0.<ExecuteCmdlet>b__0(AzureRmProfile localProfile, RMProfileClient profileClient, String name)
at Microsoft.Azure.Commands.Profile.Common.AzureContextModificationCmdlet.ModifyContext(Action`2 contextAction)
at Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand.SetContextWithOverwritePrompt(Action`3 setContextAction)
at Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand.ExecuteCmdlet()
at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord();
Questions:
What could cause this issue on failing task when it runs using on 'server_2', while succeeding on 'server_1', or how could I find the root cause of the ##[error] logs, so this leads to the solution to solve this.