Our .NET application uses WinSCP software to work with SFTP servers. It's running on different machines, however 1 week ago 2 of them started to fail with the following exception:
WinSCP.SessionLocalException, WinSCP process terminated with exit code 3 and output "", without responding (response log file D:\local\Temp\wscp1028.00ADA812.tmp was not created). This could indicate lack of write permissions to the log folder or problems starting WinSCP itself.,
at WinSCP.Session.Open(SessionOptions sessionOptions)
This error doesn't occur always, but after some period of time. Once error occurs it fails always until we restart the entire application. After application restart it successfully works 10-20 hours and approx. 15K-30K succeeded requests. Then the error starts again.
We have already upgraded to the last version of WinSCP (5.7.7) and restarted the machines, however the issue still takes place. What could be the root cause and what steps could be done to fix the issue? Any help is appreciated.
Related
We have roughly 12 Service Fabric clusters running in Azure. They are running correctly in both our Production and Test environments. We have found recently that one of them will not start locally. We have not ran this one locally in quite a while, and I am having a hard time tracking down what might have happened that is causing this error. It is happening on any machine I try to run locally on.
Specifically, after the type is registered, and the app created, the host process immediately terminates:
"Message": "EventName: ApplicationProcessExited Category: StateTransition EventInstanceId 158f38d1-47ac-4b70-9830-0d8d3cdf8f9c ApplicationName fabric:/Office.Ocv.CustomerTalkback.AutomatedService.ServiceFabric Application terminated, ServiceName=fabric:/Office.Ocv.CustomerTalkback.AutomatedService.ServiceFabric/MS.Internal.Office.Ocv.Services.CustomerTalkback.Automated, ServicePackageName=MS.Internal.Office.Ocv.Services.CustomerTalkback.Automated.Package, ServicePackageActivationId=d58e53d1-af22-42fb-9003-3154bcb8d00b, IsExclusive=True, CodePackageName=Code, EntryPointType=Exe, ExeName=MS.Internal.Office.Ocv.Services.CustomerTalkback.Automated.exe, ProcessId=16756, HostId=e27ccd9d-cff6-4317-b168-5a4b7b724808, ExitCode=2147516563, UnexpectedTermination=True, StartTime=06/18/2019 15:47:26. ",
This is dotnet core 2.2.0. All of our Service Fabric apps are running with the same settings/dependencies, etc. Only this one fails locally.
I have tried moving the local cluster to a larger drive (800 GB free); deploying manually via PowerShell (usually VS 2019).
Any help (even if it is just a suggestion of trouble shooting steps) would be much appreciated as I have working on this for about 16 hours over last three days.
thanks!
The problem I had was with the full name of the assembly. The local path was short (like d:\src\adm), but the full assembly name was ~65 characters. It appears as though the PowerShell that deploys locally would fail silently on this. When I dropped the length of the name down to about 35 characters it started working.
Currently having an issue with my kali VM.
I installed Openvas, but when I try and initiate a scan with the admin user I get the following error message:
Operation: Run Wizard
Status Code: 400
Status message: Service temporarily down
Has any one experienced this before? Searched online and couldn't really find any solutions.
Thanks for your time.
I used the Virtual Appliance (Virtual Machine), getting error 400 first and later 503 with "Service temporarily down" when trying to scan.
Solution that worked was found at http://openvas-discuss.wald.intevation.narkive.com/TTK5H8YI/openvas-8-virtual-appliance-unable-to-start-task and can be found at http://plugins.openvas.org/ova_503.txt .
It comes down to stopping services, rebuilding the certificates, updating config and then restarting services. It is a 12 steps process that takes several minutes (maybe 30). I will not copy/paste, the second link above contains all the steps.
My cucumber/ruby/selenium scripts run by starting up a chromebrowser (2.3) session before every test and tearing down after. They have been working for months, but now I have a real problem.
I have 600 scenarios, but when I get to a specific number of scenarios run (251), it bombs out with this message:
Resource temporarily unavailable - chromedriver.exe --port=8602 (Errno::EAGAIN)
If I remove the scenario at which it fails, the issue happens at the next scenario. I tried removing many scenarios but still happens.
I also tried randomizing the ports as I thought that chromedriver connecting to the same port was causing the issue.
I'm stumped!
We have seen this same strange problem twice so far.
First we found our remote Linux server responds to ping but we could not ssh to it. We went to the server and found the system unresponsive, and had to restart it. After we restarted it, we checked the log. We found nothing in the /var/log/message log, but we found some error messages in Oracle's *_alert.log files:
Thread 1 cannot allocate new log, sequence 296280
Private strand flush not complete
Current log# 3 seq# 296279 mem# 0: /home/oracle/app/oracle/oradata/orcl/redo03.log
Current log# 3 seq# 296279 mem# 1: /home/oracle/app/oracle/oradata/orcl/redo09.log
Thread 1 advanced to log sequence 296280 (LGWR switch)
Current log# 2 seq# 296280 mem# 0: /home/oracle/app/oracle/oradata/orcl/redo02.log
Current log# 2 seq# 296280 mem# 1: /home/oracle/app/oracle/oradata/orcl/redo08.log
Process P098 died, see its trace file
Process P098 died, see its trace file
Process P098 died, see its trace file
Our questions are:
Could Oracle cause Linux to hang? I thought even if Oracle is dead, Linux should be alive. We thought this is relevant because these events happened at the same time i.e. Oracle died before Linux hung.
What could be the reason for the server to respond to ping but not allow ssh to it?
We did a test when the ssh login failed, and it showed port 22 was okay:
[administrator#localhost ~]$ nc -v -w 1 172.16.*.* -z 22
Connection to 172.16.*.* 22 port [tcp/ssh] succeeded!
When we do ssh -v, it stops at "load ssh key".
3.Why a linux server is hang,but it can still be ping?How can we prevent system hang?
Any ideas what could be the explanation?
The log snippet you showed doesn't say the database crashed; it looks like a delay in a log switch, and a parallel query slave process dying. That should certainly be investigated - you can start by looking at the trace file - but it could be a symptom rather than the cause.
A very high load can make the server behave like this, responding to some network events, but unable to (or extremely slow to) create new processes. That would explain why you can connect to port 22 but sshd isn't progressing very far and doesn't complete the connection process. It could also explain your P098 dying - it might not be ale to start in the first place.
It's unlikely that Oracle would be causing this directly. It's more likely that you have a script or application process which is spinning for some reason, spawning new processes until the system runs put of resources. (You can certainly get an infinite loop in a PL/SQL block, which would cause high load, but wouldn't cause you to run out of processes - so you'd be able to connect eventually). You could be getting an ORA error that is making a script/app loop if it isn't handled well, but you'd have to hope that's revealed in an application log. It might not even be something that's talking to the DB.
It's basically impossible to know what happened if it wasn't logged. You might have a starting point if you know what was being run at the time. You could also look back at what the DB was doing before the problem, with the AWR reports in Oracle Enterprise Manager, for example.
Unfortunately there isn't much you can do to recover if you can't connect, and even if you have an existing shell running you might not be able to run useful tools to see what's going on. Sometimes a hard reboot is the only option, though obviously it's a last resort.
Yes oracle can hang your system if the process load is very high. Please let us know that do you have multipath on this system with oracle or RACK.
I'm encountering such an error after expanding disk (done by Hyper-V) space on virtual machine.
STOP: c000021a {Fatal System Error} The initial session process or system process terminated unexpectedly with a status of (0x00000000) (0xc000012d 0x001003f0).
The virtual server there is Windows Server 2008 R2 Enterprise Edition, which is also Domain Controller, now my whole environment is down :/
I've tried to repair Windows - but there is no restore point, and using command line, I've also tried the sfc /SCANNOW /OFFBOOTDIR /OFFWINDIR, but got error "Windows Resource
Protection could not perform the requested operation"
I initially responded to this question to ask if Christof ever found a solution. That's not allowed, so my post was deleted.
I'm back to share that I solved the above problem for myself using a mix-and-match set of backed-up registry files. I believe the only reason this worked for me is that there had been ZERO changes to the server between the different times the registry files were backed up. Most of the registry files I used in the recovery were from c:\windows\system32\config\system\regbak, but the SOFTWARE file had a timestamps too close to the time of my initial failure, so I used one that I had created in \windows\tmp when I initially began this recovery process. I followed a guide which apparently was deleted but you can find references by searching on site:microsoft.com kb307545, Also make sure you have a backup of the COMPONENTS hive/file.