Enabling verboseGC on WebSphere Application Server - garbage-collection

I have an issue while enabling verboseGC. I have 8 JVMs (clustered) and out of those, 3 JVMs have been already enabled for verboseGC. I am now enabling verboseGC on the remaining 5 JVMs.
Once I enabled verboseGC at below path (on the 'Runtime' tab) , I have restarted the JVMs.
Application servers > applicationservername > Process definition > Java Virtual Machine.
But when I look for verboseGC logs in the native_stderr.log file, I don't see anything logged. When I went back to the above path, I found that the 'Verbose garbage collection' setting on the 'Runtime' tab was unchecked automatically.
Has anyone encountered a similar issue? What was done to fix it ?

Changes made on the 'Runtime' tab do not persist across restarts of WebSphere Application Server. Any such changes are applied to the currently running JVM process, but are lost when the process is restarted. For the VerboseGC setting to be applied permanently, update the setting under the Configuration tab. Then restart the server for the changes to take effect.
As for the log not being written, verboseGC output is written to native_stderr.log on AIX, Linux, & Windows and to native_stdout.log on Solaris & HP-UX. If you are using Solaris or HP-UX, you have to set Generic JVM Arguments for the logging to work correctly. Refer to this IBM Document for details.

Related

ASP.NET Core IHostedService stops when hosted on linux

I have an ASP.NEt Core 3.1 application with Angular 8 frontend. It runs fine when hosted on IIS but as I have moved it onto a new Ubuntu 18 Server with Nginx above Kestrel sometimes the long running background processes stop working (IHostedService). Then the app runs towards accepting new requests so only the background process is stopped.
These processes get files from clients and give immediate responses with a process ids. The clients can query the process state by their id. Everything have been running fine for months now on IIS but the new config must have some limits that kills these processes. I suppose there is some kestrel or nginx option I don't know about and affect processes started by http requests.
What options can I try and where can I get some logs?
I've tried to log everything from .net core but even the most verbose logs are useless here. Nginx logs doesn't contain any info about the stopped process either.
Although the application runs fine hosted on IIS I tried to find catch blocks without any output and added logging into them but still nothing. Are there anything I can add to my application globals to log any exceptions handled or unhandled?
I forgot to say that I use a local Microsoft SQL Server Express both on windows and linux. The linux Sql Server install was done by the official ms docs (as dotnet and nginx config, too). The database is restored from a windows sql server backup. The connection string is the same with multipleresultsets=true. Are there any differences I should aware of?
For anyone getting here in the future: this was caused by a bug in Microsoft.Data.SqlClient, so I had to update it (independently from EF Core 3.1.2) from nuget to the newer 1.1.2 version.
When it stucked I had two threads waiting for each other, both in SqlClient. With Just my code enabled VS debugger stopped at one of my linq queries. The only interesting part was that it never threw any exceptions and there was no deadlock event on the sql server either. It just waited there so all logs were empty.
https://github.com/dotnet/efcore/issues/18480
https://github.com/dotnet/SqlClient/issues/262

set memory-size for tomcat running as service

As I´m running into memory-problems running my web-app I want to know how I can set the memory for tomcat-8 running on an AWS-linux as a service.
GI-cat needs at least 1000MB free heap space to work properly. You
have 506MB free (total 1752MB). Increase the memory if possible, by
adding -Xmx1000m or more to the java arguments.
I´ve read How do I increase memory on Tomcat 7 when running as a Windows Service? but it only handles windows-services, not linux.
I suppose I have to manipulate catalina.sh, don´t it but I´m unsure if this will effect the service when using service tomcat8 restart.
Non-persistent method
You can set the environment variable before you start your tomcat service:
export CATALINA_OPTS="-Xmx1000m"
And then start your service with:
service tomcat8 restart
Side note: this variable is only set until it gets unset/set by another process or your linux box restarts.
Persistent method
To make this persistent, you want to modify tomcat.conf in $CATALINA_HOME/conf/ and append/modify the environment variable with:
CATALINA_OPTS="-Xmx1000m"
Ref: https://unix.stackexchange.com/a/244197
Updated #1: Changed the response to better suit the author's needs.

What arguments can we add to 'Generic JVM Argument' to display the GC info in RAD Websphere v7.5?

In RAD WebSphere, I want to add the some arguments to the JVM, in order to display some information about the garbage collection. I notice that this action is in the Administration Console-->Server-->Application Server-->Server1-->Java and Process Management-->Process Definition-->Java Virtual Machine-->Generic JVM Argument.
Normally in Oracle JDK, there are arguments that we can add like: -Xms1000m -Xmx1000m -XX:PermSize=256m -XX:MaxPermSize=256m -XX:MaxNewSize=500m -XX:NewSize=500m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/home/sherpa/Sherpa/Server/log/gc.log -XX:+HeapDumpOnOutOfMemoryError.
I tried to add these arguments in it, but it seems that it doesn't work with these arguments.
So is there some specific arguments that can only be used in RAD WebSphere? What arguments can we use in RAD?
I am looking forward to your opinions!
If you're running Websphere application server, you'll be using the IBM JVM.
You can get additional information about garbage collection using the -verbose:gc option.
You can find more information about the output produced by this option on this Knowledge Center page.
RAD is just your developing environment, an IDE. You can have WebSphere Application Server runtime as test environment installed together with RAD, but it is separate server, it just happens to be already integrated in the Servers View.
So in general all rules related to WebSphere apply. WebSphere runs IBM JVM, so if you want to enable verbose gc check the WebSphere infocenter, and do not provide Oracle JVM params as they simply dont work in IBM JVM :-)
So for WebSphere you should enable it via admin console in the JVM settings page, just call it via browser, but it is available to invoke it from RAD in servers view also.
For details check:
Java virtual machine settings
Garbage Collector options - to be specified via Generic JVM options

QSslSocket timeouts in Ubuntu Server, but not in Desktop

We have problem with our Qt based production server for our business application. When total SSL connections increases with time, some clients does not manage to connect at all.
QSslSocket::waitForEncrypted() starts to fail with no QSslError, regardless of that timeout where set. There are more then ~100 active connections when this problem starts to kick in.
So there are ~170 connections, twice of threads, and "lsof" mentions a little more then 1000 opened files (we had to increase file "ulimit" for that..).
It does not look like it's clients problem, since IPs that are failing and reconnecting changes with time (some "leaps in" with success, but then other don't).
As mentioned, this happens in Ubuntu Server (Zentyal 10.04 and "vanilla" 9.10), but does NOT in Ubuntu Desktop 9.10.
Everything runs inside VMWare ESX 4.1, systems there tested with same resources attached. System loads stays below 1.0. Daemon runs with root permissions.
It looks like it's something with "server"/"desktop" kernel or other configuration differences, but I couldn't tell what exactly could make SSL connection not to handshake... in "server editions"...
We are using Qt 4.5.3 compiled by ourselves.
EDIT: after all it's the same on any Linux I tried. It feels like it's some kind socket limit per process, witch is about 1016 - other_opened_files. I'll try to create new question about that.
EDIT 2: It's select and FD_SETSIZE limit problem...
Problem is with fact that Qt uses select() which is limited with FD_SETSIZE macro for maximum selected sockets/files. I had to change FD_SETSIZE value inside /usr/include/bits/typesizes.h before compiling libQtNetwork and libQtCore.

STOP: c000021a {Fatal System Error} The initial session process or system process terminated unexpectedly

I'm encountering such an error after expanding disk (done by Hyper-V) space on virtual machine.
STOP: c000021a {Fatal System Error} The initial session process or system process terminated unexpectedly with a status of (0x00000000) (0xc000012d 0x001003f0).
The virtual server there is Windows Server 2008 R2 Enterprise Edition, which is also Domain Controller, now my whole environment is down :/
I've tried to repair Windows - but there is no restore point, and using command line, I've also tried the sfc /SCANNOW /OFFBOOTDIR /OFFWINDIR, but got error "Windows Resource
Protection could not perform the requested operation"
I initially responded to this question to ask if Christof ever found a solution. That's not allowed, so my post was deleted.
I'm back to share that I solved the above problem for myself using a mix-and-match set of backed-up registry files. I believe the only reason this worked for me is that there had been ZERO changes to the server between the different times the registry files were backed up. Most of the registry files I used in the recovery were from c:\windows\system32\config\system\regbak, but the SOFTWARE file had a timestamps too close to the time of my initial failure, so I used one that I had created in \windows\tmp when I initially began this recovery process. I followed a guide which apparently was deleted but you can find references by searching on site:microsoft.com kb307545, Also make sure you have a backup of the COMPONENTS hive/file.

Resources