Log into an additional file at Apache Karaf - log4j

I have a system that uses both Apache Camel and Karaf.
I need some specific processor to log into a new log instead of the default one ,karaf.log.
I have reached that there is a file called "org.ops4j.pax.logging.cfg" that is responsible for logging configuration .
Here it's the file before I mess with it :
################################################################################
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
################################################################################
# Root logger
log4j.rootLogger=INFO, out, osgi:VmLogAppender
log4j.throwableRenderer=org.apache.log4j.OsgiThrowableRenderer
# CONSOLE appender not used by default
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{ABSOLUTE} | %-5.5p | %-16.16t | %-32.32c{1} | %-32.32C %4L | %X{bundle.id} - %X{bundle.name} - %X{bundle.version} | %m%n
# File appender
log4j.appender.out=org.apache.log4j.RollingFileAppender
log4j.appender.out.layout=org.apache.log4j.PatternLayout
log4j.appender.out.layout.ConversionPattern=%d{ABSOLUTE} | %-5.5p | %-16.16t | %-32.32c{1} | %-32.32C %4L | %X{bundle.id} - %X{bundle.name} - %X{bundle.version} | %m%n
log4j.appender.out.file=${karaf.data}/log/karaf.log
log4j.appender.out.append=true
log4j.appender.out.maxFileSize=1MB
log4j.appender.out.maxBackupIndex=10
# Sift appender
log4j.appender.sift=org.apache.log4j.sift.MDCSiftingAppender
log4j.appender.sift.key=bundle.name
log4j.appender.sift.default=karaf
log4j.appender.sift.appender=org.apache.log4j.FileAppender
log4j.appender.sift.appender.layout=org.apache.log4j.PatternLayout
log4j.appender.sift.appender.layout.ConversionPattern=%d{ABSOLUTE} | %-5.5p | %-16.16t | %-32.32c{1} | %-32.32C %4L | %m%n
log4j.appender.sift.appender.file=${karaf.data}/log/$\\{bundle.name\\}.log
log4j.appender.sift.appender.append=true
Now I have added an extra File appender :
# Additional File appender
log4j.rootLogger=INFO, out, osgi:VmLogAppender # I removed the first log4j.rootLogger
log4j.appender.new=org.apache.log4j.RollingFileAppender
log4j.appender.new.layout=org.apache.log4j.PatternLayout
log4j.appender.new.layout.ConversionPattern=%d{ABSOLUTE} | %-5.5p | %-16.16t | %-32.32c{1} | %-32.32C %4L | %X{bundle.id} - %X{bundle.name} - %X{bundle.version} | %m%n
log4j.appender.new.file=${karaf.data}/log/new.log
log4j.appender.new.append=true
log4j.appender.new.maxFileSize=1MB
log4j.appender.new.maxBackupIndex=10
And from Camel , I logged from DSL :
process(exceptionProcessor).
process(doSmth).log(LoggingLevel.INFO,"new","That is a new file to log into")
The result now that it logs erverything on BOTH "new.log" & "karaf.log" file .
The question is : how could I log into"new.log" just in case a specific processor .
Others will be logged into the karaf.log !!

The solution for my problem is :
1- u should update "org.ops4j.pax.logging.cfg" and add a new file appender .
In my case my new file appender is "new"
################################################################################
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
################################################################################
# Root logger
log4j.rootLogger=INFO, out,osgi:VmLogAppender
log4j.logger.new=INFO,new
log4j.throwableRenderer=org.apache.log4j.OsgiThrowableRenderer
# CONSOLE appender not used by default
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{ABSOLUTE} | %-5.5p | %-16.16t | %-32.32c{1} | %-32.32C %4L | %X{bundle.id} - %X{bundle.name} - %X{bundle.version} | %m%n
# new File appender
log4j.appender.new=org.apache.log4j.RollingFileAppender
log4j.appender.new.layout=org.apache.log4j.PatternLayout
log4j.appender.new.layout.ConversionPattern=%d{ABSOLUTE} | %-5.5p | %-16.16t | %-32.32c{1} | %-32.32C %4L | %X{bundle.id} - %X{bundle.name} - %X{bundle.version} | %m%n
log4j.appender.new.file=${karaf.data}/log/new.log
log4j.appender.new.append=true
log4j.appender.new.maxFileSize=1MB
log4j.appender.new.maxBackupIndex=10
# File appender
log4j.appender.out=org.apache.log4j.RollingFileAppender
log4j.appender.out.layout=org.apache.log4j.PatternLayout
log4j.appender.out.layout.ConversionPattern=%d{ABSOLUTE} | %-5.5p | %-16.16t | %-32.32c{1} | %-32.32C %4L | %X{bundle.id} - %X{bundle.name} - %X{bundle.version} | %m%n
log4j.appender.out.file=${karaf.data}/log/karaf.log
log4j.appender.out.append=true
log4j.appender.out.maxFileSize=1MB
log4j.appender.out.maxBackupIndex=10
# Sift appender
log4j.appender.sift=org.apache.log4j.sift.MDCSiftingAppender
log4j.appender.sift.key=bundle.name
log4j.appender.sift.default=karaf
log4j.appender.sift.appender=org.apache.log4j.FileAppender
log4j.appender.sift.appender.layout=org.apache.log4j.PatternLayout
log4j.appender.sift.appender.layout.ConversionPattern=%d{ABSOLUTE} | %-5.5p | %-16.16t | %-32.32c{1} | %-32.32C %4L | %m%n
log4j.appender.sift.appender.file=${karaf.data}/log/$\\{bundle.name\\}.log
log4j.appender.sift.appender.append=true
2- I use DSL to log :
process(doSmth).log(LoggingLevel.INFO,"new","HelloZ my new Logger ${exception.stacktrace}")
Enjoy :)

Related

kafka + too many open files between Kafka brokers or may between broker and clients

we are facing very major problem with our Apache Kafka servers
our Kafka servers are creased because "Too many open files"
we have production Kafka cluster with 7 machines , while Kafka version is 0.1
we already increases twice the "kafka_user_nofile_limit" to 500000 , while the original value is 128000
but again we are hit the high value and brokers are crashed
additional we can see clearly that many lines as "Too many open files" are in Kafka logs
we saw the following Jira case
https://issues.apache.org/jira/browse/KAFKA-3317
but we not see what is the solution for this jira or what is the workaround for this major problem ?
from jira
Bug
Status:OPEN
Priority: Major
Resolution:Unresolved
Affects Version/s:
0.9.0.0, 0.9.0.1
Fix Version/s:None
Component/s:None
Labels:None
Environment:CentOS release 6.5 (Final)
kafka_2.11-0.9.0.1
Flags:
Important
Description
there is much open files in my java app client which write msg to kafka broker, i used
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.9.0.1</version>
</dependency>
new api to write msg. when i use
lsof -p 8780 | wc -l
19184
lsof -p 8780 | grep XmlIpcRegSvc | wc -l
4920
lsof -p 8780 | grep pipe | wc -l
9576
lsof -p 8780 | grep eventpoll | wc -l
4792
where 8780 is my java app pid.
java 37121 app *796u IPv6 960673997 0t0 TCP mad183:50213->mad180:XmlIpcRegSvc (ESTABLISHED)
there are many ESTABLISHED XmlIpcRegSvc , and seems not closed.
and It likes use ipv6 .
example from my kafka machines
more /etc/security/limits.d/kafka.conf
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
kafka - nofile 500000
kafka - nproc 65536

Operating Systems used by AWS RDS

What operating systems does Amazon RDS use. While I understand that when using RDS we are just exposed to a endpoint and internally the database we use might be supported by multiple systems, I would like to know what is the OS used by those systems.
To check the underlying operating of your MySQL DB instance on AWS RDS, you can use the following command:
mysql> SHOW variables LIKE '%version%';
Result:
+-------------------------+------------------------------+
| Variable_name | Value |
+-------------------------+------------------------------+
| innodb_version | 5.6.39 |
| protocol_version | 10 |
| slave_type_conversions | |
| version | 5.6.39-log |
| version_comment | MySQL Community Server (GPL) |
| version_compile_machine | x86_64 |
| version_compile_os | Linux |
+-------------------------+------------------------------+
7 rows in set (0.01 sec)
Those systems are running under the Amazon Linux Distribution

ModemManager: mmcli get signal strength

I am unable to retrieve the signal strength on a Sierre Wireless MC7304.
Also sending an AT command via mmcli does not seem to work.
:~ $ sudo mmcli -m /org/freedesktop/ModemManager1/Modem/0 --signal-get
error: modem has no extended signal capabilities
:~ $ sudo mmcli -m /org/freedesktop/ModemManager1/Modem/0 --command=AT+CSQ
error: command failed: 'GDBus.Error:org.freedesktop.ModemManager1.Error.Core.Unauthorized: Cannot send AT command to modem: operation only allowed in debug mode'
/org/freedesktop/ModemManager1/Modem/0 (device id
'dd26a5483d051c6e25a7ca10f4d36a94791c7ebf')
-------------------------
Hardware | manufacturer: 'Sierra Wireless, Incorporated'
| model: 'MC7304'
| revision: 'SWI9X15C_05.05.16.02 r21040 carmd-fwbuild1
2014/03/17 23:49:48'
| supported: 'gsm-umts, lte'
| current: 'gsm-umts, lte'
| equipment id: 'unknown'
-------------------------
System | device:
'/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.5'
| drivers: 'qmi_wwan, qcserial'
| plugin: 'Gobi'
| primary port: 'ttyUSB2'
| ports: 'ttyUSB0 (qcdm), ttyUSB2 (at), wwan0 (net),
wwan1 (net)'
-------------------------
Numbers | own : 'unknown'
-------------------------
Status | lock: 'none'
| unlock retries: 'unknown'
| state: 'connected'
| power state: 'on'
| access tech: 'lte'
| signal quality: '0' (recent)
-------------------------
Modes | supported: 'allowed: 2g, 3g, 4g; preferred: none'
| current: 'allowed: 2g, 3g, 4g; preferred: none'
-------------------------
Bands | supported: 'unknown'
| current: 'unknown'
-------------------------
IP | supported: 'ipv4, ipv6, ipv4v6'
-------------------------
3GPP | imei: 'unknown'
| enabled locks: 'none'
| operator id: 'hidden'
| operator name: 'hidden'
| subscription: 'unknown'
| registration: 'home'
-------------------------
SIM | path: '/org/freedesktop/ModemManager1/SIM/0'
-------------------------
Bearers | paths: '/org/freedesktop/ModemManager1/Bearer/0'
When using the modem inside for example an Mikrotik 922 card i seem to get the signal strength just fine.
I am using mmcli version: 1.4.0-1 and Kernel 4.450 on a raspberry pi.
Several issues here:
The modem wasn't properly detected as a QMI modem by ModemManager (see that there is no cdc-wdm port listed in the Ports section output of mmcli). Being this a custom system, you need to make sure that the kernel USB cdc-wdm driver is built and installed, otherwise you won't be able to use the QMI mode as there won't be any /dev/cdc-wdm port to probe
You were not able to run AT commands via mmcli --command because that operation is only allowed when the ModemManager daemon runs in --debug mode. This is a command that developers use to run quick checks on the modem during development, not supported for general users.
You are seeing a signal quality equal to 0 in the mmcli output because ModemManager had only one AT port to use (ttyUSB2) and if that was connected (with PPP) there was no way to refresh the quality. If you used the modem in QMI mode you wouldn't be using PPP at all over the TTY.
Replying to the other answer in this question: ModemManager of course supports signal quality retrieval in QMI modems! Actually, the special Signal interface (which provides more detailed signal quality values, like access technology specific items) was originally developed only for QMI modems themselves.
The MC7304 modem is using QMI, Modemmanager does not support signal strength via QMI.
Use libqmi instead:
sudo qmicli -d /dev/cdc-wdm0 --nas-get-signal-strength

Running cudaHashcat-1.33 on AWS g2.2xlarge - Error cuModuleLoad() 209 when trying cudaExample0.sh

As it says in the description I have installed cudaHashcat-1.33 on an AWS g2.2xlarge instance.
I've used the .run file to install the CUDA Toolkit and then performed this test: deviceQuery ; as explained here in the official documentation (http://docs.nvidia.com/cuda/cuda-getting-started-guide-for-linux/index.html#running-binaries).
Then I installed cudaHashcat-1.33, following these instructions.
sudo apt-get install p7zip-full
wget http://hashcat.net/files/cudaHashcat-1.33.7z
7za x cudaHashcat-1.33.7z
cd cudaHashcat-1.33
Then I tried to run this: cudaExample0.sh in ~/cudaHashcat-1.33/cudaExample0.sh and I end up getting this output:
cudaHashcat v1.33 starting...
Device #1: GRID K520, 4095MB, 797Mhz, 8MCU
Device #1: WARNING! Kernel exec timeout is not disabled, it might cause you errors of code 702
Hashes: 6494 hashes; 6494 unique digests, 1 unique salts
Bitmaps: 16 bits, 65536 entries, 0x0000ffff mask, 262144 bytes
Applicable Optimizers:
* Zero-Byte
* Precompute-Init
* Precompute-Merkle-Demgard
* Meet-In-The-Middle
* Early-Skip
* Not-Salted
* Not-Iterated
* Single-Salt
* Scalar-Mode
* Raw-Hash
Watchdog: Temperature abort trigger set to 90c
Watchdog: Temperature retain trigger set to 80c
ERROR: cuModuleLoad() 209
A second example is this one, where I actually use the file I want to attack.
ubuntu#ip-172-31-58-154:~$ ~/maskprocessor/src/mp64.bin ?l?l?l?l?l?l?l?l | ~/cudaHashcat-1.33/cudaHashcat64.bin -m 2500 xxx.hccap
cudaHashcat v1.33 starting...
Device #1: GRID K520, 4095MB, 797Mhz, 8MCU
Device #1: WARNING! Kernel exec timeout is not disabled, it might cause you errors of code 702
Hashes: 1 hashes; 1 unique digests, 1 unique salts
Bitmaps: 8 bits, 256 entries, 0x000000ff mask, 1024 bytes
Rules: 1
Applicable Optimizers:
* Zero-Byte
* Single-Hash
* Single-Salt
Watchdog: Temperature abort trigger set to 90c
Watchdog: Temperature retain trigger set to 80c
ERROR: cuModuleLoad() 209
nvidia-smi
[root#ip-xxx-xxx-xxx-xxx cudaHashcat-1.33]$ nvidia-smi
Wed Mar 4 19:07:35 2015
+------------------------------------------------------+
| NVIDIA-SMI 340.32 Driver Version: 340.32 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GRID K520 On | 0000:00:03.0 Off | N/A |
| N/A 43C P8 17W / 125W | 10MiB / 4095MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| No running compute processes found |
+-----------------------------------------------------------------------------+
If someone knows what is going on, I'd appreciate any help.
So after a lot of searching through forums I finally found an answer. #Robert Crovella, thanks for pointing out that the driver was the wrong one. So it turns out that finding the linux drivers for NVIDIA is not that easy, but I came across this page, which then lead me to the Linux Drivers of NVIDIA. Just download the driver required for your architecture (if you use wget click on 'Download' first, since there is an acceptance page). After that do 'chmod +x nvidia-driver.run' and then install it with 'sudo ./nvidia-driver.run'.
Hope that my experience helps someone else.

What is at physical memory 0x8000 (32Kb) to 0x10000 (1Mb) on Linux

I'm compiling the kernel with a custom kernel module that prints out the kernel's code start and end (physical) addresses. It starts at 0x8000 and ends at 0xefe6d8. Looking through the generated System.map, I see that almost all functions in the kernel sit at 0x10000 (1Mb) in physical memory and onwards. But the code starts at 0x8000. I cannot figure out what lives in between those two addresses. Can anyone shed some light on this?
Snippet from System.map (virtual mapping starts on 0xc0000000):
c0008000 T _text
c0008000 T stext
c000804c t __create_page_tables
c000814c t __turn_mmu_on_loc
c0008158 t __vet_atags
c0100000 T __exception_text_start
The __create_page_tables function is indicative that the page tables live after the __vet_atags code. But why would they be part of executable memory?
From the kernel boot procotol, the kernel memory layout is as follows:
~ ~
| Protected-mode kernel |
100000 +------------------------+
| I/O memory hole |
0A0000 +------------------------+
| Reserved for BIOS | Leave as much as possible unused
~ ~
| Command line | (Can also be below the X+10000 mark)
X+10000 +------------------------+
| Stack/heap | For use by the kernel real-mode code.
X+08000 +------------------------+
| Kernel setup | The kernel real-mode code.
| Kernel boot sector | The kernel legacy boot sector.
X +------------------------+
| Boot loader | <- Boot sector entry point 0000:7C00
001000 +------------------------+
| Reserved for MBR/BIOS |
000800 +------------------------+
| Typically used by MBR |
000600 +------------------------+
| BIOS use only   |
000000 +------------------------+

Resources