I am developing software in an embedded system (512 MB RAM). I'm using redis to take the place of a shared memory between processes inside a django application.
We are talking about 150 values, stored every second, coming from a MODBUS device. They all have the same key and their expire time is 10 minutes.
After some work hours (tipically a day), redis ceases to function, due to memory problems. Can someone help me out?
output of ps aux | grep redis
redis 1934 1.9 2.2 76216 8400 ? Ssl 07:49 10:37 /usr/bin/redis-server 127.0.0.1:6379
redis.info
# Server
redis_version:2.8.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:7cc01333adfd1c61
redis_mode:standalone
os:Linux 3.4.79+ armv7l
arch_bits:32
multiplexing_api:epoll
gcc_version:4.6.3
process_id:1934
run_id:be418b5a05b6670bb4bff9c73cc7126589d6b5c8
tcp_port:6379
uptime_in_seconds:33584
uptime_in_days:0
hz:10
lru_clock:858206
config_file:/etc/redis/redis.conf
# Clients
connected_clients:138
client_longest_output_list:54
client_biggest_input_buf:0
blocked_clients:0
# Memory
used_memory:6893800
used_memory_human:6.57M
used_memory_rss:6152192
used_memory_peak:47902480
used_memory_peak_human:45.68M
used_memory_lua:25600
mem_fragmentation_ratio:0.89
mem_allocator:jemalloc-3.0.0
# Persistence
loading:0
rdb_changes_since_last_save:1370469
rdb_bgsave_in_progress:0
rdb_last_save_time:1434611837
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
# Stats
total_connections_received:155
total_commands_processed:3207775
instantaneous_ops_per_sec:55
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:297
evicted_keys:0
keyspace_hits:2141758
keyspace_misses:12495
pubsub_channels:6
pubsub_patterns:0
latest_fork_usec:0
# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:295.85
used_cpu_user:333.85
used_cpu_sys_children:0.00
used_cpu_user_children:0.00
# Keyspace
db0:keys=2,expires=2,avg_ttl=3185003
db1:keys=937,expires=242,avg_ttl=585637
snippet of redis-server.log
[1969] 18 Jun 22:17:21.819 # Server started, Redis version 2.8.6
[1969] 18 Jun 22:17:21.919 * DB loaded from disk: 0.100 seconds
[1969] 18 Jun 22:17:21.919 * The server is now ready to accept connections on port 6379
[1969] 18 Jun 22:17:21.919 * The server is now ready to accept connections at /var/run/redis/redis.sock
[1969] 19 Jun 09:16:50.444 # Client addr=127.0.0.1:38745 fd=9 name= age=39516 idle=4330 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=13724 oll=415 omem=8509160 events=rw cmd=subscribe scheduled to be closed ASAP for overcoming of output buffer limits.
[1969] 19 Jun 09:20:54.056 # Client addr=127.0.0.1:38759 fd=14 name= age=4713 idle=4331 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=13688 oll=415 omem=8509160 events=rw cmd=subscribe scheduled to be closed ASAP for overcoming of output buffer limits.
[1941] 19 Jun 09:17:17.134 # Unable to set the max number of files limit to 10032 (Operation not permitted), setting the max clients configuration to 3984.
Some lines from redis-cli monitor, if someone finds that useful. Being the same keys rewritten over and over again, it puzzles me with the high amount of memory used and all those files descriptors.
http://pastebin.com/rQqThUHF
Related
Amazon Linux 1:
Server version: Apache/2.4.54 (Amazon)
Server built: Jul 11 2022 21:47:38
Server's Module Magic Number: 20120211:124
Server loaded: APR 1.6.3, APR-UTIL 1.5.4, PCRE 8.21 2011-12-12
PHP 7.3.30 (cli) (built: Oct 6 2021 20:34:22) ( NTS )
Amazon Linux 2:
Server version: Apache/2.4.54 ()
Server built: Jun 30 2022 11:02:23
Server's Module Magic Number: 20120211:124
Server loaded: APR 1.7.0, APR-UTIL 1.6.1, PCRE 8.32 2012-11-30
PHP 7.4.30 (cli) (built: Jun 23 2022 20:19:00) ( NTS )
The server's are configured via automation and loaded into ALB/ASG's.
Instance size is m4.large (2x vCPU, 8GiB Memory)
Auto-Scaling group is configured with Min:4 Max:8
This is what my httpd.conf file looks like:
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxConnectionsPerChild 4000
</IfModule>
On Amazon Linux 1, the site works as expected. The ASG spawns 4 instances and they each hover at 40-60% CPU utilization during peak hours.
The same build on Amazon Linux 2 yields wildly different results. All instances immediately get bombarded by a huge number of httpd processes.
The ASG scales up to 8 instances with every single one at 90%+ CPU usage. The instances start to lock up which causes the target group to mark them as "unhealthy" and the ASG to rotate them out endlessly. The website obviously does not work.
What could be causing them to behave so differently? What steps can I take to try and mitigate this? I'm honestly pretty new to all this so I don't know where to start.
Code
To reproduce requires two application running and connecting to each other through TCP. So I've made a tiny repo that also includes the powershell build script. link to the full repo
However to avoid the extra click, here is the code for clientA.go.
package main
import (
"fmt"
"net"
"time"
)
func main() {
clientA, err := net.ResolveTCPAddr("tcp4", fmt.Sprintf(":%v", "2222"))
if err != nil {
fmt.Println(err)
return
}
clientB, err := net.ResolveTCPAddr("tcp4", fmt.Sprintf(":%v", "3333"))
if err != nil {
fmt.Println(err)
return
}
for {
clientAtoB, err := net.DialTCP("tcp4", clientA, clientB)
if err != nil {
fmt.Println(err)
} else {
defer clientAtoB.Close()
clientAtoB.SetLinger(0)
clientAtoB.SetNoDelay(true)
clientAtoB.SetKeepAlive(false)
fmt.Println("connected as Client A!")
buffer := make([]byte, 64)
_, err = clientAtoB.Read(buffer)
if err != nil {
continue
}
}
time.Sleep(time.Second)
}
}
The code for clientB.go is identical except the local and remote endpoints are swapped around:
clientBtoA, err := net.DialTCP("tcp4", clientB, clientA)
Problem
I build the same go code for both Windows and Linux but at runtime the applications produce different results. Specifically with how TCP connections are dialed on each platform.
On Windows, when I run the two executables clientA.exe and clientB.exe (built from the build.ps1 script) I get the desired result. As seen in this screenshot:
However when I upload and execute the Linux binaries, the result is different:
ubuntu#ip-172-31-16-224:~/go/src/github.com/fanmanpro/dial-vs-listen$ sudo chmod +x clientA clientB
ubuntu#ip-172-31-16-224:~/go/src/github.com/fanmanpro/dial-vs-listen$ ls -la
total 10984
drwxrwxr-x 3 ubuntu ubuntu 4096 Apr 27 03:09 .
drwxrwxr-x 4 ubuntu ubuntu 4096 Apr 27 03:08 ..
drwxrwxr-x 8 ubuntu ubuntu 4096 Apr 27 03:08 .git
-rw-rw-r-- 1 ubuntu ubuntu 11255 Apr 27 03:12 A.txt
-rw-rw-r-- 1 ubuntu ubuntu 11255 Apr 27 03:12 B.txt
-rw-rw-r-- 1 ubuntu ubuntu 247 Apr 27 03:08 build.ps1
-rwxrwxr-x 1 ubuntu ubuntu 2950662 Apr 27 03:08 clientA
-rw-rw-r-- 1 ubuntu ubuntu 2642944 Apr 27 03:08 clientA.exe
-rw-rw-r-- 1 ubuntu ubuntu 718 Apr 27 03:08 clientA.go
-rwxrwxr-x 1 ubuntu ubuntu 2950662 Apr 27 03:08 clientB
-rw-rw-r-- 1 ubuntu ubuntu 2642944 Apr 27 03:08 clientB.exe
-rw-rw-r-- 1 ubuntu ubuntu 718 Apr 27 03:08 clientB.go
ubuntu#ip-172-31-16-224:~/go/src/github.com/fanmanpro/dial-vs-listen$ ./clientA > A.txt & ./clientB > B.txt &
[1] 24914
[2] 24915
ubuntu#ip-172-31-16-224:~/go/src/github.com/fanmanpro/dial-vs-listen$ cat A.txt
dial tcp4 :2222->:3333: connect: connection refused
ubuntu#ip-172-31-16-224:~/go/src/github.com/fanmanpro/dial-vs-listen$ cat B.txt
dial tcp4 :3333->:2222: connect: connection refused
ubuntu#ip-172-31-16-224:~/go/src/github.com/fanmanpro/dial-vs-listen$
I don't expect the connection refused error since these two applications are running under the same environment, so no firewalls are in effect, and the permissions are identical.
How can I get the same result regardless of platform? Or why are they different in the first place?
Edit
The successful connection on Windows is not just the luck of good timing. On Windows, I can run A for 5 minutes, then when I run B, both connect successfully.
Update (2020-04-27)
After receiving feedback from Go developers, I've been told that this is likely a Linux configuration issue and not specific to Go. Other than permissions, I can't thing of anything that would prevent two applications in the same environment from establishing a TCP connection like this? (These low level Linux stuff isn't really my forte.)
Why this doesn't work on Linux is quite obvious. Both A and B are clients that are connecting to counterpart that needs to listen. On Linux (or UNIX) if you try to run ClientA it will try to dial in to ClientB's address and port. If there's no process already listening on this address and port to accept the connection in that moment ClientA will immediately end up with connection refused error (this is not entirely true, but most of time is, see my EDIT at the end of answer).
On Windows, under the hood Golang uses (for tcp, tcp4 and tcp6 protocols) ConnectEx API which is for connection-oriented sockets. This API behaves different from Linux connect API. If ConnectEx cannot connect immediately it returns error code ERROR_IO_PENDING and behind the scenes OS waits/retries until connection is accepted and established (or it gives up and makes it definitively failed) and then notifies back - this is called overlapped I/O.
Relevant part of MSDN ConnectEx documentation:
Connection-oriented sockets are often unable to complete their connection immediately, and therefore the operation is initiated and the function immediately returns with the ERROR_IO_PENDING or WSA_IO_PENDING error. When the connect operation completes and success or failure is achieved, status is reported using the completion notification mechanism indicated in lpOverlapped.
Now, what happens in your case on Windows is that you try to ConnectEx from both sides and OS connects those sockets for you. This will only work if other side gets connected within certain period. If you try to reasonably increase time.Sleep interval in both clients (e.g. 17 and 28), you can see even on Windows they will have hard time to connect anymore.
Answer to your question is that your code as it is written now depends on OS-specific behavior of TCP dialing in Golang on Windows and is not portable. To fix your software to be portable on any platform supported by Golang you probably want to change logic so both ClientA and ClientB listen for incoming connection and also periodically try to connect to the opposite side.
EDIT: I'm not saying your code can not work on Linux at all. It actually uses rare connection mode called TCP simultaneous connect where you can connect two processes without having any of them listen. Both dialing sides send their SYN simultaneously, so each side responds with SYN/ACK and then ACK to complete the 3-way handshake and ESTABLISH connection. That requires very precise timing and syncing of the connect call in both clients. Both sides would connect if TCP simultaneous connect is allowed in Linux kernel and that sync between connects is achieved (hardly done by just running both clients by hand or from same script; even simulating within same process and thread is not that easy).
Well it seems I've hit my first issue with my BigInsights Image, not a massive problem, but something to think about. On my Ambari browser services page it was showing that the Kafka service was not running, I tried a restart a number of times, but this seemed to continuously fail. So I figured that I best look into it a bit further. In this case the issue was on the Ambari Master server which has the most services running on it.
So first call of action is to see if maybe Ambari is not making the call correctly:
[root#master ~]# kafka
Usage: /usr/bin/kafka {start|stop|status|clean}
[root#master ~]# kafka status
Kafka is not running.
[root#master ~]# kafka start
Starting Kafka succeeded with PID=15815.
[root#master ~]# kafka status
Kafka is not running.
Next I tired a clean start, not that I figured it would make much difference, but maybe there was a issue with the logs not allowing it to restart:
[root#master ~]# kafka clean
Removed the Kafka PID file: /var/run/kafka/kafka.pid.
Removed the Kafka OUT file: /var/log/kafka/kafka.out.
Removed the Kafka ERR file: /var/log/kafka/kafka.err.
[root#master ~]# kafka status
Kafka is not running. No pid file found.
[root#master ~]# kafka start
Starting Kafka succeeded with PID=15875.
[root#master-01 ~]# kafka status
Kafka is not running.
So lets take a proper look at the logs:
[root#master ~]# ls -ltr /var/log/kafka/
-<cut>-
-rw-r--r-- 1 kafka hadoop 6588 Aug 11 13:55 controller.log.2015-08-11-13
-rw-r--r-- 1 kafka hadoop 6000 Aug 11 13:59 server.log.2015-08-11-13
-rw-r--r-- 1 kafka hadoop 6588 Aug 11 14:55 controller.log
-rw-r--r-- 1 kafka hadoop 5700 Aug 11 14:56 server.log
-rw-r--r-- 1 root root 284 Aug 11 15:09 kafka.err
-rw-r--r-- 1 root root 522 Aug 11 15:09 kafka.out
-rw-r--r-- 1 kafka hadoop 707 Aug 11 15:09 kafkaServer-gc.log
Lets look at the error and out files:
[root#master ~]# cat /var/log/kafka/kafka.err
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c5330000, 986513408, 0) failed; error='Cannot allocate memory' (errno=12)
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c5330000, 986513408, 0) failed; error='Cannot allocate memory' (errno=12)
[root#master ~]# cat /var/log/kafka/kafka.out
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 986513408 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /root/hs_err_pid15875.log
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 986513408 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /root/hs_err_pid16305.log
Ah, that's odd, as I asked for at least 4GB of memory for my VMs, lets check:
[root#master ~]# cat /proc/meminfo
MemTotal: 1922260 kB
MemFree: 278404 kB
Buffers: 8600 kB
Cached: 43384 kB
Best get some more memory allocated!
Normally the minimum that you should install BigInsights with, as recommended by the IBM support pages is 8GB, so this gives you rather a insight into why. At least 2GB of it is just to run the installed services on the system, even before you start loading the DB and running queries.
Most of time, my rsh cycle is general OK, we could get following logs from rshd:
Aug 19 04:36:34 shmm500 authpriv.info in.rshd[21343]: connect from 172.17.0.40 (172.17.0.40)
Aug 19 04:36:34 shmm500 auth.info rshd[21344]: root#172.17.0.40 as root: cmd='echo 481'
While for some error case, the rsh could success but there are several seconds delay, see the following timestamp:
Aug 19 04:12:24 shmm500 authpriv.info in.rshd[17968]: connect from 172.17.0.40 (172.17.0.40)
Aug 19 04:12:27 shmm500 auth.info rshd[17972]: root#172.17.0.40 as root: cmd='echo 18'
I also found that, for most normal case, the PID increased by 1, while for most error case, PID increasd by 4, see the PID in above logs, seems rshd forks some processes. So would you provide any explanation for why rshd took these several seconds and PID increase.
Our rsh is the old rsh, not ssh, I'm not sure, but seems the rsh is from netkit. And this is an embedded board with busybox, no strace/pstack.
For client side, I just 'rsh 172.17.0.8 pwd', not hostname is used.
Answer the question by myself:
This issue was caused by a frame loss. Either SYN or SYN+ACK in 3-way handshake was dropped at a rare rate for some reason, anyway the client peer didn't get the SYN+ACK within in 3 seconds timeout(this timeout is hardcoded in Linux kernel), then the connect() resent SYN again, and usually successful at the second try.
From the viewpoint of application, we got 3 seconds delay, or even 6 seconds if it failed at the second try.
Other relevant information:
The first log is from tcpd(aka tcp wrapper)
Aug 19 04:36:34 shmm500 authpriv.info in.rshd[21343]: connect from 172.17.0.40 (172.17.0.40)
The second log is from rshd in netkit 0.17
Aug 19 04:36:34 shmm500 auth.info rshd[21344]: root#172.17.0.40 as root: cmd='echo 481'
rsh need two tcp connections, the first is from rsh client to rshd, and the second tcp connection is from rshd to rsh client, which means the rshd is the tcp client. And my issue is frame loss on the second tcp connection.
We are encountering clock drift issues with our MongoDB replica set running on AWS. This just seemed to start happening recently after we added additional data to the set, before then we did not really notice this issue unless the system was under heavy load. The following error is logged in the mongod.log file sporadically and the system is not under load.
To test this we have isolated a set of machines with the same dataset and not in use by our web application though the error is still occurring;
2014-12-12T13:33:51.333+0000 [rsBackgroundSync] changing sync target
because current sync target's most recent OpTime is Dec 12 13:32:42:c
which is more than 30 seconds behind member mongo1:27017 whose most
recent OpTime is 1418391230
From the above the time stamp shows that one of the mongodb replica set members is over a minute behind. The worst we have seen is 12 minutes out of sync.
This error in turn causes replication lag and we receive the notification about this from the Mongo Monitoring Service although it does correct itself.
The setup is 3 x r3.xlarge AWS Linux instances, 1 in each availability zone of the EU-West-1A region. The machines have been setup using the Mongo recommended settings with a Raid array and the cloud formation scripts provided by Mongo. The data is around 4GB in size.
We think the issue is related to the NTP sync, by default on the AWS Linux Amazon Machine Image the ntpd service is configured to go to a pool of aws ntp servers hosted on www.pool.ntp.org.
To try and rule this out we setup our own NTP server on AWS that the MongoDB servers could sync to. The issue still occurred so we changed the maxpoll and minpoll time for the ntpd service on the mongo machines to sync the time every 16 seconds from the NTP server but the error is still occurring.
We increased the MongoDB OpLog size as well to see if that would make any difference but it didn’t.
Does anyone else encounter this type of issue? Is there something we are missing?
Cheers,
Colin.
ps -ef |grep ntp;
mongodb1
ntp 5163 1 0 Dec11 ? 00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
ec2-user 15865 15839 0 09:31 pts/2 00:00:00 grep ntp
mongodb2
ntp 4834 1 0 Dec11 ? 00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
ec2-user 19056 19029 0 09:31 pts/0 00:00:00 grep ntp
mongodb3
ntp 5795 1 0 Dec11 ? 00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
ec2-user 26199 26173 0 09:31 pts/0 00:00:00 grep ntp
cat /etc/ntp.conf;
# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).
driftfile /var/lib/ntp/drift
# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
# Permit all access over the loopback interface. This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict -6 ::1
# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.amazon.pool.ntp.org iburst dynamic
#server 1.amazon.pool.ntp.org iburst dynamic
#server 2.amazon.pool.ntp.org iburst dynamic
#server 3.amazon.pool.ntp.org iburst dynamic
server time-server.domain.com iburst
#broadcast 192.168.1.255 autokey # broadcast server
#broadcastclient # broadcast client
#broadcast 224.0.1.1 autokey # multicast server
#multicastclient 224.0.1.1 # multicast client
#manycastserver 239.255.254.254 # manycast server
#manycastclient 239.255.254.254 autokey # manycast client
# Enable public key cryptography.
#crypto
includefile /etc/ntp/crypto/pw
# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography.
keys /etc/ntp/keys
# Specify the key identifiers which are trusted.
#trustedkey 4 8 42
# Specify the key identifier to use with the ntpdc utility.
#requestkey 8
# Specify the key identifier to use with the ntpq utility.
#controlkey 8
# Enable writing of statistics records.
#statistics clockstats cryptostats loopstats peerstats
# Enable additional logging.
logconfig =clockall =peerall =sysall =syncall
# Listen only on the primary network interface.
interface listen eth0
interface ignore ipv6
ntpq -npcrv;
remote refid st t when poll reach delay offset jitter
==============================================================================
*172.31.14.137 91.*.*.* 3 u 557 1024 377 1.121 -0.264 0.161
associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync,
version="ntpd 4.2.6p5#1.2349-o Sat Mar 23 00:37:31 UTC 2013 (1)",
processor="x86_64", system="Linux/3.14.23-22.44.amzn1.x86_64", leap=00,
stratum=4, precision=-23, rootdelay=23.597, rootdisp=109.962,
refid=172.31.14.137,
reftime=d83a757a.175b5fa1 Tue, Dec 16 2014 9:10:18.091,
clock=d83a77a7.82431efa Tue, Dec 16 2014 9:19:35.508, peer=27361,
tc=10, mintc=3, offset=-0.264, frequency=-13.994, sys_jitter=0.000,
clk_jitter=0.358, clk_wander=0.053
After upgrading to MongoDB 3 using the WiredTiger storage engine we do not see this issue any more.