I'm new to Db2 and just installed v11.5 on my Ubuntu 18.04.
I referred these two links for setup purpose:
IBM and DCON
I'm using DB2 CLI to create a database. On typing in create database <database_name> and press enter, it just stays there; there's no output.
I checked the db2diag.log as well, it stops at this:
2021-05-18-15.41.46.618309+330 E653248E505 LEVEL: Event
PID : 29136 TID : 140104312022784 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : SOURCE
APPHDL : 0-7 APPID: *LOCAL.db2inst1.210518101139
AUTHID : DB2INST1 HOSTNAME: Host
EDUID : 23 EDUNAME: db2agent (instance) 0
FUNCTION: DB2 UDB, base sys utilities, sqeLocalDatabase::FirstConnect, probe:1000
START : DATABASE: SOURCE : ACTIVATED: NO
Tried 3-4 times; on one occasion I just let it be and it took around 30-40 mins but it created the database. I'm not sure if I'm missing out any initialization step.
Kindly guide.
System Spec:
RAM: 16GB, CPU(s): 8, Model name: Intel(R) Core(TM) i5-1035G1 CPU # 1.00GHz
The time it takes to create your skeleton Db2-LUW database is mainly determined by your I/O and logging configuration, and whether or not you get swapping/paging. The CPU speed is less important than the I/O throughput for the create database action.
As an example: my Db2-LUW v11.5 on ubuntu 18.04 and 20.04, the create database completes with the following times as reported by the time tool (with zero paging) and no containerization/virtualization:
with nvme(ssd): around 2 minutes
with spinning disk ext4 4k sector size, 256mb cache sata3, 3.5inch 7200rpm: around 4 minutes
with spinning disk ext4 512byte sector size, 64mb disk cache sata3 2.5 inch 5400rpm : around 10 minutes.
If you have more than one physical drive and/or controller, it can help to put the transaction logs on a different drive / controller to the tablespaces.
So you can see that the performance varies greatly with the I/O configuration. You get what you pay for, and how you configure it.
For creation of other objects on the skelton database like tablespaces, tables, views, indexes, mqt's, routines etc, the performance will vary further, again dependent on your I/O and logging configuration.
Related
I've consumer-applications which reads (no-write) the database of size ~4GiB and performs some tasks. To make sure same database is not duplicated across applications, I've stored it on all node machines of k8s-cluster.
daemonset
I've used one daemonset which is using "hostpath" volume. The daemonset pod extracts the database on each node machine (/var/lib/DATABASE).
For health-check of daemonset pod, I've written the shell script which checks the modification time of the database file (using date command).
For database extraction, approximately 300MiB memory is required and to perform health-check 50MiB is more than sufficient. Hence I've set the memory-request as 100MiB and memory-limit as 1.5GiB.
When I run the daemonset, I observed memory usage is high ~300MiB for first 10 seconds (to perform database extraction) and after that it goes down to ~30MiB. The daemonset works fine as per my expectation.
Consumer Application
Now, The consumer applications (written in golang) pods are using same "hostPath" volume (/var/lib/DATABASE) and reading the database from that location (/var/lib/DATABASE). This consumer applications does not perform any write operations on /var/lib/DATABASE directory.
However, when I deploy this consumer application on k8s then I see huge increase in memory usage of the daemonset-pod from 30MiB to 1.5GiB. The memory-usage by daemonset-pods is almost same as that of memory-limit.
I am not able to understand this behaviour, why consumer application is causing memory usage of daemonset pod ?
Any help/suggestion/truobleshooting steps would be of great help !!
Note : I'm using 'kubernetes top" command to measure the memory (working-set-bytes).
I've found this link (Kubernetes: in-memory shared cache between pods),
which says
hostPath by itself poses a security risk, and when used, should be scoped to only the required file or directory, and mounted as ReadOnly. It also comes with the caveat of not knowing who will get "charged" for the memory, so every pod has to be provisioned to be able to absorb it, depending how it is written. It also might "leak" up to the root namespace and be charged to nobody but appear as "overhead"
However, I did not find any reference from official k8s documentation. It would be helpful if someone can elaborate on it.
Following are the content of memory.stat file from daemonset pod.
cat /sys/fs/cgroup/memory/memory.stat*
cache 1562779648
rss 1916928
rss_huge 0
shmem 0
mapped_file 0
dirty 0
writeback 0
swap 0
pgpgin 96346371
pgpgout 95965640
pgfault 224070825
pgmajfault 0
inactive_anon 0
active_anon 581632
inactive_file 37675008
active_file 1522688000
unevictable 0
hierarchical_memory_limit 1610612736
hierarchical_memsw_limit 1610612736
total_cache 1562779648
total_rss 1916928
total_rss_huge 0
total_shmem 0
total_mapped_file 0
total_dirty 0
total_writeback 0
total_swap 0
total_pgpgin 96346371
total_pgpgout 95965640
total_pgfault 224070825
total_pgmajfault 0
total_inactive_anon 0
total_active_anon 581632
total_inactive_file 37675008
total_active_file 1522688000
total_unevictable 0
I'm currently using MariaDB version 10.5.12 and the mariadb-connector-nodejs package to interact with it from NodeJS
The pool is created as such
const pool = MariaDB.createPool({
host: process.env.DB_HOST,
user: process.env.DB_USER,
password: process.env.DB_PASS,
database: process.env.DB_DATABASE,
connectionLimit: 25
});
A simple request such as connection.Query("SELECT * FROM Users"); Where the users table only contains 5 users can take anywhere between 90ms and 650ms
profiling the query in MariaDB says the query itself only takes a few nanoseconds.
The time a request takes seems to be pretty random, the first request could be 100ms and the second one right after using the same connection could take 600ms which I feel is way too high for a query that takes less than a millisecond to process.
But I'm unable to find where the issue is coming from, I have tried doing all the changes mysqltuner recommended, I've tried disabling name resolve with no visible speedup (I assume the requests are being cached as all requests come from the same ip)
top indicates there is NO Swap space available.
Consider enabling 6G of swap space to survive busy situations with minimal delay and a surviving system.
Please share the code generally used to Connect, Process, Close connections.
Your threads_connected count indicates the Close is being missed and has left 83 threads connected in 10 days.
Suggestions to consider for your my.cnf [mysqld] section,
log_error=/var/log/mysql/mariadb-error.log # from 0 to allow viewing ERRORS detected.
innodb_max_dirty_pages_pct_lwm=1 # from 0 percent to enable pre-flushing
innodb_max_dirty_pages_pct=1 # from 90 percent to minimize innodb_buffer_pool_pages_dirty of 367.
innodb_adaptive_hash_index=ON # from OFF to minimize deadlocks
max_connect_errors=10 # from 100 to frustrate hackers/crackers after 10 attempts.
connect_timeout=30 # from 10 seconds to be more tolerant of connection attempts.
Observations,
connections were 120,604 in 10 days and you had 28984 aborted_connects about 1/4 of the events. You must have a lot of unhappy people trying to use your system or they are hackers trying to break in.
View profile for contact info and free downloadable Utility Scripts to assist with performance tuning.
Is it possible to monitor all write access to the filesystem of all process under linux?
I've some different mounted filesystems. A lot of them are tempfs.
I'm interested in all writes to the root filesystem except the tempfs,devtmpfs etc.
I'm looking for something that will output: <PID xy> write n Bytes to /targe/filepath.
What monitoring tool can list all this write syscalls? Can they be filtered by mount points?
iotop (kernel version 2.6.20 or higher) or dstat could help you. E.g. iotop -o -b -d 10 like discussed in this similar thread.
/proc/diskstats has data for all the block devices.
https://www.kernel.org/doc/Documentation/iostats.txt
The /proc/diskstats file displays the I/O statistics of block devices. Each line contains the following 14 fields:
1 - major number
2 - minor mumber
3 - device name
4 - reads completed successfully
5 - reads merged
6 - sectors read
7 - time spent reading (ms)
8 - writes completed
9 - writes merged
10 - sectors written
11 - time spent writing (ms)
12 - I/Os currently in progress
13 - time spent doing I/Os (ms)
14 - weighted time spent doing I/Os (ms)
For more details refer to Documentation/iostats.txt
You can write a SystemTap script to monitor filesystem operations. Maybe you can visit the Brendan D. Gregg's blog, where there are many monitor tools.
fatrace (File Activity Trace)
fatrace reports file access events (Open, Read, Write, Close) from all running processes. Its main purpose is to find processes which keep waking up the disk unnecessarily and thus prevent some power saving.
When running it outputs one line per event in this format:
<timestamp> <processName(id)>: <accessType> </path/to/file>
For example:
23:10:21.375341 Plex Media Serv(2290): W /srv/dev-disk-by-uuid-UID/Plex/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log
From which you easily get the all necessary infos
Timestamp from the --timestamp option
Process name (who is accessing)
File operation (O-pen R-read W-rite C-lose)
Filepath (where is it writing to).
You can limit the search scope with --current-mount to only record events on partition/mount of current directory.
So simply cd into the volume which corresponds to your spinning HDD first, and there run ftrace with the --current-mount option.
Without this option, all (real) partitions/mount points are being watched.
Very practical
With it I found out easily that the reason why my NAS disk was spinning 24/7 also when nobody accessed the NAS and also no maintenance tasks where about to run was unnecessary logging of the Plex Media Server.
I am having issues with a large query, that I expect to rely on wrong configs of my postgresql.config. My setup is PostgreSQL 9.6 on Ubuntu 17.10 with 32GB RAM and 3TB HDD. The query is running pgr_dijkstraCost to create an OD-Matrix of ~10.000 points in a network of 25.000 links. Resulting table is thus expected to be very big ( ~100'000'000 rows with columns from, to, costs). However, creating simple test as select x,1 as c2,2 as c3
from generate_series(1,90000000) succeeds.
The query plan:
QUERY PLAN
--------------------------------------------------------------------------------------
Function Scan on pgr_dijkstracost (cost=393.90..403.90 rows=1000 width=24)
InitPlan 1 (returns $0)
-> Aggregate (cost=196.82..196.83 rows=1 width=32)
-> Seq Scan on building_nodes b (cost=0.00..166.85 rows=11985 width=4)
InitPlan 2 (returns $1)
-> Aggregate (cost=196.82..196.83 rows=1 width=32)
-> Seq Scan on building_nodes b_1 (cost=0.00..166.85 rows=11985 width=4)
This leads to a crash of PostgreSQL:
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the
current transaction and exit, because another server process exited
normally and possibly corrupted shared memory.
Running dmesg I could trace it down to be an Out of memory issue:
Out of memory: Kill process 5630 (postgres) score 949 or sacrifice child
[ 5322.821084] Killed process 5630 (postgres) total-vm:36365660kB,anon-rss:32344260kB, file-rss:0kB, shmem-rss:0kB
[ 5323.615761] oom_reaper: reaped process 5630 (postgres), now anon-rss:0kB,file-rss:0kB, shmem-rss:0kB
[11741.155949] postgres invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null), order=0, oom_score_adj=0
[11741.155953] postgres cpuset=/ mems_allowed=0
When running the query I also can observe with topthat my RAM is going down to 0 before the crash. The amount of committed memory just before the crash:
$grep Commit /proc/meminfo
CommitLimit: 18574304 kB
Committed_AS: 42114856 kB
I would expect the HDD is used to write/buffer temporary data, when RAM is not enough. But the available space on my hdd does not change during the processing. So I began to dig for missing configs (expecting issues due to my relocated data-directory) and following different sites:
https://www.postgresql.org/docs/current/static/kernel-resources.html#LINUX-MEMORY-OVERCOMMIT
https://www.credativ.com/credativ-blog/2010/03/postgresql-and-linux-memory-management
My original settings of postgresql.conf are default except for changes in the data-directory:
data_directory = '/hdd_data/postgresql/9.6/main'
shared_buffers = 128MB # min 128kB
#huge_pages = try # on, off, or try
#temp_buffers = 8MB # min 800kB
#max_prepared_transactions = 0 # zero disables the feature
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#replacement_sort_tuples = 150000 # limits use of replacement selection sort
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
#max_stack_depth = 2MB # min 100kB
dynamic_shared_memory_type = posix # the default is the first option
I changed the config:
shared_buffers = 128MB
work_mem = 40MB # min 64kB
maintenance_work_mem = 64MB
Relaunched with sudo service postgresql reload and tested the same query, but found no change in behavior. Does this simply mean, such a large query can not be done? Any help appreciated.
I'm having similar trouble, but not with PostgreSQL (which is running happily): what is happening is simply that the kernel cannot allocate more RAM to the process, whichever process it is.
It would certainly help to add some swap to your configuration.
To check how much RAM and swap you have, run: free -h
On my machine, here is what it returns:
total used free shared buff/cache available
Mem: 7.7Gi 5.3Gi 928Mi 865Mi 1.5Gi 1.3Gi
Swap: 9.4Gi 7.1Gi 2.2Gi
You can clearly see that my machine is quite overloaded: about 8Gb of RAM, and 9Gb of swap, from which 7 are used.
When the RAM-hungry process got killed after Out of memory, I saw both RAM and swap being used at 100%.
So, allocating more swap may improve our problems.
Our redis servers are, since yesterday, gradually (200MB/hour) using more memory, while the amount of keys (330K) and their data (132MB redis-rdb-tools) stay about the same.
Output of redis-cli info shows 6.89G used memory?!
redis_version:2.4.10
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.6
process_id:3437
uptime_in_seconds:296453
uptime_in_days:3
lru_clock:1905188
used_cpu_sys:8605.03
used_cpu_user:1480.46
used_cpu_sys_children:1035.93
used_cpu_user_children:3504.93
connected_clients:404
connected_slaves:0
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:7400076728
used_memory_human:6.89G
used_memory_rss:7186984960
used_memory_peak:7427443856
used_memory_peak_human:6.92G
mem_fragmentation_ratio:0.97
mem_allocator:jemalloc-2.2.5
loading:0
aof_enabled:0
changes_since_last_save:1672
bgsave_in_progress:0
last_save_time:1403172198
bgrewriteaof_in_progress:0
total_connections_received:3616
total_commands_processed:127741023
expired_keys:0
evicted_keys:0
keyspace_hits:18817574
keyspace_misses:8285349
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:1619791
vm_enabled:0
role:slave
master_host:***BLOCKED***
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
db0:keys=372995,expires=372995
db6:keys=68399,expires=68399
The problem started when we updated our (.net) client code from BookSleeve 1.1.0.4 to ServiceStack v3.9.71 to prepare for an upgrade to Redis 2.8. But a lot of other stuff was updated to And our session state store (also redis, but with harbour client) does not show the same symptoms.
Where is all that Redis memory going? How can I troubleshoot it's usage?
Edit: I just restarted this instance and memory returned to 350M and is now climbing again. The top 10 largest objects are still the same size, ranging from 100K to 25M for nr 1. The amount of keys has dropped to 270K (330K earlier).
Here are some sources of "hidden" memory consumption in Redis:
Marc already mentioned the buffers maintained by the master to feed the slave. If a slave is lagging behind its master (because it runs on a slower box for instance), then some memory will be consumed on the master.
when long running commands are detected, Redis logs them in the SLOWLOG area, which takes some memory. You may want to use the SLOWLOG LEN command to check the number of records you have here.
communication buffers can also take memory. As far as I remember, with old versions of Redis (and 2.4 is quite old - you should really upgrade), it was unbounded, meaning that if you transfer a big object at a point, the communication buffer associated to this client connection will grow and never shrink. If there are many clients dealing occasionally with large objects, it could be a possible explanation. If you use commands retrieving very large data from Redis (in one shot), it can be an explanation as well. For instance, a simple KEYS * command applied on a Redis server storing millions of keys will consume a significant amount of memory.
You mentioned that you have objects as big as 25 MB. You have 404 client connections, if each of them needs to access such objects at a point in time, it will consume 10 GB of memory.