On my CentOS 6.5,
A couple of sites are running and I set logs directory separately.
loglotate works weekly but the results is not good. it archive already archived logs again every week.
here is /etc/logrotate.d/sites file.
/home/site1/logs/httpd/* /home/site2/logs/ httpd/* /home/site3/logs/httpd/* {
copytruncate
weekly
rotate 4
missingok
notifempty
dateext
}
and file list on /home/siate1/logs/httpd
-rw-r--r-- 1 root root 35100863 May 14 17:37 site1-access_log
...
-rw-r--r-- 1 root root 0 Apr 27 03:29 site1-access_log-20140413-20140420
-rw-r--r-- 1 root root 0 May 4 03:24 site1-access_log-20140413-20140420-20140427
-rw-r--r-- 1 root root 0 May 11 03:35 site1-access_log-20140413-20140420-20140427-20140504
-rw-r--r-- 1 root root 70221784 May 11 03:35 site1-access_log-20140413-20140420-20140427-20140504-20140511
-rw-r--r-- 1 root root 0 Apr 27 03:29 site1-access_log-20140420
-rw-r--r-- 1 root root 0 May 4 03:24 site1-access_log-20140420-20140427
-rw-r--r-- 1 root root 0 May 11 03:35 site1-access_log-20140420-20140427-20140504
-rw-r--r-- 1 root root 69335598 May 11 03:35 site1-access_log-20140420-20140427-20140504-20140511
-rw-r--r-- 1 root root 0 May 4 03:24 site1-access_log-20140427
-rw-r--r-- 1 root root 0 May 11 03:35 site1-access_log-20140427-20140504
-rw-r--r-- 1 root root 68505478 May 11 03:35 site1-access_log-20140427-20140504-20140511
-rw-r--r-- 1 root root 0 May 11 03:35 site1-access_log-20140504
-rw-r--r-- 1 root root 68772871 May 11 03:35 site1-access_log-20140504-20140511
-rw-r--r-- 1 root root 68054687 May 11 03:35 site1-access_log-20140511
-rw-r--r-- 1 root root 24701 May 14 16:54 site1-error_log
...
-rw-r--r-- 1 root root 0 Apr 27 03:29 site1-error_log-20140413-20140420
-rw-r--r-- 1 root root 0 May 4 03:24 site1-error_log-20140413-20140420-20140427
-rw-r--r-- 1 root root 0 May 11 03:35 site1-error_log-20140413-20140420-20140427-20140504
-rw-r--r-- 1 root root 109645452 May 11 03:35 site1-error_log-20140413-20140420-20140427-20140504-20140511
-rw-r--r-- 1 root root 0 Apr 27 03:29 site1-error_log-20140420
-rw-r--r-- 1 root root 0 May 4 03:24 site1-error_log-20140420-20140427
-rw-r--r-- 1 root root 0 May 11 03:35 site1-error_log-20140420-20140427-20140504
-rw-r--r-- 1 root root 43345 May 11 03:35 site1-error_log-20140420-20140427-20140504-20140511
-rw-r--r-- 1 root root 0 May 4 03:24 site1-error_log-20140427
-rw-r--r-- 1 root root 0 May 11 03:35 site1-error_log-20140427-20140504
-rw-r--r-- 1 root root 84371 May 11 03:35 site1-error_log-20140427-20140504-20140511
-rw-r--r-- 1 root root 0 May 11 03:35 site1-error_log-20140504
-rw-r--r-- 1 root root 38442 May 11 03:35 site1-error_log-20140504-20140511
-rw-r--r-- 1 root root 31182 May 11 03:35 site1-error_log-20140511
What is the problem?
Your match is too broad. It should not match rotated files, but only the current logfile for each rotation group. Try logs/httpd/*_log rather than logs/httpd/* (if all log files actually end in _log). Refer to man 7 glob for details.
Related
I have a folder with many jpg files, where each file name presents the creation time in the form of: "20220204T160311.746....jpg" ("..." represent additional data).
I'm trying to get all files created on the last X minutes. There are tons of results for such query if one asks Google, nevertheless, I'm getting weird results when trying the following:
find . -cmin -50 \( ! -iname "*.csv" ! -iname ".*" \) -ls | sort -n
the results do not start ok ("now" is 16:26):
14466561 2636 -rw-r--r-- 1 root root 2695349 Feb 4 16:03 ./20220204T160348.526-cam4-791-AREASbw-32358.jpg
14466569 2208 -rw-r--r-- 1 root root 2257356 Feb 4 16:03 ./20220204T160354.069-cam4-614.jpg
14466571 1044 -rw-r--r-- 1 root root 1068758 Feb 4 16:03 ./20220204T160354.069-cam4-614__chng_182.jpg
14466579 2208 -rw-r--r-- 1 root root 2258697 Feb 4 16:03 ./20220204T160359.296-cam4-352.jpg
14466581 1048 -rw-r--r-- 1 root root 1070111 Feb 4 16:03 ./20220204T160359.296-cam4-352__chng_175.jpg
14466589 2216 -rw-r--r-- 1 root root 2268675 Feb 4 16:04 ./20220204T160404.682-cam4-179.jpg
14466591 1048 -rw-r--r-- 1 root root 1070053 Feb 4 16:04 ./20220204T160404.682-cam4-179__chng_273.jpg
14466597 2204 -rw-r--r-- 1 root root 2253706 Feb 4 16:04 ./20220204T160409.957-cam4-571.jpg
14466599 1048 -rw-r--r-- 1 root root 1070335 Feb 4 16:04 ./20220204T160409.957-cam4-571__chng_154.jpg
14466600 1044 -rw-r--r-- 1 root root 1068675 Feb 4 16:04 ./20220204T160409.957-cam4-571__chng_309.jpg
14466601 2652 -rw-r--r-- 1 root root 2712638 Feb 4 16:04 ./20220204T160409.957-cam4-571-AREASbw-31696.jpg
14466610 2208 -rw-r--r-- 1 root root 2259408 Feb 4 16:04 ./20220204T160415.421-cam4-959.jpg
14466611 1048 -rw-r--r-- 1 root root 1070365 Feb 4 16:04 ./20220204T160415.421-cam4-959__chng_272.jpg
14466620 2212 -rw-r--r-- 1 root root 2264606 Feb 4 16:04 ./20220204T160420.666-cam4-742.jpg
14466621 1048 -rw-r--r-- 1 root root 1070276 Feb 4 16:04 ./20220204T160420.666-cam4-742__chng_173.jpg
and when I continue to scroll up, it arrives to a part where the files are wrongly sorted (like what supposed to be the start of the list):
14457644 2240 -rw-r--r-- 1 root root 2293209 Feb 4 16:26 ./20220204T162633.985-cam4-188.jpg
14457645 1056 -rw-r--r-- 1 root root 1080805 Feb 4 16:26 ./20220204T162633.985-cam4-188__chng_171.jpg
14457658 2236 -rw-r--r-- 1 root root 2286664 Feb 4 16:26 ./20220204T162639.299-cam4-801.jpg
14457659 1056 -rw-r--r-- 1 root root 1080042 Feb 4 16:26 ./20220204T162639.299-cam4-801__chng_158.jpg
14462048 2200 -rw-r--r-- 1 root root 2252735 Feb 4 15:49 ./20220204T154927.284-cam4-095.jpg
14462049 1044 -rw-r--r-- 1 root root 1065249 Feb 4 15:49 ./20220204T154927.284-cam4-095__chng_134.jpg
14462088 2204 -rw-r--r-- 1 root root 2255657 Feb 4 15:49 ./20220204T154932.490-cam4-571.jpg
14462089 1044 -rw-r--r-- 1 root root 1066449 Feb 4 15:49 ./20220204T154932.490-cam4-571__chng_228.jpg
14462118 2204 -rw-r--r-- 1 root root 2254481 Feb 4 15:49 ./20220204T154937.767-cam4-237.jpg
14462127 1044 -rw-r--r-- 1 root root 1066700 Feb 4 15:49 ./20220204T154937.767-cam4-237__chng_79.jpg
How come in the middle it goes back from 15:49 to 16:26?
It happens on every query I'm running, unless it is on a short term like 10 min.
By default sort uses the first column. Use -k to specify column number to sort by.
find . -cmin -50 \( ! -iname "*.csv" ! -iname ".*" \) -ls | sort -n -k 7
MWE
I have a GitHub Actions workflow that recently stopped working without me making changes. The error:
==> ERROR: /etc/makepkg.conf not found.
Aborting...
This is from running # sudo -Eu builder makepkg --printsrcinfo.
Source
Logs
The order of the logs seem to be wrong but it is correct in an earlier log (possibly due to ls -l having a large output).
The source of this error seems to be libmakepkg/util/config.sh.in:
# Source the config file; fail if it is not found
if [[ -r $MAKEPKG_CONF ]]; then
source_safe "$MAKEPKG_CONF"
else
error "$(gettext "%s not found.")" "$MAKEPKG_CONF"
plainerr "$(gettext "Aborting...")"
exit $E_CONFIG_ERROR
fi
I added the following to my entrypoint script:
echo "Writing SRCINFO..."
# Debug
echo "---"
ls -l /
echo "---"
ls -l /etc
echo "---"
sudo -Eu builder cat /etc/makepkg.conf
echo "---"
sudo -Eu builder /bin/bash -c "[[ -r "/etc/makepkg.conf" ]] && echo 1 || echo 0"
echo "---"
sudo -Eu builder makepkg --printsrcinfo > .SRCINFO
The builder user is created in build.sh:
useradd builder -m
echo "builder ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
I got:
Setting permissions...
Writing SRCINFO...
---
total 52
lrwxrwxrwx 1 root root 7 Feb 1 19:19 bin -> usr/bin
drwxr-xr-x 2 root root 4096 Jan 19 01:32 boot
drwxr-xr-x 5 root root 340 Feb 7 11:58 dev
-rwxr-xr-x 1 root root 836 Feb 7 11:57 entrypoint.sh
drwxr-xr-x 1 root root 4096 Feb 7 11:58 etc
drwxr-xr-x 6 root root 4096 Feb 7 11:58 github
drwxr-xr-x 1 root root 4096 Feb 7 11:57 home
lrwxrwxrwx 1 root root 7 Feb 1 19:19 lib -> usr/lib
lrwxrwxrwx 1 root root 7 Feb 1 19:19 lib64 -> usr/lib
drwxr-xr-x 2 root root 4096 Jan 19 01:32 mnt
drwxr-xr-x 2 root root 4096 Jan 19 01:32 opt
dr-xr-xr-x 159 root root 0 Feb 7 11:58 proc
drwxr-x--- 2 root root 4096 Jan 19 01:32 root
drwxr-xr-x 1 root root 4096 Feb 7 11:58 run
lrwxrwxrwx 1 root root 7 Feb 1 19:19 sbin -> usr/bin
drwxr-xr-x 4 root root 4096 Feb 1 19:19 srv
dr-xr-xr-x 12 root root 0 Feb 7 11:58 sys
drwxrwxrwt 2 root root 4096 Jan 19 01:32 tmp
drwxr-xr-x 1 root root 4096 Feb 7 11:57 usr
drwxr-xr-x 1 root root 4096 Feb 1 19:19 var
---
total 640
-rw-r--r-- 1 root root 0 Jan 19 01:32 arch-release
drwxr-xr-x 3 root root 4096 Feb 1 19:19 audit
-rw-r--r-- 1 root root 28 Dec 20 18:44 bash.bash_logout
-rw-r--r-- 1 root root 618 Dec 20 18:44 bash.bashrc
-rw-r--r-- 1 root root 447 Dec 2 16:02 bindresvport.blacklist
drwxr-xr-x 2 root root 4096 Dec 16 14:38 binfmt.d
drwxr-xr-x 4 root root 4096 Feb 1 19:19 ca-certificates
-rw------- 1 root root 722 Jan 19 01:32 crypttab
drwxr-xr-x 2 root root 4096 Feb 1 19:19 default
drwxr-xr-x 2 root root 4096 Jan 7 19:51 depmod.d
-rw-r--r-- 1 root root 685 Jan 31 20:31 e2scrub.conf
-rw-r--r-- 1 root root 97 Jan 13 22:50 environment
-rw-r--r-- 1 root root 1362 Jan 20 21:31 ethertypes
-rw-r--r-- 1 root root 126 Jan 19 01:32 fstab
-rw-r--r-- 1 root root 2584 Feb 6 00:09 gai.conf
-rw-r--r-- 1 root root 626 Feb 7 11:57 group
-rw-r--r-- 1 root root 610 Jan 31 00:20 group-
-rw------- 1 root root 558 Feb 7 11:57 gshadow
-rw------- 1 root root 546 Jan 31 00:20 gshadow-
-rw-r--r-- 1 root root 73 Jan 19 01:32 host.conf
-rw-r--r-- 1 root root 13 Feb 7 11:58 hostname
-rw-r--r-- 1 root root 174 Feb 7 11:58 hosts
-rw-r--r-- 1 root root 714 Dec 8 17:48 inputrc
drwxr-xr-x 2 root root 4096 Feb 1 19:19 iproute2
drwxr-xr-x 2 root root 4096 Feb 1 19:19 iptables
-rw-r--r-- 1 root root 20 Jan 19 01:32 issue
drwxr-xr-x 3 root root 4096 Feb 1 19:19 kernel
drwxr-xr-x 2 root root 4096 Jul 7 2020 keyutils
-rw-r--r-- 1 root root 369 Jan 14 00:32 krb5.conf
-rw-r--r-- 1 root root 18096 Feb 7 11:57 ld.so.cache
-rw-r--r-- 1 root root 117 Jan 19 01:32 ld.so.conf
drwxr-xr-x 1 root root 4096 Feb 7 11:57 ld.so.conf.d
-rw-r----- 1 root root 191 Jan 13 22:33 libaudit.conf
drwxr-xr-x 2 root root 4096 Feb 1 19:19 libnl
-rw-r--r-- 1 root root 17 Jan 31 00:19 locale.conf
-rw-r--r-- 1 root root 18 Jan 31 00:19 locale.gen
-rw-r--r-- 1 root root 9984 Feb 6 00:09 locale.gen.pacnew
-rw-r--r-- 1 root root 5645 Sep 7 13:42 login.defs
-rw-r--r-- 1 root root 5792 Jul 1 2020 makepkg.conf
-rw-r--r-- 1 root root 812 Jan 31 20:31 mke2fs.conf
drwxr-xr-x 2 root root 4096 Jan 7 19:51 modprobe.d
drwxr-xr-x 2 root root 4096 Dec 16 14:38 modules-load.d
-rw-r--r-- 1 root root 0 Jan 19 01:32 motd
lrwxrwxrwx 1 root root 12 Feb 7 11:58 mtab -> /proc/mounts
-rw-r--r-- 1 root root 767 Dec 2 16:02 netconfig
-rw-r--r-- 1 root root 2717 Feb 6 00:09 nscd.conf
-rw-r--r-- 1 root root 328 Jan 19 01:32 nsswitch.conf
drwxr-xr-x 1 root root 4096 Feb 7 11:57 openldap
lrwxrwxrwx 1 root root 19 Feb 1 19:19 os-release -> /usr/lib/os-release
-rw-r--r-- 1 root root 3264 Feb 7 11:57 pacman.conf
-rw-r--r-- 1 root root 2883 Jul 1 2020 pacman.conf.pacnew
drwxr-xr-x 1 root root 4096 Feb 7 11:57 pacman.d
drwxr-xr-x 1 root root 4096 Feb 7 11:57 pam.d
-rw-r--r-- 1 root root 744 Feb 7 11:57 passwd
-rw-r--r-- 1 root root 699 Jan 31 00:20 passwd-
drwxr-xr-x 2 root root 4096 Feb 1 19:19 pkcs11
-rw-r--r-- 1 root root 1020 Jan 19 01:32 profile
drwxr-xr-x 1 root root 4096 Feb 7 11:57 profile.d
-rw-r--r-- 1 root root 3171 Jan 3 17:14 protocols
-rw-r--r-- 1 root root 1814 Jul 7 2020 request-key.conf
drwxr-xr-x 2 root root 4096 Jul 7 2020 request-key.d
-rw-r--r-- 1 root root 649 Feb 7 11:58 resolv.conf
-rw-r--r-- 1 root root 1634 Feb 6 00:09 rpc
-rw-r--r-- 1 root root 139 Jan 19 01:32 securetty
drwxr-xr-x 2 root root 4096 Feb 1 19:19 security
-rw-r--r-- 1 root root 297708 Jan 3 17:14 services
-rw------- 1 root root 375 Feb 7 11:57 shadow
-rw------- 1 root root 346 Jan 31 00:20 shadow-
-rw-r--r-- 1 root root 83 Jan 19 01:32 shells
drwxr-xr-x 2 root root 4096 Feb 1 19:19 skel
drwxr-xr-x 5 root root 4096 Feb 1 19:19 ssl
-rw-r--r-- 1 root root 3975 Jan 26 18:34 sudo.conf
-r--r----- 1 root root 3160 Feb 7 11:57 sudoers
drwxr-x--- 2 root root 4096 Jan 26 18:34 sudoers.d
-rw-r--r-- 1 root root 6169 Jan 26 18:34 sudo_logsrvd.conf
drwxr-xr-x 2 root root 4096 Dec 16 14:38 sysctl.d
drwxr-xr-x 1 root root 4096 Feb 7 11:57 systemd
drwxr-xr-x 2 root root 4096 Dec 16 14:38 tmpfiles.d
drwxr-xr-x 1 root root 4096 Feb 7 11:57 udev
drwxr-xr-x 1 root root 4096 Feb 1 19:19 X11
-rw-r--r-- 1 root root 642 May 7 2020 xattr.conf
drwxr-xr-x 1 root root 4096 Feb 1 19:19 xdg
drwxr-xr-x 2 root root 4096 Feb 1 19:19 xinetd.d
---
#!/hint/bash
#
# /etc/makepkg.conf
#
...
SRCEXT='.src.tar.gz'
---
0
---
==> ERROR: /etc/makepkg.conf not found.
Aborting...
(The contents of /etc/makepkg.conf was concatenated.)
I also accidentally did [[ -r "/etc/makepkg.conf" ]] && echo 1 || echo 0 (as root) and I also got 0.
How is it possible that a file is not readable yet I can cat it? I also tried running the exact same commands in a local container and couldn't reproduce this issue, but this has happened every GitHub Actions run since it started.
This makes me think the Actions setup is causing an issue, but nothing seems odd there:
/usr/bin/docker build -t 442333:35a065f0b9f356b32f2852ba2f6b7296 -f "/home/runner/work/visual-studio-code-insiders-arch/visual-studio-code-insiders-arch/./.github/actions/pkg/Dockerfile" "/home/runner/work/visual-studio-code-insiders-arch/visual-studio-code-insiders-arch/.github/actions/pkg"
/usr/bin/docker run --name a065f0b9f356b32f2852ba2f6b7296_baf94b --label 442333 --workdir /github/workspace --rm -e pythonLocation -e LD_LIBRARY_PATH -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RETENTION_DAYS -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e GITHUB_ACTION_REPOSITORY -e GITHUB_ACTION_REF -e GITHUB_PATH -e GITHUB_ENV -e RUNNER_OS -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/visual-studio-code-insiders-arch/visual-studio-code-insiders-arch":"/github/workspace" 442333:35a065f0b9f356b32f2852ba2f6b7296
Thanks for sharing this. I have precisely the same problems.
I added the following in my Dockerfile
RUN sed -i 's/\tif \[\[ -r $MAKEPKG_CONF \]\]; then/\tif \[\[ -f $MAKEPKG_CONF \]\]; then/' /usr/share/makepkg/util/config.sh
which replaces the check for read permission (-r) for the check whether the file exists and is a regular file (-f).
now my github action gets over this particular location, but fails with the next check:
==> ERROR: You do not have write permission for the directory $BUILDDIR (/tmp/aurutils).
Aborting...
This is not a solution and does not answer what's the underlying issue, but I hope it helps anyways.
This issue is caused by using glibc >= 2.33 on the container and an outdated version of the Docker engine on the host.
You can fix it by patching glibc in your container:
patched_glibc=glibc-linux4-2.33-4-x86_64.pkg.tar.zst
curl -LO https://repo.archlinuxcn.org/x86_64/$patched_glibc
bsdtar -C / -xvf $patched_glibc
Thanks to lxqt-panel for the workaround.
# ls -la /var/lib/my-app/my-app*
-rw-rw---- 1 root root 1073742635 May 4 21:01 /var/lib/my-app/my-app.001211
-rw-rw---- 1 root root 1073741971 May 5 20:07 /var/lib/my-app/my-app.001212
-rw-rw---- 1 root root 1073741874 May 6 21:01 /var/lib/my-app/my-app.001213
-rw-rw---- 1 root root 1134106954 May 8 06:50 /var/lib/my-app/my-app.001214
-rw-rw---- 1 root root 1073743025 May 9 10:57 /var/lib/my-app/my-app.001215
-rw-rw---- 1 root root 1073742068 May 10 15:19 /var/lib/my-app/my-app.001216
-rw-rw---- 1 root root 1073741886 May 11 19:26 /var/lib/my-app/my-app.001217
-rw-rw---- 1 root root 1073742868 May 12 21:15 /var/lib/my-app/my-app.001218
-rw-rw---- 1 root root 1073742220 May 13 23:36 /var/lib/my-app/my-app.001219
-rw-rw---- 1 root root 1073742065 May 15 07:21 /var/lib/my-app/my-app.001220
-rw-rw---- 1 root root 1073742428 May 16 12:18 /var/lib/my-app/my-app.001221
-rw-rw---- 1 root root 1073741955 May 17 15:04 /var/lib/my-app/my-app.001222
-rw-rw---- 1 root root 1073742263 May 18 17:21 /var/lib/my-app/my-app.001223
-rw-rw---- 1 root root 1073742288 May 19 19:46 /var/lib/my-app/my-app.001224
-rw-rw---- 1 root root 1073741896 May 20 22:45 /var/lib/my-app/my-app.001225
-rw-rw---- 2 root root 1073742131 May 22 10:37 /var/lib/my-app/my-app.001226
-rw-rw---- 1 root root 967375006 May 23 13:37 /var/lib/my-app/my-app.001227
Given this as my folder content, what I want is to list all files after the file
my-app.001219
Output would look like:
-rw-rw---- 1 root root 1073742220 May 13 23:36 /var/lib/my-app/my-app.001219
-rw-rw---- 1 root root 1073742065 May 15 07:21 /var/lib/my-app/my-app.001220
-rw-rw---- 1 root root 1073742428 May 16 12:18 /var/lib/my-app/my-app.001221
-rw-rw---- 1 root root 1073741955 May 17 15:04 /var/lib/my-app/my-app.001222
-rw-rw---- 1 root root 1073742263 May 18 17:21 /var/lib/my-app/my-app.001223
-rw-rw---- 1 root root 1073742288 May 19 19:46 /var/lib/my-app/my-app.001224
-rw-rw---- 1 root root 1073741896 May 20 22:45 /var/lib/my-app/my-app.001225
-rw-rw---- 2 root root 1073742131 May 22 10:37 /var/lib/my-app/my-app.001226
-rw-rw---- 1 root root 967375006 May 23 13:37 /var/lib/my-app/my-app.001227
NB: given filename will be ever increasing natural numbers.
Edit: Can I also include the file as well(index 001219 in this case)
You can use sed to delete lines in the given range:
ls -lat /var/lib/new_folder/my-app* | sed '1,/my-app\.001219/d'
To print also the matching line, just invert the condition, i.e. tell sed to *print from the matching line to the end:
sed -n '/my-app\.001219/,$p'
I'm using Ubuntu with kernel 3.2.1, x86_64. I'm trying to benchmark a file system, and I want to limit the page cache size to avoid the file system cache taking up too much RAM, which would obviously improve performance (but would not reflect the results on systems with less memory).
Is there a way to do that? I've seen an option in some RHEL distribution for tuning /proc/sys/vm/pagecaches which seems to satisfy this, but I dont see anything useful in Ubuntu except dirty_background_ratio, which will only trigger flushing to disk, not more caching (so I can get a lot more sync I/O etc)
Thank you
ubuntu does not seem to have vm.pagecache settings
ls -l /proc/sys/vm/
total 0
-rw-r--r-- 1 root root 0 Jun 17 14:13 block_dump
--w------- 1 root root 0 Jun 17 14:13 compact_memory
-rw-r--r-- 1 root root 0 Jun 17 14:13 dirty_background_bytes
-rw-r--r-- 1 root root 0 Jun 17 09:16 dirty_background_ratio
-rw-r--r-- 1 root root 0 Jun 17 14:13 dirty_bytes
-rw-r--r-- 1 root root 0 Jun 17 14:13 dirty_expire_centisecs
-rw-r--r-- 1 root root 0 Jun 17 09:16 dirty_ratio
-rw-r--r-- 1 root root 0 Jun 17 09:16 dirty_writeback_centisecs
-rw-r--r-- 1 root root 0 Jun 17 14:13 drop_caches
-rw-r--r-- 1 root root 0 Jun 17 14:13 extfrag_threshold
-rw-r--r-- 1 root root 0 Jun 17 14:13 hugepages_treat_as_movable
-rw-r--r-- 1 root root 0 Jun 17 14:13 hugetlb_shm_group
-rw-r--r-- 1 root root 0 Jun 17 09:16 laptop_mode
-rw-r--r-- 1 root root 0 Jun 17 14:13 legacy_va_layout
-rw-r--r-- 1 root root 0 Jun 17 14:13 lowmem_reserve_ratio
-rw-r--r-- 1 root root 0 Jun 17 14:13 max_map_count
-rw-r--r-- 1 root root 0 Jun 17 14:13 memory_failure_early_kill
-rw-r--r-- 1 root root 0 Jun 17 14:13 memory_failure_recovery
-rw-r--r-- 1 root root 0 Jun 17 14:13 min_free_kbytes
-rw-r--r-- 1 root root 0 Jun 17 14:13 min_slab_ratio
-rw-r--r-- 1 root root 0 Jun 17 14:13 min_unmapped_ratio
-rw-r--r-- 1 root root 0 Jun 17 09:15 mmap_min_addr
-rw-r--r-- 1 root root 0 Jun 17 14:13 nr_hugepages
-rw-r--r-- 1 root root 0 Jun 17 14:13 nr_hugepages_mempolicy
-rw-r--r-- 1 root root 0 Jun 17 14:13 nr_overcommit_hugepages
-r--r--r-- 1 root root 0 Jun 17 14:13 nr_pdflush_threads
-rw-r--r-- 1 root root 0 Jun 17 14:13 numa_zonelist_order
-rw-r--r-- 1 root root 0 Jun 17 14:13 oom_dump_tasks
-rw-r--r-- 1 root root 0 Jun 17 14:13 oom_kill_allocating_task
-rw-r--r-- 1 root root 0 Jun 17 09:15 overcommit_memory
-rw-r--r-- 1 root root 0 Jun 17 14:13 overcommit_ratio
-rw-r--r-- 1 root root 0 Jun 17 14:13 page-cluster
-rw-r--r-- 1 root root 0 Jun 17 14:13 panic_on_oom
-rw-r--r-- 1 root root 0 Jun 17 14:13 percpu_pagelist_fraction
-rw-r--r-- 1 root root 0 Jun 17 14:13 scan_unevictable_pages
-rw-r--r-- 1 root root 0 Jun 17 14:13 stat_interval
-rw-r--r-- 1 root root 0 Jun 17 14:13 swappiness
-rw-r--r-- 1 root root 0 Jun 17 14:13 vfs_cache_pressure
-rw-r--r-- 1 root root 0 Jun 17 14:13 zone_reclaim_mode
you could try the following:
vi /etc/sysctl.conf
vm.min_free_kbytes=1024
vm.swappiness = 100
then run
sysctl -p
vm.min_free_kbytes = 1024
vm.swappiness = 100
Unsure if it is of any help.
The swapiness 100 Swap more application data to disk when ram is exhausted
Set the parameter echo 3019053 > /proc/sys/vm/min_free_kbytes , the current value represents 2.8-GB. then check the value by giving the command line : sysctl -a | grep min_free_kbytes.
now to answer this question, if you want to avoid too much of cache to be accumulated set this value to about 70-90% of the Total RAM.
now to avoid swap to be accumulated, turn the swap off by giving the command as : swapoff -a and commenting the line for swap under /etc/fstab to turn it off permanently after reboots. this will make things happen as the questioner wants.
There is a process which is running under root user.
ps aux | grep ProcessX
root 11565 0.0 0.7 82120 22976 ? Ssl 14:57 0:02 ProcessX
Now ls -l /proc/11565/ (pid ) gives this result.
total 0
dr-xr-xr-x 2 root root 0 Aug 9 16:06 attr
-rw-r--r-- 1 root root 0 Aug 9 16:06 autogroup
-r-------- 1 root root 0 Aug 9 16:06 auxv
-r--r--r-- 1 root root 0 Aug 9 16:06 cgroup
--w------- 1 root root 0 Aug 9 16:06 clear_refs
-r--r--r-- 1 root root 0 Aug 9 16:06 cmdline
-rw-r--r-- 1 root root 0 Aug 9 16:06 coredump_filter
-r--r--r-- 1 root root 0 Aug 9 16:06 cpuset
lrwxrwxrwx 1 root root 0 Aug 9 16:06 cwd -> /usr/local/bin
-r-------- 1 root root 0 Aug 9 16:06 environ
lrwxrwxrwx 1 root root 0 Aug 9 16:06 exe -> /usr/local/bin/ProcessX
dr-x------ 2 root root 0 Aug 9 16:06 fd
dr-x------ 2 root root 0 Aug 9 16:06 fdinfo
-r-------- 1 root root 0 Aug 9 16:06 io
-rw------- 1 root root 0 Aug 9 16:06 limits
-rw-r--r-- 1 root root 0 Aug 9 16:06 loginuid
-r--r--r-- 1 root root 0 Aug 9 16:06 maps
-rw------- 1 root root 0 Aug 9 16:06 mem
-r--r--r-- 1 root root 0 Aug 9 16:06 mountinfo
-r--r--r-- 1 root root 0 Aug 9 16:06 mounts
-r-------- 1 root root 0 Aug 9 16:06 mountstats
dr-xr-xr-x 6 root root 0 Aug 9 16:06 net
-r--r--r-- 1 root root 0 Aug 9 16:06 numa_maps
-rw-r--r-- 1 root root 0 Aug 9 16:06 oom_adj
-r--r--r-- 1 root root 0 Aug 9 16:06 oom_score
-rw-r--r-- 1 root root 0 Aug 9 16:06 oom_score_adj
-r--r--r-- 1 root root 0 Aug 9 16:06 pagemap
-r--r--r-- 1 root root 0 Aug 9 16:06 personality
lrwxrwxrwx 1 root root 0 Aug 9 16:06 root -> /
-rw-r--r-- 1 root root 0 Aug 9 16:06 sched
-r--r--r-- 1 root root 0 Aug 9 16:06 schedstat
-r--r--r-- 1 root root 0 Aug 9 16:06 sessionid
-r--r--r-- 1 root root 0 Aug 9 16:06 smaps
-r--r--r-- 1 root root 0 Aug 9 16:06 stack
-r--r--r-- 1 root root 0 Aug 9 16:06 stat
-r--r--r-- 1 root root 0 Aug 9 16:06 statm
-r--r--r-- 1 root root 0 Aug 9 16:06 status
-r--r--r-- 1 root root 0 Aug 9 16:06 syscall
dr-xr-xr-x 6 root root 0 Aug 9 16:06 task
-r--r--r-- 1 root root 0 Aug 9 16:06 wchan
Now the file permission for both status and maps are same (-r--r--r--). But when I issue cat /proc/11565/maps with a non privileged (not root) user, it gives me a permission denied. But for cat /proc/11565/status, it outputs as expected.
Is there something I am missing here?
It's because the file permissions are not the only protection you're encountering.
Those aren't just regular text files on a file system, procfs is a window into process internals and you have to get past both the file permissions plus whatever other protections are in place.
The maps show potentially dangerous information about memory usage and where executable code is located within the process space. If you look into ASLR, you'll see this was a method of preventing potential attackers from knowing where code was loaded and it wouldn't make sense to reveal it in a world-readable entry in procfs.
This protection was added way back in 2007:
This change implements a check using "ptrace_may_attach" before allowing access to read the maps contents. To control this protection, the new knob /proc/sys/kernel/maps_protect has been added, with corresponding updates to the procfs documentation.
Within ptrace_may_attach() (actually within one of the functions it calls) lies the following code:
if (((current->uid != task->euid) ||
(current->uid != task->suid) ||
(current->uid != task->uid) ||
(current->gid != task->egid) ||
(current->gid != task->sgid) ||
(current->gid != task->gid)) && !capable(CAP_SYS_PTRACE))
return -EPERM;
so that, unless you have the same real user/group ID, saved user/group ID and effective user/group ID (i.e., no sneaky setuid stuff) and they're the same as the user/group ID that owns the process, you're not allowed to see inside that "file" (unless your process has the CAP_SYS_PTRACE capability of course).
The process uid must match the smaps uid, and the process gid must match the smaps gid.
$ ls -l /proc/15889/smaps /proc/16139/smaps
-r--r--r--. 1 oracle dba 0 Feb 10 16:42 /proc/15889/smaps
-r--r--r--. 1 oracle asmadmin 0 Feb 10 16:42 /proc/16139/smaps
$ wc /proc/15889/smaps /proc/16139/smaps
6851 23498 224275 /proc/15889/smaps
wc: /proc/16139/smaps: Permission denied
6851 23498 224275 total
$ id
uid=400(oracle) gid=400(dba) groups=400(dba),522(asmadmin),etc.
Same for environ, io, and all memory maps.