I am using a Windows Server 2019 with SMBServer Shares which get mounted on a SLES via CIFS.
When Windows does its periodic system cleanup (closing idle SMBServer sessions) the Linux server reconnects with following Kernel message:
CIFS: VFS: \\filer.example.com has not responded in 180 seconds. Reconnecting...
The reconnect seems to be successful, as reading and writing files to the mount is possible.
But querying disk stats is not possible anymore.
Bad file descriptor on df:
user#suse:~$ df -h
df: /mnt/test: Bad file descriptor
Filesystem Size Used Avail Use% Mounted on
...
Wrong data on stat:
user#suse:~$ stat /mnt/test
File: /mnt/test
Size: 0 Blocks: 0 IO Block: 1048576 directory
Device: 38h/56d Inode: 281474976710700 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 1100/ application-user) Gid: ( 80/ application-group)
Access: 2023-02-15 11:04:55.977807600 +0100
Modify: 2023-02-15 11:04:55.977807600 +0100
Change: 2023-02-16 09:50:07.638662000 +0100
Birth: 2023-02-10 14:07:14.408836200 +0100
I noticed the same problem when mounting subdirectories of a single share multiple times. The SLES does mount multiple shares, but each share only once.
Related
Setup:
NFS server: NFSServerHOST
NFS Share: NFSServerHOST:/MYSHARE
NFS Client1: CLNT1
NFS Client2: CKNT2
NFS Client3: CLNT3
Client OS: RHEL 7 and 8
Client users: User1, User2
Local Mount on Client: /var/NFSSHARE
mount -t nfs4 NFSServerHOST:/MYSHARE /var/NFSSHARE
Mount created successfully on all clients. Both User1 and User2 can read/write on /var/NFSSHARE from all 3 Clients.
Now Something happens on Client2 (we're yet to find out if it's related to server patch or some cron job) User1 cannot read/write to /var/NFSSHARE only on Client1. User2 can still read/write to NFSSHare on CLient1. Both users can still read/write on Client2 and client3.
Error while performing read/write on Client1 for User1: Remote I/O error
If we reboot client1 the issue is gone and User1 can again perform I/O operatrion on NFSSahre from Client1.
Some of the things we checked:
No version mismatch: Both NFS client and NFS Server config is for NFS V4
Nothing wrong with whitelisting: ALl 3 Client IPs are whitelisted on NFSServer
Have checked the inodes and lsof usage which is well within the limit.
nfs4_getafacl /var/NFSSHARE
# file: /var/NFSSHARE
A::EVERYONE#:rwaDxtTnNcy
running getfacl /var/NFSSHARE with User1 as logged in User on CLient1
# file: var/MQHA/
# owner: nobody
# group: nobody
user::rwx
group::rwx
other::rwx
comparing rpcdebug log while performing I/O operation on Client1 (FAILURE) vs Client2 (SUCCESS)
kernel: NFS: nfs_update_inode(0:57/3963604504 fh_crc=0xbf9e74c8 ct=2 info=0x427e7f)
kernel: NFS: (0:57/3963604504) revalidation complete
kernel: NFS: permission(0:57/3963604504), mask=0x1, res=0
kernel: NFS: permission(0:57/3963604504), mask=0x3, res=0
kernel: NFS: atomic_open(0:57/3963604504), Abhi
kernel: --> nfs_put_client({2})
kernel: --> nfs4_alloc_slot used_slots=0002 highest_used=1 max_slots=1024
kernel: <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=0
Logs chane after this point. Before both SUCCESS and FAILURE are more or less same just the numeric values are different.
Client1 (FAILURE)
kernel: nfs4_free_slot: slotid 0 highest_used_slotid 1
kernel: NFS: permission(0:57/3963604504), mask=0x81, res=-10
kernel: --> nfs4_alloc_slot used_slots=0002 highest_used=1 max_slots=1024
kernel: <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=0
kernel: decode_attr_type: type=00
Client2 (SUCCESS)
kernel: decode_attr_type: type=0100000
kernel: decode_attr_change: change attribute=7148460619683717735
kernel: decode_attr_size: file size=0
Looking for suggestions to diagnose this issue. What more can we do to enable more verbose logging either on Client or server side to know more about the error ?
Thanks
I am running Oracle 11g on Linux server and one the below Database issues occurred suddenly (every 2 weeks or 3 weeks sometimes):
Some times:
ORA-01034: ORACLE not available ORA-27102: out of memory Linux-x86_64 Error: 12: Cannot allocate memory Additional information: 1 Additional information: 163844 Additional information: 8
And last time:
ORA-01034: ORACLE not available ORA-27101: shared memory realm does not exist Linux-x86_64 Error: 2: No such file or directory
When I tried to startup database after setting up SID but I had the below error:
SQL> startup
ORA-00845: MEMORY_TARGET not supported on this system
I rebooted the server then everything is OK
My page size: 4096
kernel.shmall = 4294967296
How can I prevent these issues from happening again? should I update anything in Oracle memory settings?
Make sure your /dev/shm allocation is greater than what you have set for MEMORY_MAX_TARGET
Example fix for a memory allocation of 4Gb:
mount -o remount,size=4096m /dev/shm
Entry for /etc/fstab file to make the change permanent
tmpfs /dev/shm tmpfs size=4096m 0 0
Also see Oracle support: Doc ID 1399209.1 - ORA-00845 - Which value for /dev/shm is needed to startup database without ORA-00845
See, this is what worked for me. My ORACLE_SID, ORACLE_HOME etc., were just fine.
Restart the listener - lsnrctl start
sqlplus /nolog
connect /as sysdba
startup
I have properly connected an iSCSI target to my FreeBSD host using iscsictl. This new device shows up as da7. The disk shows up with:
geom disk list
as
Geom name: da7
Providers:
1. Name: da7
Mediasize: 4294967296000 (3.9T)
Sectorsize: 512
Stripesize: 8192
Stripeoffset: 0
Mode: r0w0e0
descr: SYNOLOGY iSCSI Storage
lunname: SYNOLOGYiSCSI Storage:44281bed-ce3d-4a9f-b95e-c89b6c74c345
lunid: 600140544281beddce3dd4a9fdb95edc
ident: 44281bed-ce3d-4a9f-b95e-c89b6c74c345
rotationrate: unknown
fwsectors: 63
fwheads: 255
I wanted to create a new ZFS zpool on this single disk with the command:
zpool create backuppool /dev/da7
The zpool command will now utilise a lot of cpu, but newer finishs. (Let it run for 2h).
If I create an ufs filesystem on the properly partitioned disk, the process is extremly fast. Also if I create a pool on a different raw disks, zpool finishs within seconds.
After some research I could not find any information if creating a zpool on a iSCSI target is allowed or not. Does anyone get this working?
Tested on: FreeBSD 11.1-RELEASE-p4 #0: Tue Nov 14 06:12:40 UTC 2017
What I want to achieve
I want to connect to an oracle database within an oracle container via sqlplus
What my problem is
I can connect to the database of the same image within Docker for Windows but not on a linux host
What I did
Windows
c:\>docker pull wnameless/oracle-xe-11g
c:\>docker run -d -p 49160:22 -p 49161:1521 wnameless/oracle-xe-11g
c:\>docker exec -it f79f7aa5222c /bin/bash
root#f79f7aa5222c:/# sqlplus system/oracle
Result:
SQL*Plus: Release 11.2.0.2.0 Production on Wed Jul 19 07:28:45 2017
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production
Linux
[xx#yy Docker]$ sudo docker pull wnameless/oracle-xe-11g
[xx#yy Docker]$ sudo docker run --shm-size=2g -d -p 49160:22 -p 49161:1521 wnameless/oracle-xe-11g
[xx#yy Docker]$ sudo docker exec -it cea28583cb1c /bin/bash
root#cea28583cb1c:/# sqlplus system/oracle
Result:
SQL*Plus: Release 11.2.0.2.0 Production on Wed Jul 19 07:34:16 2017
Copyright (c) 1982, 2011, Oracle. All rights reserved.
ERROR:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
Infos
Linux system:
docker info
[xx#yy Docker]$ sudo docker info
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 33
Server Version: 1.13.1
Storage Driver: devicemapper
Pool Name: docker-8:17-4460587-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 4.741 GB
Data Space Total: 107.4 GB
Data Space Available: 23.21 GB
Metadata Space Used: 4.289 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.143 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use --storage-opt dm.thinpooldev to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.137 (2016-11-30)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: oci runc
Default Runtime: oci
Init Binary: /usr/libexec/docker/docker-init-current
containerd version: (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: N/A (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
seccomp
WARNING: You're not using the default seccomp profile
Profile: /etc/docker/seccomp.json
selinux
Kernel Version: 4.11.9-300.fc26.x86_64
Operating System: Fedora 26 (Workstation Edition)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 8
Total Memory: 7.776 GiB
Name: yy.domain.de
ID: 4YVE:4FQ7:27SF:5JZJ:H6RA:P4H5:TK6O:FAGY:CKHS:RQGQ:3BDS:L7W7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://registry.fedoraproject.org/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Registries: registry.fedoraproject.org (secure), registry.access.redhat.com (secure), docker.io (secure)
Docker logs
[xx#yy Docker]$ sudo docker logs cea28583cb1c
Starting Oracle Net Listener.
Starting Oracle Database 11g Express Edition instance.
/usr/sbin/startup.sh: ignoring /docker-entrypoint-initdb.d/*
df
root#cea28583cb1c:/# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/docker-8:17-4460587-f479be876efbf45834ea013ac119d63629944a2b315c7cbe7da3a22c1b1e566e 10G 1.9G 8.2G 19% /
tmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sdb1 110G 89G 17G 85% /etc/hosts
shm 2.0G 0 2.0G 0% /dev/shm
tmpfs 3.9G 0 3.9G 0% /sys/firmware
stat
root#cea28583cb1c:/# stat /
File: '/'
Size: 253 Blocks: 0 IO Block: 4096 directory
Device: fd01h/64769d Inode: 131 Links: 22
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2017-07-11 07:12:36.676904944 +0000
Modify: 2017-07-19 07:33:44.026249772 +0000
Change: 2017-07-19 07:33:44.216248030 +0000
Birth: -
lsnrctrl
xx#yy:/etc/selinux# lsnrctl status
LSNRCTL for Linux: Version 11.2.0.2.0 - Production on 19-JUL-2017 11:30:37
Copyright (c) 1991, 2011, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC_FOR_XE)))
STATUS of the LISTENER
Alias LISTENER
Version TNSLSNR for Linux: Version 11.2.0.2.0 - Production
Start Date 19-JUL-2017 07:33:46
Uptime 0 days 3 hr. 56 min. 51 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Default Service XE
Listener Parameter File /u01/app/oracle/product/11.2.0/xe/network/admin/listener.ora
Listener Log File /u01/app/oracle/diag/tnslsnr/cea28583cb1c/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC_FOR_XE)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cea28583cb1c)(PORT=1521)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
What I already tried
I tried both users, oracle and root.
I changed permissions of /u01/app/oracle to 755
I checked selinux, this seems not to be active/installed. At least thats what the docker info tells and I cant find sestatus command and /etc/selinux only contains semanage.conf.
I tried several different oracle images, same result in every container.
EDIT
As requested, I tried starting oracle service myself:
service oracle-xe start
Result of alert_XE.log:
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST
Autotune of undo retention is turned on.
IMODE=BR
ILAT =19
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production.
Using parameter settings in server-side spfile /u01/app/oracle/product/11.2.0/xe/dbs/spfileXE.ora
System parameters with non-default values:
sessions = 176
sga_target = 576M
control_files = "/u01/app/oracle/oradata/XE/control.dbf"
compatible = "11.2.0.0.0"
db_recovery_file_dest = "/u01/app/oracle/fast_recovery_area"
db_recovery_file_dest_size= 10G
undo_management = "AUTO"
undo_tablespace = "UNDOTBS1"
remote_login_passwordfile= "EXCLUSIVE"
dispatchers = "(PROTOCOL=TCP) (SERVICE=XEXDB)"
shared_servers = 4
job_queue_processes = 4
audit_file_dest = "/u01/app/oracle/admin/XE/adump"
db_name = "XE"
open_cursors = 300
pga_aggregate_target = 195840K
diagnostic_dest = "/u01/app/oracle"
Solved the puzzle, finally.
It turned out that SELinux was active. I don't know how I missed that some weeks ago. I definitely checked that, but maybe I was within the container and not the host.
So I ran the container with the --privileged flag and suddenly I can connect to the db.
Thanks for your comments and sorry that I posted it here, you are right #jww. I will make sure to only post programming questions here in the future. Sorry for that.
You can also edit /etc/sysconfig/docker and remove --selinux-enabled from options that's usually added by default. Then restart docker service:
sudo systemctl restart docker
or
sudo service docker restart
When running sudo make install I run into permission issues. This makes no sense to me as I would expect sudo to have permission to do anything.
Example of sudo vs non sudo execution:
This is also used by node.js make but I ran it separately.
[tydup13#kirk node]$ stat /home/tydup13/git/node/out/Release/obj.target/deps/cares/libcares.a
File: ‘/home/tydup13/git/node/out/Release/obj.target/deps/cares/libcares.a’
Size: 740904 Blocks: 1448 IO Block: 1048576 regular file
Device: 27h/39d Inode: 116264377 Links: 2
Access: (0664/-rw-rw-r--) Uid: (13993/ tydup13) Gid: (13993/ tydup13)
Context: system_u:object_r:nfs_t:s0
Access: 2014-01-30 15:31:52.473071581 -0700
Modify: 2014-01-30 15:29:08.425186992 -0700
Change: 2014-01-30 15:29:08.567191219 -0700
Birth: -
[tydup13#kirk node]$ sudo stat /home/tydup13/git/node/out/Release/obj.target/deps/cares/libcares.a
stat: cannot stat ‘/home/tydup13/git/node/out/Release/obj.target/deps/cares/libcares.a’: Permission denied
I discovered that sudo was squashed on the computer I was using and doesn't allow sudo to cross user boundaries for security. Useful and annoying at the same time.