I have hosted WSO2 API Manager and WSO2 Idenity server in AWS Ec2 instance.
Now its running out of memory.
I have cleared logs and tried ; but its still lacking space.
I got solutions on how to increase the size.But I want to know if there is something which i am missing. May be is there some configuration or temp directories which i should be clearing?
My df command shows the below output and /dev/xvda1 is fully user
$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 16G 56K 16G 1% /dev
tmpfs 16G 0 16G 0% /dev/shm
/dev/xvda1 9.8G 9.7G 0 100% /
Related
I was trying to copy a 1.5GiB file from a location to another and was warned that my disk space is full, so I proceeded to a verification using df -h, which gave the following output:
Filesystem Size Used Avail Use% Mounted on
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 114M 16G 1% /dev/shm
tmpfs 16G 2.0M 16G 1% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/mapper/RHELCSB-Root 50G 11G 40G 21% /
/dev/nvme0n1p2 3.0G 436M 2.6G 15% /boot
/dev/nvme0n1p1 200M 17M 184M 9% /boot/efi
/dev/mapper/RHELCSB-Home 100G 100G 438M 100% /home
tmpfs 3.1G 88K 3.1G 1% /run/user/4204967
where /dev/mapper/RHELCSB-Home seemed to cause the issue. But when running sudo du -xsh /dev/mapper/RHELCSB-Home, I got the following result:
0 /dev/mapper/RHELCSB-Home
and same thing for /dev/ and /dev/mapper/. After researching this issue, I figured out that this might have been caused by undeleted log files in /var/log/, but the total size of files there is far from approaching the 100GiB. What could cause my disk space to be full?
Additional context: I was running a local postgresql database when this happened, but I can't see how this can relate to my issue as postgres log files are not taking that much space either.
The issue was solved by deleting podman container volumes in ~/.local/share/containers/
I have mounted 600 GB of extra space in my AWS EC2 at /data. But as I started using Jenkins i realized that My Jenkins is not using any of that extra space and now I am left with only 1.5 GB of storage.
Is there any way to merge the extra storage with root storage?
Result of df -h command
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.9G 68K 7.9G 1% /dev
tmpfs 7.9G 4.0K 7.9G 1% /dev/shm
/dev/xvda1 7.9G 6.3G 1.5G 82% /
/dev/xvdf 600G 1.8G 598G 1% /data
I want to merge /dev/xvda1 and /dev/xvdf.
Is it even possible?
Edit: Someone suggested to move my jenkins to new drive. If it will not not hamper my current work then i think it will be a good solution. Any opinions on this?
Quick way:
you can stop your jenkins instance and create AMI image on it.
Then create base on this image as a new EC2 instance with large storage directly.
I was trying to build squid container which work as proxy for list of users and use different configuration such as "login using custom python script" ssl bump to stop https urls and some ACL rules
but i get that most of time squid is very slow and take much time to be up and ready and slow to use
when i read the cache.log file i get always
ipcCreate: fork: (12) Cannot allocate memory
I tried also to run docker container using "--memory-swap -1" flag and still same
df -h in the docker container :
Filesystem Size Used Avail Use% Mounted on
none 30G 16G 14G 54% /
tmpfs 496M 0 496M 0% /dev
tmpfs 496M 0 496M 0% /sys/fs/cgroup
/dev/xvda1 30G 16G 14G 54% /etc/hosts
shm 64M 2.1M 62M 4% /dev/shm
tmpfs 496M 0 496M 0% /sys/firmware
I have 5 Linux machines cluster. There are 3 data nodes and one master. At now about 50% hdfs storage is available on each data nodes. But I run a mapreduce job, It is failed with following error
2017-08-21 17:58:47,627 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for blk_6835454799524976171_3615612 bad datanode[0] 10.11.1.42:50010
2017-08-21 17:58:47,628 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_6835454799524976171_3615612 in pipeline 10.11.1.42:50010, 10.11.1.43:50010: bad datanode 10.11.1.42:50010
2017-08-21 17:58:51,785 ERROR org.apache.hadoop.mapred.Child: Error in syncLogs: java.io.IOException: No space left on device
While on each system df -h gives following information
Filesystem Size Used Avail Use% Mounted on
devtmpfs 5.9G 0 5.9G 0% /dev
tmpfs 5.9G 84K 5.9G 1% /dev/shm
tmpfs 5.9G 9.1M 5.9G 1% /run
tmpfs 5.9G 0 5.9G 0% /sys/fs/cgroup
/dev/mapper/centos-root 50G 6.8G 44G 14% /
/dev/sdb 1.8T 535G 1.2T 31% /mnt/11fd6fcc-1f87-4f1e-a53c-54cc7117759c
/dev/mapper/centos-home 412G 155G 59M 100% /home
/dev/sda1 494M 348M 147M 71% /boot
tmpfs 1.2G 16K 1.2G 1% /run/user/42
tmpfs 1.2G 0 1.2G 0% /run/user/1000
As clear from above that my sdb dicsk (SDD) is only 31% used but centos-home is 100%. While hadoop is using local file system in mapreduce job when there is enough HDFS available? Where is the problem? I have search at google and found many such problem but no one covers my situation.
syncLogs does not use HDFS, it writes to hadoop.log.dir so
if you're using MapReduce, look for the value of hadoop.log.dir in /etc/hadoop/conf/taskcontroller.cfg.
If you're using YARN, look for the value of yarn.nodemanager.log-dirs in the yarn-site.xml.
One of these should point you to where you're writing your logs. Once you figure out which filesystem has the problem, you can free space from there.
Another thing to remember is you could get "No space left on device" if you've exhausted your inodes on your disk. df -i would show this.
Please check how many inodes are used. If I undertand it right, if it is still the full disk, but all inodes has gone, the error would be still the same, "no space left".
I am creating an Azure VM using docker-machine as follows.
docker-machine create --driver azure --azure-size Standard_DS2_v2 --azure-subscription-id #### --azure-location southeastasia --azure-image canonical:UbuntuServer:14.04.2-LTS:latest --azure-open-port 80 AwesomeMachine
following the instructions here. Azure VM docs say - Max. disk size of Standard_DS2_v2 is 100GB,
however when I login to the machine (or create a container on this machine), the max available disk size I see is 30GB.
$ docker-machine ssh AwesomeMachine
docker-user#tf:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 29G 6.9G 21G 25% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 3.4G 12K 3.4G 1% /dev
tmpfs 698M 452K 697M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 3.5G 1.1M 3.5G 1% /run/shm
none 100M 0 100M 0% /run/user
none 64K 0 64K 0% /etc/network/interfaces.dynamic.d
/dev/sdb1 14G 35M 13G 1% /mnt
What is the meaning of Max. disk size then? Also what is this /dev/sdb1? Is it usable space?
My bad, I didn't look at the documentation carefully.
Wo when --azure-size is Standard_DS2_v2, Local SSD disk = 14 GB, which is /dev/sdb1, while
--azure-size Standard_D2_v2 gives you Local SSD disk = 100 GB.
Not deleting the question in case somebody else makes the same stupid mistake.