I have noticed that my small 8MB database is cached by PostgreSQL 12 in this folder:
/var/lib/postgresql/12/main/base/16384
This folder contains multiple 1GB files for example "16417" that weighs 1073741824 bytes.
How to remove these files and limit cache file space to max 100 GB? Now it uses as much space as it can and crashes my disk (no space left).
In postgresql.conf file i have changed these options:
temp_file_limit = 10000000
shared_buffers = 128MB
checkpoint_timeout = 12h
max_wal_size = 100MB
min_wal_size = 80MB
but unfortunately it did not help.
What else can I do to resolve this issue? In one our these files grew up to 80 GB...
EDIT: This issue occurs even with default settings. My system is Ubuntu 18.04.4.
This is not a cache, these are the actual tables and indexes. If you mess with these files, you will break your database and lose data.
Figure out what database 16384 is:
SELECT datname FROM pg_database WHERE oid = 16384;
Then connect to that database and figure out what 16417 is:
SELECT relname, relnamespace::regnamespace, relkind
FROM pg_class WHERE relfilenode = 16417;
If the size of that object is bigger than it should be, perhaps you have a bloated table or index, and VACUUM (FULL) tab can make it smaller (but don't forget that the table is inaccessible while it is rewritten!).
Again, make sure you don't manipulate any of those files yourself.
Related
i was looking for a specific file in windows explorer when i noticed that there are copies of "desktop.ini" file everywhere, i went to the parent folder and ran a windows search for "desktop.ini" and found about 5000 files with the same name at various locations (subfolders), all in the range of about 100-200 bytes, i selected all of them and deleted them, why did it take windows about 2.5 minutes to delete all of those files? assuming an average file size of about 150 bytes, the amount of data to be deleted should be approximately (150*5000) / 1024 kBytes, i.e, about 732 kB. how is windows then able to delete a single file of much greater sizes instantaniously?
Delete function is O(n), where n equals the item count, not the file size....
in English:
your file's aren't "erased".... only it's registration entry is deleted. This depends mostly on the amount of files, not the amount of bytes.
By the way, this is how you can recover "deleted files"; by restoring it's registration entry.
I have a Liferay 6.2 server that has been running for years and is starting to take a lot of database space, despite limited actual content.
Table Size Number of rows
--------------------------------------
DLFileRank 5 GB 16 million
DLFileEntry 90 MB 60,000
JournalArticle 2 GB 100,000
The size of the DLFileRank table sounds to me as abnormally big (if it is totally normal please let me know).
While the file ranking feature of Liferay is nice to have, we would not really mind resetting it if it halves the size of the database.
Question: Would a DELETE * FROM DLFileRank be safe? (stop Liferay, run that SQL command, maybe set dl.file.rank.enabled=false in portal-ext.properties, start Liferay again)
Is there any better way to do it?
Bonus if there is a way to keep recent ranking data and throw away only the old data (not a strong requirement).
Wow. According to the documentation here (Ctrl-F rank), I'd not have expected the number of entries to be so high - did you configure those values differently?
Set the interval in minutes on how often CheckFileRankMessageListener
will run to check for and remove file ranks in excess of the maximum
number of file ranks to maintain per user per file. Defaults:
dl.file.rank.check.interval=15
Set this to true to enable file rank for document library files.
Defaults:
dl.file.rank.enabled=true
Set the maximum number of file ranks to maintain per user per file.
Defaults:
dl.file.rank.max.size=5
And according to the implementation of CheckFileRankMessageListener, it should be enough to just trigger DLFileRankLocalServiceUtil.checkFileRanks() yourself (e.g. through the scripting console). Why you accumulate that large number of files is beyond me...
As you might know, I can never be quoted by stating that direct database manipulation is the way to go - in fact I refuse thinking about the problem from that way.
please check the workbench output after 20 mints its writing only 100-200 records persecond before that its writing 1211-2000 writes per second I am trying to insert 2 million records on the MySQL 5.7.10 RDS server, Its taking almost 40 mints to insert the data on Linux environment where as the same data are inserted in 28 mints on Windows platform.
On Linux I am using SSD disk still its taking long time to insert.
My hardware configuration is:-
SSD Disk
RAM:- 122 GB
CPU:- 16 Cores
My MySQL configuration is:-
innodb_buffer_pool_size=80 G
innodb_log_file_size= 1G
innodb_log_buffer_size= 64MB
innodb_buffer_pool_instances = 28
tmp_table_size =4G
max_heap_size= 4G
table_open_cache=32262
innodb_file_per_table=ON
innodb_flush_log_at_trx_commit=2
innodb_flush_method=O-DIRECT
Team please check and help on this.
Thanks in advance.
tmp_table_size =4G -- lower to 1G
max_heap_size= 4G -- lower to 1G
table_open_cache=32262 -- lower to, say, 1000
What filesystem (xfs, ext4, etc)? RAID?
Please show us the insert command(s). Where is the source data coming from (same drive, different machine, etc)?
More
Batch the INSERTs -- but BEGIN and COMMIT around 100-1000 rows at a time.
There is a single line setting that can resolve the issue:
innodb_flush_log_at_trx_commit = 2
I have a filesystem with 40 million files in a 10 level tree structure (around 500 GB in total). The problem I have is the backup. An Incr backup (bacula) takes 9 hours (around 10 GB) with a very low performance. Some directories have 50k files, other 10k files. The HDs are HW RAID, and I have the default Ubuntu LV on top. I think the bottleneck here is the # of files (the huge # of inodes.) I'm trying to improve the performance (a full backup on the same FS takes 4+ days, at 200k/s read speed).
- Do you think that partitioning the FS into several smaller FS would help? I can have 1000 smaller FS...
- Do you think that moving from HD to SSD would help?
- Any advice?
Thanks!
Moving to SSD will improve the speed of the backup. The SSD will get tired very soon and you will need the backup...
Can't you organise things that you know where to look for changed/new files?
In that way you pnlu need to increment-backup those folders.
Is it necessary your files are online? Can you have tar files of old trees 3 levels deep?
I guess a find -mtime -1 will take hours as well.
I hope that the backup is not using the same partition as de tree structure
(everything under /tmp is a very bad plan), the temporary files the bavkup might make should be on a different partition.
Where are the new files coming from? When all files are changed by a process you control, your process can make a logfile with a list of files changed.
I am selecting billions of queries from oracle database. Running sqlplus on RHEL 5. I am doing following:
>spool on
>set heading off
>set trimsp on pages 0
1) Problem is that spool file is becoming larger than 4GB and it doesn't write selected rows in this file after that.
How can i achieve that? (i found that resultant file should be larger approx. 9GB)
2) Also i don't want the result queries to be displayed on console. They should go directly in the spool file. (as it decreases the speed of this operation)
1) Im not sure but according to this source there seems to be a limit on the size of spool file. Usually file system used also places limits on the size of the file. So this has something to do with file size limit perhaps.
2) For this you can add SET ECHO OFF which displays any output from coming onto the console