I am interested to know how the file system data structures are handled when suspending to disk.
I am wondering if it is possible to hibernate and then switch a disk (physically) and then resume. My guess is that all of the inodes in memory for hibernation would be invalid. Is there any way to refresh the inode data or any place in the Kernel source I could look that might provide an example?
It is not possible for me to umount the disk, I am curious what options are available to me.
I think one thing you can do is this:
boot on disk 1, hibernate.
Change to disk 2, and press the power button. The computer will boot again (not wake up).
hibernate on disk 2.
Switch to disk 1, and it has a better chance to wake up successfully.
When you remove it when should down, the hibernate image does not fit any longer to the configuration, as the swap does not happen when the system either can notice it or it doesnt care (because its really off). Maybe you are lucky, but I wont risk it. Many people dont know it, but SATA (and Linux) are capable of hot swap - maybe its an option for you to remove the old disk and connect a new one after you came back from hibernate.
Related
So, I have a scientific server with a HDD and a SSD hard drive.
Where for computations involving lot's of data reading/writing a user can use the SSD but all the home directories are on the HDD.
Is there an automatic way to redirect the output of any program writing on the SSD to the home directory of the user running the program if the SSD is full?
If the best solution is to write my own script, then what is the best way to determine if the SSD runs out of space?
My OS is Ubuntu 18.04 LTS
In short, I do not think there is such a thing and I do believe that you should implement a bash script that checks (my tool of choice would simply be df) that there is enough space for you to run the next computation run before actually doing it. Maybe you should pre-allocate the space you intend to use, if possible, to avoid other concurrent runs to crash/run out of space? Maybe you should have an automated procedure to clean up some space?
Obviously, you could have the ssd available on some mountpoint in /home/, and then periodically check with a cron job whether it is full. And the maybe unmount it and send a warning mail. This will sort of do what you want. Sort of. But what happens then when also the HDD gets full? Watch out- these kind of problems can easily cause a server to crash, or otherwise experience issues.
This looks like a problem you might partially solve/mitigate by e.g., using a quota scheme (that is, limiting the amount of space that each user can allocate) or better yet by using a dedicated system for queueing jobs and allocating resources.
I need to have a robust filesystem on an debian or ubuntu linux. The problem is, that the system can be "shutdown" by just cut the power ( without a real shutdown ).
After such a scenario I don't want to have the filesystem check or corrupted data on the root filesystem. I need to make sure that the system starts without problems again the next time. What could be the solution?
Along with journaling , Soft Updates can also be a method.Regardless of the method you use, please keep in mind that these methods can only guarantee you the consistency of the file system and neither of them can guarantee the safety of your data.Disk can be mounted more quickly in case of soft updates.
I am trying to measure the effects of adding memory to a LAMP server.
How can I find which processes try to read from the Linux buffer cache, but miss and read from disk instead?
SystemTap is one of the best ways to do this, but fair warning it's difficult to get a great answer. The kernel simply doesn't provide this data directly. You have to infer it based on how many times the system requested a read and how many times a disk was read from. Usually they line up fairly well and you can attribute the difference to the VFS cache, but not always. One problem is LVM- LVM is a "block device", but so is the underlying disk(s), so if you're not careful it's easy to double-count the disk reads.
A while back I took a stab at it and wrote this:
https://sourceware.org/systemtap/wiki/WSCacheHitRate
I do not claim that it is perfect, but it works better than nothing, and usually generates reasonable output as long as the environment is fairly "normal". It does attempt to account for LVM in a fairly crude way.
I'm creating a web application running on a Linux server. The application is constantly accessing a 250K file - it loads it in memory, reads it and sends back some info to the user. Since this file is read all the time, my client is suggesting to use something like memcache to cache it to memory, presumably because it will make read operations faster.
However, I'm thinking that the Linux filesystem is probably already caching the file in memory since it's accessed frequently. Is that right? In your opinion, would memcache provide a real improvement? Or is it going to do the same thing that Linux is already doing?
I'm not really familiar with neither Linux nor memcache, so I would really appreciate if someone could clarify this.
Yes, if you do not modify the file each time you open it.
Linux will hold the file's information in copy-on-write pages in memory, and "loading" the file into memory should be very fast (page table swap at worst).
Edit: Though, as cdhowie points out, there is no 'linux filesystem'. However, I believe the relevant code is in linux's memory management, and is therefore independent of the filesystem in question. If you're curious, you can read in the linux source about handling vm_area_struct objects in linux/mm/mmap.c, mainly.
As people have mentioned, mmap is a good solution here.
But, one 250k file is very small. You might want to read it in and put it in some sort of memory structure that matches what you want to send back to the user on startup. Ie, if it is a text file an array of lines might be a good choice, etc.
The file should be cached, but make sure the noatime option is set on the mount, otherwise the access time will attempt to be saved to the file, invalidating the cache.
Yes, definitely. It will keep accessed files in memory indefinitely, unless something else needs the memory.
You can control this behaviour (to some extent) with the fadvise system call. See its "man" page for more details.
A read/write system call will still normally need to copy the data, so if you see a real bottleneck doing this, consider using mmap() which can avoid the copy, by mapping the cache pages directly into the process.
I guess putting that file into ramdisk (tmpfs) may make enough advantage without big modifications. Unless you are really serious about response time in microseconds unit.
If I start copying a huge file tree from one position to another or if some other process starts doing lots of disk activity, the foreground app (GUI) slows way down. For example, take a 2gb file tree with 100k files in it. Open a console and do cp -r bigtree bigtree2. Then go to firefox and start browsing. Firefox is almost unusable. Even if I set firefox's nice level to really high priority (-20), it's still super slow with huge delays.
I remember some years ago when I worked on a Solaris box, the system behaved much better in similar circumstances.
My HD is using DMA, not PIO. It's SATA. Not mounted with the atime flag.
Linux has long had a problem with programs that hog all the system's "dirty" cache memory. What is happening is that the copy process is filling the write cache with the file data it is copying and it is doing it very quickly. So when Firefox comes along and needs to write it must first wait for dirty buffer space or an available disk queue write slot. While waiting it is competing with the copy process and the kernel's pdflush thread, which moves data from dirty buffers to the disk write queue.
Firefox has yet another problem in this scenario. It uses SQLite to store its bookmarks, history and other things. SQLite is a ACID compliant database and it uses a transaction system with its disk writes flushed to disk. So not only does it have to wait for buffer space, it must wait for the disk queue, which is full of copied file, to clear out before it can acknowledge a successful write.
There has been a lot of tweaking done to the Linux disk queuing and buffering system. There are changes in almost every kernel release. Try one of the newer releases. You can also try tweaking the sysctl values. I sort of like these:
vm.dirty_writeback_centisecs = 100
vm.dirty_expire_centisecs = 9000
vm.dirty_background_ratio = 4
vm.dirty_ratio = 80
You can also try tweaking the number of slots in the disk queue. This value is in /sys/block/sda/queue/nr_requests. You need to substitute sda with whatever your drive really is. More slots means more chances to merge IO requests and the CFQ IO scheduler can do a better job with priorities. Fewer slots usually means a shorter wait to get written to disk for synchronous IO like SQLite's transactions. Fewer slots also means a shorter wait to get read IO into the disk queue if a write-heavy process completely stuffs the queue with write IO.
Try ionice-ing or nice-ing the copy process. The issue is due to the fact that IO gets the same priority as the GUI, which for a desktop, affects perceived responsiveness.
There's an Ubuntu brainstorm about this currently.
You're not the first to notice this problem. Former kernel developer [Con Kolivas] (http://en.wikipedia.org/wiki/Con_Kolivas) found that a lot of companies are paying to improve linux server performance at the expense of desktop performance. Con had an impressive set of patches for making the desktop more responsive. Unfortunately there was some sort of code war and eventually Con dropped out.
I would love to know how to petition the Linux kernel developers for better desktop performance. In the meantime, if you are willing to run kernel 2.6.22, you can run with the -ck patch set.
Make sure that DMA is enabled on all your drives that support it. Depending on your distribution this may not be the default. Read man hdparm, and look into your systems init mechanism.