I have experienced a weird behaivour on our production environment.
We have a NFS and a linux server which runs our application.
On the linux server there is a mount to the NFS,
from: /configuration/data (on the NFS)
To: /software (on the linux server).
Which the application modifies the files there periodically.
Some time ago someone accidentally moved to "data" folder to: /configuration/other/data
The application kept running without any side effect, and modified the files periodically, and the files inside /configuration/other/data also changed even though the mount (/configuration/data) point to nothing.
I guess there is a shortcut to the origin of the mount which being modified on the folder relocation, but that just a guess.
I would like to know why and how this behaivour is possible, and how it works internally.
and how it works internally.
File descriptor refers to a file. You can move the file, you can remove the file - the file descriptor refers to the same "entity". So in shell you can for example:
# open fd 10 to refer to /tmp/10
$ exec 10>/tmp/10
# Just write something so that it works
$ echo abc >&10
$ cat /tmp/10
abc
# Move the file to some dir
$ mkdir /tmp/dir
$ mv /tmp/10 /tmp/dir/10
# now it still writes to the same "file", even when moved
$ echo def >&10
$ cat /tmp/dir/10
abc
def
# You can remove the file and still access it
# The file still "exists"
$ exec 11</tmp/dir/10
$ rm /tmp/dir/10
$ echo 123 >&10
$ cat <&11
abc
def
123
Creating a file descriptor to a file and then removing the file is typically in C programs:
char *filename = mkstemp(...);
int fd = open(filename);
unlink(filename);
// use fd
The file is really "removed" when there are no links to it - when the last file descriptor is closed. Research posix file descriptors and see for example man 2 unlink and other resources explaining what is a file descriptor.
Most probably you application has continously opened file descriptors to files inside /configuration/data, so after the file is moved, the data become available at the new location but application still uses same file descriptors.
Related
I have a list of machine names and hostnames
ex)
# cat /tmp/machine_list.txt
[one]apple machine #1 myserver1
[two]apple machine #2 myserver2
[three]apple machine #3 myserver3
and, server each directory
and each directory contains an tar file and a file with the host name written on it.
# ls /tmp/sos1/*
sosreport1.tar.gz
hostname_map.txt
# cat /tmp/sos1/hostname_map.txt
myserver1
# ls /tmp/sos2/*
sosreport2.tar.gz
hostname_map.txt
# cat /tmp/sos2/hostname_map.txt
myserver2
# ls /tmp/sos3/*
sosreport3.tar.gz
hostname_map.txt
# cat /tmp/sos3/hostname_map.txt
myserver3
Is it possible to rename the sosreport*.tar.gz by referencing the hostname_map in each directory relative to the /tmp/machine_list.txt file? (like below)
# ls /tmp/sos1/*
[one]apple_machine_#1_myserver1_sosreport1.tar.gz
# ls /tmp/sos2/*
[two]apple_machine_#2_myserver2_sosreport2.tar.gz
# ls /tmp/sos3/*
[three]apple_machine_#3_myserver3_sosreport3.tar.gz
A single change is possible, but what about multiple changes?
Something like this?
srvname () {
awk -v srv="$(cat "$1")" -F '\t' '$2==srv { print $1; exit }' machine_list.txt
}
for dir in /tmp/sos*/; do
server=$(srvname "$dir"/hostname_map.txt)
mv "$dir"/sosreport*.tar.gz "$dir/$server.tar.gz"
done
Demo: https://ideone.com/TS5VyQ
The function assumes your mapping file is tab-delimited. If you want underscores instead of spaces in the server names, change the mapping file.
This should be portable to POSIX sh; the cat could be replaced with a Bash redirection, but I feel that it's not worth giving up portability for such a small change.
If this were my project, I'd probably make the function into a self-contained reusable script (with the input file replaced with a here document in the script itself) since there will probably be more situations where you need to perform the same mapping.
I'm having a hard time understanding something about redirections in bash.
I'll start with what I know:
Each process has file descriptors opened which it can write to/read from. These file descriptors may represent files on disk, terminals, devices, etc.
When we start teminal with bash, we have file stdin (0) stdout (1) and stderr (2) opened, pointing to the terminal. Whenever we run a command (a new process), that process inherits the file descriptors of its parent (bash), so by default, it will print stdout and stderr messages to the terminal, and read from terminal also.
When we redirect, for example:
$ ls 1>filelist
We're actually changing file descriptor 1 of the ls process, to point to the filelist file, instead of the terminal. So when ls will write(1, ...) it will go to the file.
So to sum it up, a redirection is basically changing the file to which the file descriptor to which the program writes/reads to/from refers to.
Now, let's say I have the following C program:
#include <stdio.h>
#include <fcntl.h>
int main()
{
int fd = 0;
fd = open("info.log", O_CREAT | O_RDWR);
printf("%d", fd);
write(fd, "INFO::", 6);
return 0;
}
This program opens a file info.log, which is referred to by a file descriptor (usually 3).
Indeed, if I now compile this program and run it:
$ ./app
3
It creates the file info.log which contains the "INFO::" text in it.
But here's what I don't get: according to the logic described above, if I now redirect FD 3 to another file:
$ ./app 3> another_file
The text should be written to this other file, but for some reason, it doesn't.
Can someone explain?
Hint: when you run ./app 3> another_file, it'll print "4" instead of "3".
More detailed explanation: when you run ./app 3> another_file in the shell, a series of things happens:
The shell fork()s a subprocess that'll run ./app. The subprocess is basically a clone of its parent process so, it'll still be running the shell program.
In that subprocess, the shell opens "another_file" on file descriptor #3 for writing.
Then it uses one of the execl() family of calls to execute the ./app binary (with "another_file" still open on FD#3).
The program runs open("info.log", O_CREAT | O_RDWR), which creates "info.log" and opens it on the next available file descriptor. Since FD#3 is already in use, that's FD#4.
The program writes "INFO::" to FD#4, which is "info.log".
Since open() uses a new FD, it's not really affected by any active redirects. And actually, if the program did open something on FD#3, that'd replace the connection to "another_file" with whatever it had opened instead, essentially overriding the redirect.
If the program wanted to use the redirect, it'd have to write to FD#3 without first opening anything on it. This is what's normally done with FD#1 and 2 (standard output and error), and that's why redirecting those works.
I am trying to write a file using tcl scripting (Via VMD). when I type command "dir" on tk/tcl console, it shows file name which I am trying to generate. But when I tried to open that file manually in that working directory folder, it is not even shown in it.
Here is the piece of code.
set fp [open "input.txt" w+]
puts $fp "test"
close $fp
Sometimes, Windows can be deeply tricky about what is going on with the actual folders.
If you do this in your script as well:
# This is quite an incantation!
exec {*}[auto_execok cmd] /c start "" [file nativename [file normalize .]]
then it should open an Explorer window on the actual current directory. That will let you check whether the file is actually there, and whether where you are is where you expect to be.
This should match up with what this says:
# Print the current working directory
puts [pwd]
# Print the names of text files in the current directory, with full names
puts [glob -directory [pwd] *.txt]
If it doesn't, that might point to your real problem…
Upon looping a directory to delete txt files ONLY - a message is returned indicating The System cannot find the file specified: 'File.txt'.
I've made sure the txt files that I'm attempting to delete exist in the directory I'm looping. I've also checked my code and to make sure it can see my files by printing them in a list with the print command.
import os
fileLoc = 'c:\\temp\\files'
for files in os.listdir(fileLoc):
if files.endswith('.txt'):
os.unlink(files)
Upon initial execution, I expected to see all txt files deleted except for other non-txt files. The actual result was an error message "FileNotFoundError: [WinError 2] The system cannot find the file specified: 'File.txt'.
Not sure what I'm doing wrong, any help would be appreciated.
It isn't found because the the path you intended to unlink is relative to fileLoc. In fact with your code, the effect is to unlink the file relative to the current working directory. If there were *.txt files
in the cwd then the code would have unfortunate side-effects.
Another way to look at it:
Essentially, by analogy, in the shell what you're trying to do is equivalent to this:
# first the setup
$ mkdir foo
$ touch foo/a.txt
# now your code is equvalent to:
$ rm *.txt
# won't work as intended because it removes the *.txt files in the
# current directory. In fact the bug is also that your code would unlink
# any *.txt files in the current working directory unintentionally.
# what you intended was:
$ rm foo/*.txt
The missing piece was the path to the file in question.
I'll add some editorial: The Old Bard taught us to "when in doubt, print variables". In other words, debug it. I don't see from the OP an attempt to do that. Just a thing to keep in mind.
Anyway the new code:
Revised:
import os
fileLoc = 'c:\\temp\\files'
for file in os.listdir(fileLoc):
if file.endswith('.txt'):
os.unlink(os.path.join(fileLoc,file))
The fix: os.path.join() builds a path for you from parts. One part is the directory (path) where the file exists, aka: fileLoc. The other part is the filename, aka file.
os.path.join() makes a whole valid path from them using whatever OS directory separator is appropriate for your platform.
Also, might want to glance through:
https://docs.python.org/2/library/os.path.html
I working with linux, bash.
I have one directory with 100 folders in it, each one named different.
In each of these 100 folders, there is a file called first.bars (so I have 100 files named first.bars). Although all named first.bars, the files are actually slightly different.
I want to get all these files moved to one new folder and rename/number these files so that I know which file comes from which folder. So the first first.bars file must be renamed to 001.bars, the second to 002.bars.. etc.
I have tried the following:
ls -d * >> /home/directorywiththe100folders/list.txt
cat list.txt | while read line;
do cd $line;
mv first.bars /home/newfolder
This does not work because I can't have 100 files, named the same, in one folder. So I only need to know how to rename them. The renaming must be connected to the cat list.txt, because the first line is the folder containing the first file wich is moved and renamed. That file will be called 001.bars.
Try doing this :
$ rename 's/^.*?\./sprintf("%03d.", $c++)/e' *.bar
If you want more information about this command, see this recent response I gave earlier : How do I rename multiple files beginning with a Unix timestamp - imapsync issue
If the rename command is not available,
for d in /home/directorywiththe100folders/*/; do
newfile=$(printf "/home/newfolder/%d.bars" $(( c++ )) )
mv "$d/first.bars" "$newfile"
done