I want to check modification history of some vital files in linux(like nginx.conf). How to record concrete modification of specific files? Modification like when and which lines files are changed.
Inspired by #Jonathon's comment, I have created a shell script that monitors all the changes done on a specific file, while keeping all the changes history. the script depends on the inotifywait and git packages being installed.
You can find the script here
https://github.com/hisham-hassan/linux-file-monitor
Usage: file-monitor.sh [-f|--file] <absolute-file-path> [-m|--monitor|-h|--history]
file-monitor.sh --help
-f,--file <absolute-file-path> Adding a file to the monitored files List. The <absolute-file-path>
is the absolute file path of the file we need to action.
PLEASE NOTE: Relative file path could cause issues in the script,
please make sure to use the abolute path of the file. also try to
avoid sym links, as it has not been tested.
example: file-monitor.sh -f /absolute/path/to/file/test.txt -m
-m, --monitor Monitoring all the changes on the file. the monitoring will keep
happening as long as the script is running; you may need to run it
in the background.
example: file-monitor.sh -f /absolute/path/to/file/test.txt -m
-h, --history showing the full history of the file.
To exit, press "q"
example: file-monitor.sh -f /absolute/path/to/file/test.txt -h
--uninstall uninstalls the script from the bin direcotry,
and removes the monitoring history.
--install Adds the script to the bin directory, and creates
the directories and files needed for monitoring.
--help Prints this help message.
Related
I want to download all the files in a specific directory of my site.
Let's say I have 3 files in my remote SFTP directory
www.site.com/files/phone/2017-09-19-20-39-15
a.txt
b.txt
c.txt
My goal is to create a local folder on my desktop with ONLY those downloaded files. No parents files or parents directory needed. I am trying to get the clean report.
I've tried
wget -m --no-parent -l1 -nH -P ~/Desktop/phone/ www.site.com/files/phone/2017-09-19-20-39-15 --reject=index.html* -e robots=off
I got
I want to get
How do I tweak my wget command to get something like that?
Should I use anything else other than wget ?
Ihue,
Taking a shell programatic perspective I would recommend you try the following command line script, note I also added the citation so you can see the original threads.
wget -r -P ~/Desktop/phone/ -A txt www.site.com/files/phone/2017-09-19-20-39-15 --reject=index.html* -e robots=off
-r enables recursive retrieval. See Recursive Download for more information.
-P sets the directory prefix where all files and directories are saved to.
-A sets a whitelist for retrieving only certain file types. Strings and patterns are accepted, and both can be used in a comma separated list. See Types of Files for more information.
Ref: #don-joey
https://askubuntu.com/questions/373047/i-used-wget-to-download-html-files-where-are-the-images-in-the-file-stored
Given a path (on my computer), how can I test whether that file is under version control (ie. a copy exists in the Perforce depot)? I'd like to test this at the command line.
Check p4 help files. In short, you run p4 files <your path here> and it will give you the depot path to that file. If it isn't in the depot, you'll get "no such file(s)".
For scripting, p4 files FILE is insufficient because it doesn't change its exit code when there is no such file.
Instead, you can pass it through grep, which looks for perforce paths' telltale pair of leading slashes:
# silently true when the file exists or false when it does not.
p4_exists() {
p4 files -e "$1" 2>/dev/null |grep -q ^//
}
You can get rid of the 2>/dev/null and grep's -q if you want visible output.
Before p4 files version 2012.1 (say p4 files version 2011.1), it didn't support -e. You'll have to add |grep -v ' - delete [^-]*$' before the grep above.
Warning: A future p4 release could change the formatting and break this logic.
Similar to Adam Katz's solution except more likely to be supported by future releases of p4, you can pass global option -s which 'prepends a descriptive field' to each line. This is one of 'text', 'info', 'error' or 'exit' followed by a colon (and a space, it seems). This is intended for facilitating scripting.
For all files passed in to the p4 -s files command, you should get one line back per file. If the file exists in the depot, the line starts with info: whereas if the file does not exist in the depot, the line starts with error:. For example:
info: //depot/<depot-path-to-file>
error: <local-path-to-file>
So, essentially, the status lines are the equivalent of an exit code but on a per-file basis. An exit code alone wouldn't cope neatly with an arbitrary number of files passed in.
Note that if there is a different error (e.g. a connection error) then an error line is still output. So, if you really want the processing to be robust, you may want to combine this with what Adam Katz suggested or perhaps grep for the basename of the file in the output line.
I am experimenting some linux configuration and I want to track my changes? Of course I don't want to to put my whole OS under version control?
Is there a way (with git, mercurial or any VCS) to track the change without storing the whole OS?
This is what I imagine:
I do a kind of git init -> all hashes of all files are stored, but not the content of the files
I make some changes to my file system -> git detect that the hash of this file has changed
I commit -> the content of the file is stored (or even better the original file and the diff are stored! I know, that is impossible... )
Possible? Impossible? Work-arounds?
EDIT: What I care about is just to minimize the size of the repository and to have a repository containing only my changes. Having all files in my repository is not relevant for me. For example if i push to github I just want it to contain only the files that has changed.
Take a look at etckeeper, it will probably do the job.
What you want is git update-index --info-only or ... --index-info, from the man page: " --info-only is used to register files without placing them in the object database. This is useful for status-only repositories.". --index-info is its industrial-scale cousin.
Do that with the files you want to track, write-tree to write the index structure into the object db, commit-tree that, and update-ref to update a branch.
To get the object name use git hash-objectfilename.
Here is what we do...
su -
cd /etc
echo "*.cache" > .gitignore
git init
chmod 700 .git
cd /etc; git add . && git add -u && git commit -m "Daily Commit"
Then setup crontab:
su -
crontab -e
# Put the following in:
0 3 * * * cd /etc; git add . && git add -u && git commit -m "Daily Commit"
Now you will have a nightly commit of all changes in /etc
If you want to track more than /etc in one repo, then you could simply do it at the root of your filesystem, except add the proper ignore paths to your /.gitignore. I am unclear on the effects of having git within git, so you might want to be extra careful in that case.
I know this question is old, but I thought this might help someone. Inspired by #Jonathon's comment on the How to record concrete modification of specific files question, I have created a shell script that enables you to monitors all the changes done on a specific file, while keeping all the changes history. the script depends on the inotifywait and git packages being installed.
You can find the script here
https://github.com/hisham-hassan/linux-file-monitor
Usage: file-monitor.sh [-f|--file] <absolute-file-path> [-m|--monitor|-h|--history]
file-monitor.sh --help
-f,--file <absolute-file-path> Adding a file to the monitored files List. The <absolute-file-path>
is the absolute file path of the file we need to action.
PLEASE NOTE: Relative file path could cause issues in the script,
please make sure to use the abolute path of the file. also try to
avoid sym links, as it has not been tested.
example: file-monitor.sh -f /absolute/path/to/file/test.txt -m
-m, --monitor Monitoring all the changes on the file. the monitoring will keep
happening as long as the script is running; you may need to run it
in the background.
example: file-monitor.sh -f /absolute/path/to/file/test.txt -m
-h, --history showing the full history of the file.
To exit, press "q"
example: file-monitor.sh -f /absolute/path/to/file/test.txt -h
--uninstall uninstalls the script from the bin direcotry,
and removes the monitoring history.
--install Adds the script to the bin directory, and creates
the directories and files needed for monitoring.
--help Prints this help message.
I need to write a shell script to run as a cron task, or preferably on creation of a file in a certain folder.
I have an incoming and an outgoing folder (they will be used to log mail). There will be files created with codes as follows...
bmo-001-012-dfd-11 for outgoing and 012-dfd-003-11 for incoming. I need to filter the project/client code (012-dfd) and then place it in a folder in the specific project folder.
Project folders are located in /projects and follow the format 012-dfd. I need to create symbolic links inside the incoming or outgoing folders of the projects, that leads to the correct file in the general incoming and outgoing folders.
/incoming/012-dfd-003-11.pdf -> /projects/012-dfd/incoming/012-dfd-003-11.pdf
/outgoing/bmo-001-012-dfd-11.pdf -> /projects/012-dfd/outgoing/bmo-001-012-dfd-11.pdf
So my questions
How would I make my script run when a file is added to either incoming or outgoing folder
Additionally, is there any associated disadvantages with running upon file modification compared with running as cron task every 5 mins
How would I get the filename of recent (since script last run) files
How would I extract the code from the filename
How would I use the code to create a symlink in the desired folder
EDIT: What I ended up doing...
while inotifywait outgoing; do find -L . -type l -delete; ls outgoing | php -R '
if(
preg_match("/^\w{3}-\d{3}-(\d{3}-\w{3})-\d{2}(.+)$/", $argn, $m)
&& $m[1] && (file_exists("projects/$m[1]/outgoing/$argn") != TRUE)
){
`ln -s $(pwd)/outgoing/$argn projects/$m[1]/outgoing/$argn;`;
}
'; done;
This works quite well - cleaning up deleted symlinks also (with find -L . -type l -delete) but I would prefer to do it without the overhead of calling php. I just don't know bash well enough yet.
Some near-answers for your task breakdown:
On linux, use inotify, possibly through one of its command-line tools, or script language bindings.
See above
Assuming the project name can be extracted thinking positionally from your examples (meaning not only does the project name follows a strict 7-character format, but what precedes it in the outgoing file also does):
echo `basename /incoming/012-dfd-003-11.pdf` | cut -c 1-7
012-dfd
echo `basename /outgoing/bmo-001-012-dfd-11.pdf`| cut -c 9-15
012-dfd
mkdir -p /projects/$i/incoming/ creates directory /projects/012-dfd/incoming/ if i = 012-dfd,
ln -s /incoming/foo /projects/$i/incoming/foo creates a symbolic link from the latter argument, to the preexisting, former file /incoming/foo.
How would I make my script run when a file is added to either incoming or outgoing folder
Additionally, is there any associated disadvantages with running upon file modification compared with running as cron task
every 5 mins
If a 5 minutes delay isn't an issue, I would go for the cron job (it's easier and -IMHO- more flexible)
How would I get the filename of recent (since script last run) files
If your script runs every 5 minutes, then you can tell that all the files created in between now (and now - 5 minutes) are newso, using the command ls or find you can list those files.
How would I extract the code from the filename
You can use the sed command
How would I use the code to create a symlink in the desired folder
Once you have the desired file names, you can usen ln -s command to create the symbolic link
On Linux, I need to know which files were added/modified/moved/deleted after compiling and installing an application from source code, ie. the command-line, Linux equivalent to the venerale InCtrl5.
Is there a utility that does this, or a set of commands that I could run and would show me the changes?
Thank you.
Edit: The following commands are sort of OK, but I don't need to know the line numbers on which changes occured or that "./.." were updated:
# ls -aR /tmp > b4.txt
# touch /tmp/test.txt
# ls -aR /tmp > after.txt
# diff -u b4.txt after.txt
If you only need to know which files were touched, then you can use find for this:
touch /tmp/MARK
# install application here
find / -newercm /tmp/MARK
This will show you all files whose contents or metadata have changed since you touched /tmp/MARK (including newly added files).
I would personally use something like Mercurial (version control) to do this.
The main reason, is that it is not only effective but it is also clean, since it will only add a hidden directory to the top of the tree where you want to check these changes.
Let's say that you need to know what files changed in /etc/. So before installation (you need to have mercurial installed) you add the directory to mercurial:
cd /etc
hg init
hg add
hg ci -m "adding all files in /etc/ to track them down"
The above will effectively "add" all the files to track them. To verify nothing has changed:
hg st
Should return no files.
If you (or the installation) modifies a file, you should see something like this:
hg st
M foo.sh
The "M" before the file states the given file was modified.
For new files you would see a ? before the file like:
? bar.sh
After you are done and no longer want Mercurial, simple remove the hidden directory:
cd /etc
rm -rf .hg