How do I use file command for copying the files in a directory according to their type? I know I can use file to find the type of the file, but I don't know how to use it it the if condition.
What I want to achieve is this. I need to tidy up my downloads folder. When I run the specific script, I want the files in the mentioned folder to be moved into a dedicated folder, according to its type. For eg, image files should be moved to a folder named "Images", video files to "Videos", executables to "Programs" and so on.
Something like this?
for filename in ./*; do
case $(file -b -i "$filename") in
inode/directory* | inode/symlink*)
echo "$0: skip $filename" >&2
continue;;
application/*) dest=Random;;
image/*) dest=Images;;
text/html*) dest=Webpages;;
text/plain*) dest=Documents;;
video/*) dest=Videos;;
*) dest=Unknown;;
esac
mkdir -p "$dest"
mv "$filename" "$dest/"
done
The mapping of MIME types (-i option) to your hierarchy of directories isn't entirely straightforward. The application MIME type hierarchy in particular corresponds to a vast number of document types (PDF, Excel, etc) - some of which also have designated types - as well as the completely unspecified generic application/octet-stream. Using something else than MIME types is often even more problematic, as the labels that file prints are free-form human-readable text which can be essentially random (for example, different versions of the same file format may correspond to different detections with different labels, which are not systematically formatted, and so you might get Evil Empire Insult (tm) format 1997 from one file and Insult 2000 from another with the same extension).
Probably do a test run with file -i ./* and examine the results you get, then update the code above with cases which actually make sense for your specific files.
Related
I am looking for some kind of logic in linux where I can place files with same name in a directory or file system.
For e.g. i create a file abc.txt, so the next time if any process creates abc.txt it should automatically check and make the file named as abc.txt.1 should be created, then next time abc.txt.2 and so on...
Is there a way to achieve this.
Any logic or third party tools are also welcomed.
You ask,
For e.g. i create a file abc.txt, so the next time if any process
creates abc.txt it should automatically check and make the file named
as abc.txt.1 should be created
(emphasis added). To obtain such an effect automatically, for every process, without explicit provision by processes, it would have to be implemented as a feature of the filesystem containing the files. Such filesystems are called versioning filesystems, though typically the details are slightly different from what you describe. Most importantly, however, although such filesystems exist for Linux, none of them are mainstream. To the best of my knowledge, none of the major Linux distributions even offers one as a distribution-supported option.
Although it's a bit dated, see also Linux file versioning?
You might be able to approximate that for many programs via a customized version of the C standard library, but that's not foolproof, and you should not expect it to have universal effect.
It would be an altogether different matter for an individual process to be coded for such behavior. It would need to check for existing files and choose an appropriate name when opening each new file. In doing so, some care needs to be taken to avoid related race conditions, but it can be done. Details would depend on the language in which you are writing.
You can use BASH expression to achieve this. For example if I wanted to make 10 files all with the same name, but having a unique number value I would do the following:
# touch my_file{01..10}.txt
This would create 10 files starting at 01 all the way to 10. This method is also hand for looping over files in a sequence or if your also creating directories.
Now if i am reading you question right your asking that if you move a file or create a file in a directory. you would want the a script to automatically create a new file for you? If that is the case then just use a test and if there is a file move that file and mark it. Me personally I use time stamps to do so.
Logic:
# The [ -f ] tests if the file is present
if [ -f $MY_FILE_NAME ]; then
# If the file is present move the file and give it the PID
# That way the name will always be unique
mv $MY_FILE_NAME $MY_FILE_NAME_$$
mv $MY_NEW_FILE .
else
# Move or make the file here
mv $MY_NEW_FILE .
fi
As you can see the logic is very simple. Hope this helps.
Cheers
I don't know about Your particular use case, but You may try to look at logrotate:
https://wiki.archlinux.org/index.php/Logrotate
I have an encrypted database of about 6000 unique passwords, and I want to search about 1TB of data for any instance of these passwords. I am using Cygwin, but I could have the drive available in a real linux environment if I needed to.
I have a file "ClientPasswords.txt" which contains every unique password only once, one password per line. I am trying to compare every file in my T:/ drive to this list.
I am using this command:
grep -nr -F -f ClientPasswords.txt /cygdrive/t 2> SuspectFiles.txt
My goal is to generate a list of all files, "SuspectFiles.txt", that contain any known password in our password database in plaintext so that we can redact sensitive information from the drive.
Currently, it is getting a ton of false positives, including some that don't seem to match anything in the list. I have already eliminated all passwords that are fewer than 6 characters, can be found in the dictionary (or are otherwise known client names), or are just numbers.
I would like to:
Limit it to a select few filetypes (txt, csv, xls, xlsx, doc, docx, etc.)
Eliminate all compressed files (or find a way to search inside them)
Limit snippet output to prevent dumping entire binary files into the output file.
Anyone done something similar, or know of an easier way to search for these improperly documented passwords from a blacklist? I have also played around with the Windows program "Agent Ransack", but it seems much more limited than grep.
Thanks!
First thing you want to do is make a list of all files on drive T that are of the right type, and output that to a list of target file names. Use a shell script:
for i in txt csv xls xlsx doc docx;
do
find /cygdrive/t -name \*.$i >> target_file_names.txt
done
Now that you have the target file names, you can search your passwords amongst those target file names.
for target_file in `cat target_file_names.txt`
do
for pwd in `cat ClientPasswords.txt`
do
grep -l $pwd $target_file > /dev/null
test $? -eq 0 && echo $target_file has password $pwd
done
done
Something like that should work. You might have to tweak it a little.
I'm interested in simply adding a comment next to my files in Linux (Ubuntu). An example would be:
info user ... my_data.csv Raw data which was sent to me.
info user ... my_data_cleaned.csv Raw data with duplicates filtered.
info user ... my_data_top10.csv Cleaned data with only top 10 values selected for each ID.
So sort of the way you can comment commits in Git. I don't particularly care about searching on these tags, filtering them etc. Just seeings them when I list files in a directory. Bonus if the comments/tags follow the document around as I copy or move it.
Most filesystem types support extended attributes where you could store comments.
So for example to create a comment on "foo.file":
xattr -w user.comment "This is a comment" foo.file
The attributes can be copied/moved with the file just be aware that many utilities require special options to copy the extended attributes.
Then to list files with comments use a script or program that grabs the extended attribute. Here is a simple example to use as a starting point, it just lists the files in the current directory:
#!/bin/sh
ls -1 | while read -r FILE; do
comment=`xattr -p user.comment "$FILE" 2>/dev/null`
if [ -n "$comment" ]; then
echo "$FILE Comment: $comment"
else
echo "$FILE"
fi
done
The xattr command is really slow and poorly written (it doesn't even return error status) so I suggest something else if possible. Use setfattr and getfattr in a more complex script than what I have provided. Or maybe a custom ls command that is aware of the user.comment attribute.
This is a moderately serious challenge. Basically, you want to add attributes to files, keep the attributes when the file is copied or moved, and then modify ls to display the values of these attributes.
So, here's how I would attack the problem.
1) Store the information in a sqlLite database. You can probably get away with one table. The table should contain the complete path to the file, and your comment. I'd name the database something like ~/.dirinfo/dirinfo.db. I'd store it in a subfolder, because you may find later on that you need other information in this folder. It'd be nice to use inodes rather than pathnames, but they change too frequently. Still, you might be able to do something where you store both the inode and the pathname, and retrieve by pathname only if the retrieval by inode fails, in which case you'd then update the inode information.
2) write a bash script to create/read/update/delete the comment for a given file.
3) Write another bash function or script that works with ls. I wouldn't call it "ls" though, because you don't want to mess with all the command line options that are available to ls. You're going to be calling ls always as ls -1 in your script, possibly with some sort options, such as -t and/or -r. Anyway, your script will call ls -1 and loop through the output, displaying the file name, and the comment, which you'll look up using the script from 2). You may also want to add file size, but that's up to you.
4) write functions to replace mv and cp (and ln??). These would be wrapper functions that would update the information in your table, and then call the regular Unix versions of these commands, passing along any arguments received by the functions (i.e. "$#"). If you're really paranoid, you'd also do it for things like scp, which can be used (inefficiently) to copy files locally. Still, it's unlikely you'll catch all the possibilities. What if someone else does a mv on your file, who doesn't have the function you have? What if some script moves the file by calling /bin/mv? You can't easily get around these kinds of issues.
Or if you really wanted to get adventurous, you'd write some C/C++ code to do this. It'd be faster, and honestly not all that much more challenging, provided you understand fork() and exec(). I can't recall whether sqlite has a C API. I assume it does. You'd have to tangle with that, too, but since you only have one database, and one table, that shouldn't be too challenging.
You could do it in perl, too, but I'm not sure that it would be that much easier in perl, than in bash. Your actual code isn't that complex, and you're not likely to be doing any crazy regex stuff or string manipulations. There are just lots of small pieces to fit together.
Doing all of this is much more work than should be expected for a person answering a question here, but I've given you the overall design. Implementing it should be relatively easy if you follow the design above and can live with the constraints.
I have the original images in a directory structure that looks like this:
./Alabama/1.jpg
./Alabama/2.jpg
./Alabama/3.jpg
./Alaska/1.jpg
...the rest of the states...
I wanted to convert all of the original images into thumbnails so I can display them on a website. After a bit of digging / experimenting, I came up with the following Linux command:
find . -type f -iname '*.jpg' | sed -e 's/\.jpg$//' | xargs -I Y convert Y.jpg -thumbnail x100\> Y-small.jpg
It recursively finds all the jpg images in my subdirectories, removes the file type (.jpg) from them so I can rename them later, then makes them into a thumbnail and renames them with '-small' appended before the file type.
It worked for my purposes, but its a tad complicated and it isn't very robust. For example, I'm not sure how I would insert 'small-' at the beginning of the file's name (so ./Alabama/small-1.jpg).
Questions:
Is there a better, more robust way of creating thumbnails from images that are located in multiple subdirectories?
Can I make the existing command more robust (for example, but using sed to rename the outputted thumbnail before it is saved- basically modify the Y-small.jpg part).
No need to build up such a complicated construct. Make small blocks first, then connect.
I have decided to not insert -small in the middle of the filename. it makes everything more complicated. Better to use a prefix (simpler code, and also easier to derive the original filename from the thumb) which can either be a thumbsDir/ or a thumbsPrefix-.
#!/bin/sh
make_thumbnail() {
pic=$1
thumb=$(dirname "$1")/thumb-$(basename "$1")
convert "$pic" -thumbnail x100 "$thumb"
}
# Now we need a way to call make_thumbnail on each file.
# The easiest robust way is to restrict on files at level 2, and glob those
# with */*.jpg
# (we could also glob levels 2 and 3 with two globs: */*.jpg */*/*.jpg)
for pic in */*.jpg
do
make_thumbnail "$pic"
done
I've written a vary basic shell script that moves a specified file into the dustbin directory. The script is as follows:
#!/bin/bash
#move items to dustbin directory
mv "$#" ~/dustbin/
echo "File moved to dustbin"
This works fine for me, any file I specify gets moved to the dustbin directory. However, what I would like to do is create a new script that will move the file in the dustbin directory back to its original directory. I know I could easily write a script that would move it back to a location specified by the user, but I would prefer to have one that would move it to its original directory.
Is this possible?
I'm using Mac OS X 10.6.4 and Terminal
You will have to store where the original file is coming from then. Maybe in a seperate file, a database, or in the files attributes (meta-data).
Create a logfile with 2 columns:
The complete filename in the dustbin
The complete original path and filename
You will need this logfile anyway - what will you do when a user deleted 2 files in different directories, but with the same name? /home/user/.wgetrc and /home/user/old/.wgetrc ?
What will you do when a user deletes a file, makes a new one with the same name, and then deletes that too? You'll need versions or timestamps or something.
You need to store the original location somewhere, either in a database or in an extended attribute of the file. A database is definitely the easiest way to do it, though an extended attribute would be more robust. Looking in ~/.Trash/ I see some, but not all files have extended attributes, so I'm not sure how Apple does it.
You need to somehow encode the source directory in the file. I think the easiest would be to change the filename in the dustbin directory. So that /home/user/music/song.mp3 becomes ~/dustbin/song.mp3|home_user_music
And when you copy it back your script needs to process the file name and construct the path beginning at |.
Another approach would be to let the filesystem be your database.
A file moved from /some/directory/somewhere/filename would be moved to ~/dustbin/some/directory/somewhere/filename and you'd do find ~/dustbin -name "$file" to find it based on its basename (from user input). Then you'd just trim "~/bustbin" from the output of find and you'd have the destination ready to use. If more than one file is returned by find, you can list the proposed files for user selection. You could use ~/dustbin/$deletiondate if you wanted to make it possible to roll back to earlier versions.
You could do a cron job that would periodically remove old files and the directories (if empty).