I am trying to write a bash script to do a task, I have done pretty well so far, and have it working to an extent, but I want to set it up so it's distributable to other people, and will be opening it up as open source, so I want to start doing things the "conventional" way. Unfortunately I'm not all that sure what the conventional way is.
Ideally I want a link to an in depth online resource that discusses this and surrounding topics in depth, but I'm having difficulty finding keywords that will locate this on google.
At the start of my script I set a bunch of global variables that store the names of the dirs that it will be accessing, this means that I can modify the dir's quickly, but this is programming shortcuts, not user shortcuts, I can't tell the users that they have to fiddle with this stuff. Also, I need for individual users' settings not to get wiped out on every upgrade.
Questions:
Name of settings folder: ~/.foo/ -- this is well and good, but how do I keep my working copy and my development copy separate? tweek the reference in the source of the dev version?
If my program needs to maintain and update library of data (gps tracklog data in this case) where should this directory be? the user will need to access some of this data, but it's mostly for internal use. I personally work in cygwin, and I like to keep this data on separate drive, so the path is wierd, I suspect many users could find this. for a default however I'm thinking ~/gpsdata/ -- would this be normal, or should I hard code a system that ask the user at first run where to put it, and stores this in the settings folder? whatever happens I'm going ot have to store the directory reference in a file in the settings folder.
The program needs a data "inbox" that is a folder that the user can dump files, then run the script to process these files. I was thinking ~/gpsdata/in/ ?? though there will always be an option to add a file or folder to the command line to use that as well (it processed files all locations listed, including the "inbox")
Where should the script its self go? it's already smart enough that it can create all of it's ancillary/settings files (once I figure out the "correct" directory) if run with "./foo --setup" I could shove it in /usr/bin/ or /bin or ~/.foo/bin (and add that to the path) what's normal?
I need to store login details for a web service that it will connect to (using curl -u if it matters) plan on including a setting whereby it asks for a username and password every execution, but it currently stores it plane text in a file in ~/.foo/ -- I know, this is not good. The webservice (osm.org) does support oauth, but I have no idea how to get curl to use it -- getting curl to speak to the service in the first place was a hack. Is there a simple way to do a really basic encryption on a file like this to deter idiots armed with notepad?
Sorry for the list of questions, I believe they are closely related enough for a single post. This is all stuff that stabbing at, but would like clarification/confirmation over.
Name of settings folder: ~/.foo/ -- this is well and good, but how do I keep my working copy and my development copy separate?
Have a default of ~/.foo, and an option (for example --config-directory) that you can use to override the default while developing.
If my program needs to maintain and update library of data (gps tracklog data in this case) where should this directory be?
If your script is running under a normal user account, this will have to be somewhere in the user's home directory; elsewhere, you'll have no write permissions. Perhaps ~/.foo/tracklog or something? Again, add a command line option, and also an option in the configuration file, to override this.
I'm not a fan of your ~/gpsdata default; I don't want my home directory cluttered with all sorts of directories that programs created without my consent. You see this happen on Windows a lot, and it's really annoying. (Saved games in My Documents? Get out of here!)
The program needs a data "inbox" that is a folder that the user can dump files, then run the script to process these files. I was thinking ~/gpsdata/in/ ?
As stated above, I'd prefer ~/.foo/inbox. Also with command-line option and configuration file option to change this.
But do you really need an inbox? If the user needs to run the script manually over some files, it might be better just to accept those file names on the command line. They could just be processed wherever, without having to move them to a "magic" location.
Where should the script its self go?
This is usually up to the packaging system of the particular OS you're running on. When installing from source, /usr/local/bin is a sensible default that won't interfere with package managers.
Is there a simple way to do a really basic encryption on a file like this to deter idiots armed with notepad?
Yes, there is. But it's better not to, because it creates a false sense of security. Without a master password or something, secure storage is not possible! Pidgin, for example, explicitly stores passwords in plain text, so that users won't make any false assumptions about their passwords being stored "securely". So it's best just to store them in plain text, complain if the file is world-readable, and add a clear note to the manual to warn the user what's going on.
Bottom line: don't try to reinvent the wheel. There have been thousands of scripts and programs that faced the same issues; most of them ended up adopting the same conventions, and for good reasons. Look at what they do, and mimic them instead of reinventing the wheel.
You can start with the Filesystem Hierarchy Standard. I'm not sure how well followed it is, but it does provide some guidance. In general, I try to use the following:
$HOME/.foo/ is used for user-specific settings - it is hidden
$PREFIX/etc/foo/ is for system-wide configuration
$PREFIX/foo/bin/ is for system-wide binaries
sym-links from $PREFIX/foo/bin are added to $PREFIX/bin/ for ease of use
$PREFIX/foo/var/ is where variable data would live - this is where your input spools and log files would live
$PREFIX should default to /opt/foo even though almost everyone seems to plop stuff in /usr/local by default (thanks GNU!). If someone wants to install the package in their home directory, then substitute $HOME for $PREFIX. At least that is my take on how this should all work.
Related
There are some files like .bashrc or linux config files (cannot remind them right now), which contain lines of information inserted by different software/apps/libraries, e.g. when they were installed.
Is there any way to differentiate information in such files, i.e. to know which lines belong to which software? Maybe there is some convention for that? If yes, where can I learn about it. If no, how do software clean up their configuration and other property in OS files on uninstall?
A note about tags:
I mentioned low-level languages as well as linux and shell because I think only guys from those techs may know the answer.
"Is there any way to differentiate information in such files, i.e. to know which lines belong to which software" - No.
You are dealing with plain text files and there is no way to tell who put what into those files. It may have been a program you installed/ran or it may have simply been a user editing the file in a text editor - you don't know.
You also have no way of knowing whether the contents of the file is well formed or not until you parse it.
If the file does not parse (is messed up) you have two options: 1) read what you can and discard everything else. 2) give up and tell the user the file is garbage
In any case, you cannot trust the information in the file. Anyone could have put it there.
Recently I was asked the following question in an interview.
Suppose I try to create a new file named myfile.txt in the /home/pavan directory.
It should automatically create myfileCopy.txt in the same directory.
A.txt then it automatically creates ACopy.txt,
B.txt then BCopy.txt in the same directory.
How can this be done using a script? I may know that this script should run in crontab.
Please don't use inotify-tools.
Can you explain why you want to do?
Tools like VIM can create a backup copy of a file you're working on automatically. Other tools like Dropbox (which works on Linux, Windows, and Mac) can version files, so it backs up all the copies of the file for the last 30 days.
You could do something by creating aliases to the tools you use for creating these file. You edit a file with the tools you tend to use, and the alias could create a copy before invoking a tool.
Otherwise, your choice is to use crontab to occasionally make backups.
Addendum
let me explain suppose i have directory /home/pavan now i create the file myfile.txt in that directory , immediately now i should automatically generate myfileCopy.txt file in the same folder
paven
There's no easy user tool that could do that. In fact, the way you stated it, it's not clear exactly what you want to do and why. Backups are done for two reasons:
To save an older version of the file in case I need to undo recent changes. In your scenario, I'm simply saving a new unchanged file.
To save a file in case of disaster. I want that file to be located elsewhere: On a different computer, maybe in a different physical location, or at least not on the same disk drive as my current file. In your case, you're making the backup in the same directory.
Tools like VIM can be set to automatically backup a file you're editing. This satisfy reason #1 stated above: To get back an older revision of the file. EMACs could create an infinite series of backups.
Tools like Dropbox create a backup of your file in a different location across the aether. This satisfies reason #2 which will keep the file incase of a disaster. Dropbox also versions files you save which also is reason #1.
Version control tools can also do both, if I remember to commit my changes. They store all changes in my file (reason #1) and can store this on a server in a remote location (reason #2).
I was thinking of crontab, but what would I backup? Backup any file that had been modified (reason #1), but that doesn't make too much sense if I'm storing it in the same directory. All I would have are duplicate copies of files. It would make sense to backup the previous version, but how would I get a simple crontab to know this? Do you want to keep the older version of a file, or only the original copy?
The only real way to do this is at the system level with tools that layer over the disk IO calls. For example, at one location, we used Netapps to create a $HOME/.snapshot directory that contained the way your directory looked every minute for an hour, every hour for a day, and every day for a month. If someone deleted a file or messed it up, there was a good chance that the version of the file exists somewhere in the $HOME/.snapshot directory.
On my Mac, I use a combination of Time Machine - which backs up the entire drive every hour, and gives me a snapshot of my drive that stretches back over a year and a half) and Dropbox which keeps my files stored in the main Dropbox server somewhere. I've been saved many times by that combination.
I now understand that this was an interview question. I'm not sure what was the position. Did the questioner want you to come up with a system wide way of implementing this, like a network tech position, or was this one of those brain leaks that someone comes up with at the spur of the moment when they interview someone, but were too drunk the night before to go over what they should really ask the applicant?
Did they want a whole discussion on what backups are for, and why backing up a file immediately upon creation in the same directory is a stupid idea non-optimal solution, or were they attempting to solve an issue that came up, but aren't technical enough to understand the real issue?
Hi I'm sure there is some way of doing what I want, but maybe I'm just attacking it the wrong way. Hope someone can help.
I have a dev box that I SSH in to from several other machines. In order to debug remotely I need to configure my debugger with my client machine's IP, which changes when I log in from different machines. I'm getting bored of doing this manually all the time so thought I'd try and automate it.
I'm creating a script that is automatically run upon SSH connection that will modify a configuration setting in a PHP ini file. The problem is the PHP ini files are all owned by root so I'm not sure how to modify those files if I'm just logging in as a normal user.
There's not really a security concern with my dev box so I could just change the owner of the ini file, but I wanted it to be more automated than that.
My current attempt is a python script located in my home dir, which is called from .bashrc when I connect via SSH. I don't see how I can gain root privileges from there, I am pretty new to linux though. I thought maybe there would be some other method I'm not aware of.
You have a file that is owned by root. You clearly need to either find a way to mark the file as modifiable by you; or a way for you to elevate your privileges so that you are allowed to modify it.
This leads to the two traditional unix approachs to doing this. They are:
To create a group with which to mark the file, ie. initdebug; chgrp/chmod the file so it has the initdebug group and is group writable; and, add yourself to the initdebug group so you can use the group write permission to modify the file.
To create a very small, audited binary executable (this won't work with a script) that will perform the specific modifications you desire (for simplicity I would suggest copying one of a selection of root owned PHP ini files into the right place). Then chown'ing the file so it is owned by root, and setting the suid bit on the executable so it will execute as root.
You can also combine the two approaches, either:
Not making yourself a member of the initdebug group or suid on the executable, but rather setting group of the executable to initdebug and setting its sgid bit; or,
Keeping the executable suid root but making it only executable by initdebug and therefore only executable by users added to that group.
The security trade off is in the ease/risk of privilege escalation should someone hack your account. If there is a stack/heap overflow or similar vulnerability in the executable and it is executing as root, then you are gone. If the PHP ini file can be modified to open a remote-vulnerability then if they can directly access the ini file you are gone.
As I suspect the latter is possible, you are probably best off with a small executable.
Note: As I alluded to above, unix does not acknowledge the s[ug]id bits on scripts (defined as anything using the #!... interpreter syntax). Hence, you will have to use something you can compile down to a binary. So that probably means either C, C++, Java(using gcj), ML, Scheme(mit), Haskell(ghc).
If you haven't done any C or C++ programming before, I would recommend one of the others as a suid binary is not a project with which to learn C/C++. If you don't know any of the other languages, I would recommend either ML or Java as the easiest to to write something small and simple.
(btw, http://en.wikipedia.org/wiki/List_of_compilers includes a list of alternative compilers you can use. Just make sure it compiles to native, not bytecode. As far as the OS is concerned a bytecode vm is just another interpreter).
you can do it with insert your user to sudoers file on mechine that you want to remote,
for the example you can see my blog.
this is the url : http://nanamo3lyana.blogspot.com/2012/06/give-priviledge-normal-user-as-root.html
and then on your automaticly script add sudo on your command.
i'm sorry my english not good.
I'm developing a website, and want to create a directory after the user username, what is going to be the email address (so I don’t have to generate new ɪᴅs, etc)
I've made some tests and it seems to work fine. Also, I didn’t find any documentation against using the "#" in a directory, but could I find some problem in the future with this approach?
I mean, might some browser not be able to upload images from this directory, or some other problem?
if you plan to run perl scripts (and possibly other languages) against those files you will need to remember to escape the # sign. It's not a huge problem, but I personally would not do it.
More importantly if the path is visible to the browser you would be disclosing the user's email address to the whole world.
I would suggest using something like an MD5 hash of the user's email instead. It is (relatively) unique, and you can recalculate it very easily if you need to. Gravatar uses this approach for instance. See: http://en.gravatar.com/site/implement/hash/
no.. there should be no problems.. browsers are trying to read the file and they don't care that much about the title only file content... (header matters)
So.. there should be no problem...
Historically some remote filesystems have used the # to "escape" from normal path processing to do "interesting" stuff.
Some version control systems use # to denote a certain version of a path (e.g. Subversion, ClearCase).
Some other tools use # to denote "user#remote_host" stuff - AFAIK rsync is one of them which might bite you - you should check if that tool is used somewhere on your site for backup or syncing or something like that.
So - I would not use that character within filenames.
My team lead just added a lot of binary files that shouldn't be in source control. I have to pick and choose my battles with him and this isn't one I think is worth bringing up, but I'd like to just ignore these files on my machine without affecting everyone elses. Is this possible?
We're using TortoiseSvn. I've honestly never used the command line so until I learn how to do that I would prefer a solution using the GUI. Thanks!
If all your files resides in a special directory, you could simply use the Add to ignore list from the shell-context menu.
From the settings/general tab you can also add global ignore patterns, based on extension.