On Ubuntu, Is there a way to automatically move files to another directory as they are SFTP'd? - linux

I am SFTP'ing files to a directory on my ubuntu server. Ideally I would like these files in my apache public html folder as they are pictures that a user is uploading.
I've found that I can't simply SFTP the files directly to my public html folder, so am researching other methods. My picture server is ubuntu, so I thought there may some native command or setting that I could use to automatically move pictures that show up in my SFTP directory, to my public html directory.
Hopefully I am making sense, and I'm not sure where else I should be asking this question.


Three possibilities:
Why can you not simply upload the files directly in your public html folder? I assume that has something to do with access restrictions in writing to that directory, so you could try to change this directories write permissions for the user you are uploading as.
The access restrictions are changed with the command chmod, the ownership of diles and directories is changed with chown. Best you read documentation to these commands ("man chmod" and "man chown").
You can run a script periodically that takes all uploaded files and moves them to the specified target dir. For this you need to write a short shell script in bash, for example:
#!/bin/bash
mv /home/user/UPLOADS/*.jpg /var/www/images/
(This script takes simply all files with the extension .jpg from directory /home/user/UPLOADS and puts them without further check to the directory /var/wwww/images )
Place this script somewhere (eg. /home/user/bin/) and make it executable: chmod a+x /home/user/bin/SCRIPTNAME
This script can be run periodically via cron, call crontab -e and write a new line
like so:
*/5 * * * * /home/user/bin/SCRIPTNAME
that executes the script every 5 minutes.
Drawback is that it is called every 5 minutes, so there might be a gap between upload and move of max 5 minutes. additionally, if the script runs WHILE uploading of new images, something strange might happen...
The 3rd possibility is to execute a script as soon as the upload is finished by watching the upload directory with the inotify feature of the kernel. If you want to do this, best google for inotify examples, that is a little bit more complicated. Here is another SO answer to that:
Monitor Directory for Changes

Related

Linux bash to compare two files but the second file must be find

I have a batch that integrates an xml file time to time but could happens daily. After it integrates it puts in a folder like /archives/YYMMDD(current day). The problem is if the same file is integrated twice. So I need a script what verifys the file (with diff command its possible but risky to make a bottleneck) but the problem is I can't find to resolve how to make to give the second files location.
P.S. I can't install on the server anything.
Thanks in advance.

PyCharm project path different from interactive session path

When running an interactive session, PyCharm thinks of os.getcwd() as my project's directory. However, when I run my script from the command line, PyCharm thinks of os.getcwd() as the directory of the script.
Is there a good workaround for this? Here is what I tried and did not like:
going to Run/Edit Configurations and changing the working directory manually. I did not like this solution, because I will have to do it for every script that I run.
having one line in my code that "fixes" the path for the purposes of interactive sessions and commenting it out before running from command line. This works, but feels wrong.
Is there a way to do this or is it just the way it is supposed to be? Maybe I shouldn't be trying to run random scripts within my project?
Any insight would be greatly appreciated.
Clarification:
By "interactive session" I mean being able to run each line individually in a Python/IPython Console
By "running from command line" I mean creating a script my_script.py and running python path_to_myscript/my_script.py (I actually press the Run button at PyCharm, but I think it's the same).
Other facts that might prove worth mentioning:
I have created a PyCharm project. This contains (among other things) the package Graphs, which contains the module Graph and some .txt files. When I do something within my Graph module (e.g. read a graph from a file), I like to test that things worked as expected. I do this by running a selection of lines (interactively). To read a .txt file, I have to go (using os.path.join()) from the current working directory (the project directory, ...\\project_name) to the module's directory ...\\project_name\\Graphs, where the file is located. However, when I run the whole script via the command line, the command reading the .txt file raises an Error, complaining that no file was found. By looking on the name of the file that was not found, I see that the full file name is something like this:
...\\project_name\\Graphs\\Graphs\\graph1.txt
It seems that this time the current working directory is ...\\project_name\\Graphs\\, and my os.path.join() command actually spoils it.

I user various methods in my python scripts.
set the working directory as first step of your code using os.chdir(some_existing_path)
This would mean all your other paths should be referenced to this, as you hard set the path. You just need to make sure it works from any location and your specifically in your IDE. Obviously, another os.chdir() would change the working directory and os.getcwd() would return the new working directory
set the working directory to __file__ by using os.chdir(os.path.dirname(__file__))
This is actually what I use most, as it is quite reliable, and then I reference all further paths or file operations to this. Or you can simply refer to as os.path.dirname(__file__) in your code without actually changing the working directory
get the working directory using os.getcwd()
And reference all path and file operations to this, knowing it will change based on how the script is launched. Note: do NOT assume that this returns the location of your script, it returns the working directory of the shell !!
[EDIT based on new information]
By "interactive session" I mean being able to run each line
individually in a Python/IPython Console
By running interactively line-by-line in a Python console, the __file__ is not defined, afterall: you are not executing a file. Hence you cannot use os.path.dirname(__file__) you will have to use something like os.chdir(some_known_existing_dir) to reference a path. As a programmer you need to be very aware of working directory and changes to this, your code should reflect that.
By "running from command line" I mean creating a script my_script.py
and running python path_to_myscript/my_script.py (I actually press the
Run button at PyCharm, but I think it's the same).
This, both executing a .py from command line as well as running in your IDE, will populate the __file__, hence you can use os.path.dirname(__file__)
HTH

I am purposely adding another answer to this post, in regards the following:
Other facts that might prove worth mentioning:
I have created a PyCharm project. This contains (among other things)
the package Graphs, which contains the module Graph and some .txt
files. When I do something within my Graph module (e.g. read a graph
from a file), I like to test that things worked as expected. I do this
by running a selection of lines (interactively). To read a .txt file,
I have to go (using os.path.join()) from the current working directory
(the project directory, ...\project_name) to the module's directory
...\project_name\Graphs, where the file is located. However, when I
run the whole script via the command line, the command reading the
.txt file raises an Error, complaining that no file was found. By
looking on the name of the file that was not found, I see that the
full file name is something like this:
...\project_name\Graphs\Graphs\graph1.txt It seems that this time
the current working directory is ...\project_name\Graphs\, and my
os.path.join() command actually spoils it.
I strongly believe that if a python script takes input from any file, that the author of the script needs to cater for this in the script.
What I mean is you as the author need to make sure you know the following regardless of how your script is executed:
What is the working directory
What is the script directory
These two you have no control over when you hand off your script to others, or run it on other peoples machines. The working directory is dependent on how the script is launched. It seems that you run on Windows, so here is an example:
C:\> c:\python\python your_script.py
The working directory is now C:\ if your_script.py is in C:\
C:\some_dir\another_dir\> c:\python\python.exe c:\your_script_dir\your_script.py
The working directory is now C:\some_dir\another_dir
And the above example may even give different results if the SYSTEM PATH variable is set to the path of the location of your_script.py
You need to ensure that your script works even if the user(s) of your script are placing this in various locations on their machines. Some people (and I don't know why) tend to put everything on the Desktop. You need to ensure your script can cope with this, including any spaces in the path name.
Furthermore, if your script is taking input from a file, the you as the author need to ensure that you can cope with changes in working directory, and changes of script directory. There are a few things you may consider:
Have your script input from a known (static) directory, something like C:\python_input\
Have your script input from a known (configurable) directory, use ConfigParser, you can search here on stackoverflow on many posts
Have your script input from a known directory related to the location of the script (using os.path.dirname(__file__))
any other method you may employ to ensure your script can get to the input
Ultimately this is all in your control, and you need to code to ensure it is working.
HTH,
Edwin.

bash & inotify - monitoring and moving file

I'm not an advanced user of Linux, but I'm looking for some simple script in bash which will be working in cron or any other way and in 5-10 minutes period time will be looking for new files, after new directory/files are already uploaded to the directory script will move new directory with files to other location.
I found inotify which can be a great solution for this, but the issue is how to go with it.

I've been using inotifywait to recognize that some filesystem change ocurred in certain path.
Take a look at:
http://linux.die.net/man/1/inotifywait
You can specify on what changes you are interested (delete, create, modify, etc.) and whether the script should output it or simply exit after change.
I've been using this tool in a way that my script was starting inotifywait and when it exists, do some action and restart inotifywait again.
Hope this helps.
Martin

Handmade continuous integration with shell script

I did a temporary solution for a small continuous integration with shell script.
It updates from svn and then copies files to site's root directory.
So it looks like this
cd ...
update svn
cp -R ... ...
And then put it in crontab.
Well, it works fine for temp. solution, but it would want to make some improvement and to define somehow that svn was changed (new revision appeared) and only in this case to copy files (well, its connected with every-minut copying files makes server work slower).
But im a mean user of linux :(
So the question is:
how to define, using bash script, that svn got new commits and only in this case to make an update and other stuff, like copying files.

You can do 'svn info' in the directory (and use awk|grep|your favorite tool) to extract the revision number of what you've checked out. Do the same to the location you copy to. If the revision number in the checkout directory is higher than the one in the destination directory, then do the copy.
That's assuming that you copy everything including the .svn directories.
If you exclude them, then you should 'svn info' before you update, and again after, and compare the two revisions.

Stop.
Why in the world would you do this?
With eighty totally free, super easy to install, CI tools out there, why in the world would you start hacking your own together with shell scripts and cron?
Unless you're looking to work on your scripting / cron skills and want to use building your own CI as an fun little scenario to play through, you're just wasting your time here.

Copying just files not present with SCP

I need to move my web server directory to another server. I'd like to do it with a simple "scp -r destination:destdirectory". But in the meanwhile the directory will be filled with another stuff: so I'll take the old server down the time I need to move the newest file to the new one. How can I do an scp which is gonna write just the differences? So it'll take not much time, and I won't have to take the website down for too long!

Probably not at all, or just with pains. But if you have the possibility to use rsync, just do that. It automatically excludes files that haven't changed, and for changed files, it just transfers the differences.

Resources