How can we use multithread in PowerShell.
My Query is below:
I have a root folder which has various base folder under it.
I want to sync the folders from server to another using SFTP on remote.
So I want all the folders from the left folder structure to get synced to another server folder using multithreading so that the transfer becomes faster.
I am using WinSCP.net SynchronizeDirectories to sync, but its quite slow.
Please suggest a better way if any one can.
Related
We have a linux server for the database. Most of our data are on /var/. I would like to backup entire directory on external hard drive or on another linux system so that if something goes wrong I can entirely replace the directory. Since the directory has many files, I do not want to copy and paste every time instead I like to sync them.
Is there easy way to do that? rsyn can do that, how do I avoid of login every time the server? BTW I have limited knowledge of linux system.
Appreciated any comments and suggestions.
Bikesh
Rsyncing database files is not a recommended way of backing them up. I believe you have mysql running on the server. In that case, you can take a full database dump in the server, using steps mentioned in following link:
http://www.microhowto.info/howto/dump_a_complete_mysql_database_as_sql.html#idp134080
And, then syncing these files to your backup server. You can use rsych command for this purpose:
http://www.tecmint.com/rsync-local-remote-file-synchronization-commands/
Make sure that you have installed mysql in the backup server too. You can also copy the mysql configuration file /etc/my.cnf file to the database backup server. In case you require your database to be updated always, you can setup mysql replication. You can follow the below mentioned guide to do the same.
http://kbforlinux.blogspot.in/2011/09/setup-mysql-master-slave-replication.html
I have a website that I host on a Linux VPS which has been growing over the years. One of its primary functions is to store images/photos and these image files are typically around 20-40kB each. The way the site is organised at the moment is all images are stored in a root folder ‘photos’ and under that root folder are many subfolders determined by a random filename. For example, one image could have a file name abcdef1234.jpg and that would be stored in the folder photos/ab/cd/ef/. The advantage of this is that there are no directories with excessive numbers of images in them and accessing files is quick. However, the entire photos directory is huge and is set to grow. I currently have almost half a million photos in tens of thousands of sub-folders and whilst the system works fine, it is fairly cumbersome to back up. I need advice on what I could do to make life easier for back-ups. At the moment, I am backing up the entire photos directory each time and I do that by compressing the folder and downloading it. It takes a while and puts some strain on the server. I do this because every FTP client I use takes ages to sift through all the files and find the most recent ones by date. Also, I would like to be able to restore the entire photo set quickly in the event of a catastrophic webserver failure so even if I could back up the data recursively, how cumbersome would it be to have to upload each back stage by stage?
Does anyone have any suggestions perhaps from experience? I am not a webserver administrator and my experience of Linux is very limited. I have also looked into CDN’s and Amazon S3 but this would require a great deal of change to my site in order to make these system work – perhaps I’ll use something like this in the future.
Since you indicated that you run a VPS, I assume you have shell access which gives you substantially more flexibility (as opposed to a shared webhosting plan where you can only interact with a web frontend and an FTP client). I'm pretty sure that rsync is specifically designed to do what you need to do (sync large numbers of files between machines, and do so efficiently).
This gets into Superuser territory, so you might get more advice over on that forum.
I have multiple websites on a dedicated server running under Linux/Apache. The sites need to access common data from a directory named 'DATA' under the doc root. I cannot replicate this directory for every site. I would like to put this under a common directory (say /DATA) and provide a symbolic link to this directory from the doc root for each of the sites.
www/DATA -> /DATA
Is there a better way of doing this?
If I put this common directory (/DATA) directly under Linux root directory, can there be problems from Linux standpoint as the directory size can be several gigabytes and the sub directories under /DATA will need have write permissions.
Thanks
Use Alias along with the Directory directive. This will allow the site to access the directory via a url path.
I'm not sure what exactly it means that you'll have scripts accessing the directory to provide data. Executing shell scripts to read an produce data is a different story entirely, but you probably want to avoid this if this is what you're doing. Application pages could be included in the data directory and use a relative path to get to the data. Then all sites get the same scripts and data.
I don't know what your data is, but I'd probably opt to put it in a database. Think about how you have to update multiple machines if you have to scale your app. Maybe the data you have is simple and a DB is overkill.
I would like to create a SVN repository remotely using FTP protocol.
Is it advisable to do the following steps
mount the FTP directory as local with culftpfs
create a repository as if it is local with svnadmin create
use it like in everyday life?
Do you know any issue with that approach?
RESULT AFTER MY ATTEMPT
I did try an attempy but I get an errro that looks like a timeout. THe real problem is that this approach is too slow. The solution of copying the repository each time looks more feasable or a simple script to back-up the folder.
It is a dangerous approach, however if you are working alone(as in "single user"), it would work. The biggest problems are:
You cannot provide exclusive locking mechanics over network
All Users will have direct access to all repositorie's internal files, if somebody deletes a file in revs, your repository is damaged beyond repair
You should setup an apache with
SVNAutoversioning on
then you could mount your repoURL as WebDav folder. Each change on these files will result in a single commit without need of a workingcopy
I'm developing on my local machine (apache2, php, mysql). When I want to upload files to my live server (nginx, mysql, php5-fpm), I first backup my www folder, extract the databases, scp everything to my server (which is tedious, because it's protected with opiekey), log myself in, copy the files from my home directory on the server to my www directory and if I'm lucky and the file permissions and everything else works out, I can view the changes online. If I'm unlucky I'll have to research what went wrong.
Today, I changed only one file, and had to go through the entire process just for this file. You can imagine how annoying that is. Is there a faster way to do this? A way to automate it all? Maybe something like "commit" in SVN and off you fly?
How do you guys handle these types of things?
PS: I'm very very new to all this, so bear with me! For example I'm always copying files into my home directory on the server, because scp cannot seem to copy them directly into the /var/www folder?!
There are many utilities which will do that for you. If you know python, try fabric. If you know ruby, you may prefer capistrano. They allow you to script both local and remote operations.
If you have a farm of servers to take care of, those two might not work at the scale you want. For over 10 servers, have a look at chef or puppet to manage your servers completely.
Whether you deploy from local checkout, packaged source (my preferred solution), remote repository, or something entirely different is up to you. Whatever works for you is ok. Just make sure your deployments are reproducible (that is you can always say "5 minutes ago it wasn't broken, I want to have what now what I had 5 minutes ago"). Whatever way of versioning you use is better than no versioning (tagged releases are probably the most comfortable).
I think the "SVN" approach is very close to what you really want. You make a cron job that will run "svn update" every few minutes (or hg pull -u if using mercurial, similar with git). Another option is to use dropbox (we use it for our web servers sometimes) - this one is very easy to setyp and share with non-developers (like UI designers)...
rsync will send only the changes between your local machine and the remote machine. It would be an alternative to scp. You can look into how to set it up to do what you need.
You can't copy to /var/www because the credentials you're using to log in for the copy session doesn't have access to write on /var/www. Assuming you have root access, change the group (chown) on /var/www (or better yet, a sub directory) to your group and change the permissions to allow your group write access (chmod g+w).
rsync is fairly lightweight, so it should be simple to get going.