Our goal is to download about 5 millions of tiny files from AWS to CentOS server. We found s3cmd utility and it was very good for almost all things. Except downloading, because it is only supports one thread :( which follow to 60 days of downloading, which is just crazy!
Is there any new version of s3cmd or another way to download all files in multithread mode?
I have open s4cmd for myself, awesome tool. And yes, it supports multithreading.
Related
I'm looking for the most suitable tool to transfer 600 GB of media from a Linux server to s3, so far I found s3 sync and s3cmd , but they do not work in background mode, tell me the best option?
You can run your command in tmux, or nohup. This way the AWS CLI command will persist after you logout. There are other ways, but I personally find tmux being my preferred choice.
I´m running a Raspberry Pi Model B (512MB RAM) with a 16 gB 300MB/s SD-card and recent raspbian with all updates.
On this machine I´ve set up a apache2-server, node.js with socket.io and firmata.
Within my web-application, video streaming is a key feature.
When I access my webserver just for streaming the videos (without node/socket.io/firmata), everything streams with a good performance. But when I switch on node.js/socket.io/firmata it's rather slow, it takes 5-7 seconds to start streaming the videos.
I had problems installing node.js in the first place. Node.js from source compiled/installed like a charm, but when I tried to run it, I got this mysterious "Illegal instruction" message.
As an alternative I took the precompiled debian-packages and installed them using dpkg using this repo:
http://revryl.com/2014/01/04/nodejs-raspberry-pi/
They say that nodejs will run slower, but that´s not acceptable for me.
Any hints?
Thanks and regards!
Allright, it´s faster now.
For everyone with this issue:
Dispose apache2 and use lighttpd instead. Just check out this page and see why: http://www.jeremymorgan.com/blog/programming/raspberry-pi-web-server-comparison/
Start node.js via script
and put into /etc/rc.local Out of some reason it uses much less RAM
and CPU when in idle.
Try to avoid firmata. If you need to control
hardware that requires simple wiring, try to use the "pi-gpio". It´s
MUCH faster and uses less resources. Also you don´t need your arduino
anymore as you can use only the rpi.
I'd like to start using my localhost to develop from. I am trying to work out the best way to sync my local folder with the files directory on the remote web server. In some cases there will be 10,000+ files.
This is not for component files such as php, css, javascript etc. This is for content and media files which I do not wish to use git/svn for.
Thanks
I would recommend using rsync. It's built specifically for remote synchronization tasks. Take a look at the compression and differential modes.
There is always rsync
http://linux.about.com/library/cmd/blcmdl1_rsync.htm
It is built for exactly this kind of task, and optimized for syncing when there are small deltas to large file sets.
if rsync is out of the question you can try wget, it has some nifty fetuers.
I want to create mirrors for my website and I want to know if there is a way to upload my website on different servers at the same time ( through filezilla, wget or other tools ).
Look at using rsync
Mirroring with rsych
When you deploy code you need to make sure that your code changes land on all your servers, one way to do this would be to use a tool like Capistrano if you're inclined to ruby or Fabric or Puppet if you prefer python.
The other way is to deploy onto one server and periodically mirror them over to the other servers using rsync as Paul suggested
We develop Linux-based networking application which will run on multiple servers. We need to develop some solution for remote application update.
All I can think of now is using rpm/deb packages but we prefer not to lock this to some distro-specific solution. Besides copying files via SSH by some Bash script what would you recommend?
Thanks.
Distros does vary so much in setup and dependencies, I would actually recommend you create distro specific packages and integrate with its update tool - in the end it normally saves you a ton of trouble.
With the ease of virtualization, it's rather easy to spin up a vmware/virtualbox image foor the various distros to create/test packaging for each of them
How about puppet?
Check out Blueprit and Blueprint I/O. Blueprint is a tool that detects all of the packages, files modifications and source installs on a server. It packages them up in a reusable format called a blueprint that can be applied to another server. Blueprint I/O is a tools for pushing to and pulling from another server. Both are open-source. Hope this helps.
https://github.com/devstructure/blueprint (Blueprint # Github)
https://github.com/devstructure/blueprint-io (Blueprint I/O # Github)
I'm eight years late, but check Ansible.
Ansible is a radically simple IT automation platform that makes your
applications and systems easier to deploy. Avoid writing scripts or
custom code to deploy and update your applications— automate in a
language that approaches plain English, using SSH, with no agents to
install on remote systems.
Also, you can check this guide.