file copy from one to 100 servers [closed] - linux

Question

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking for code must demonstrate a minimal understanding of the problem being solved. Include attempted solutions, why they didn't work, and the expected results. See also: Stack Overflow question checklist
Closed 9 years ago.
Improve this question
I want to transfer a 100 GB file that resides on a single server to 100 other servers in the network over 1 Gbps line. What is the best way to do it ? My solution is copy the file to k number of servers(say 9) and then assign the remaining (100-9) servers to each of the of the 9 servers.
This is a way better solution then copying the file from 1 server to 100 sequentially. My question is how to determine k ? or what is the calculation to determine the most efficient value of k. Please suggest if there is any better solution too. sorry forgot to mention .. CANNOT USE TORRENT. not all companies allow for torrent. This is an interview question. Appreciate your response. Thanks

Answer 1

Assuming that you can copy to only one server at a time, it can go as follows.
Main server copies to Server S1.
S1 copies to S2 (1 copy)
S1 copies to S3 and S2 copies to S4 (2 copies in parallel)
S1 copies to S5, S2 copies to S6, S3 copies to S7, S4 copies to S8 (4 copies in parallel)
And so on..
So, the pattern of the number of copies is as follows: 2 pow 0, 2 pow 1, 2 pow 2 etc
1 + 2 + 4 + 8 + 16 + 32 + 64 > 100
So, the number of copies S1 has to do can be found with this formula
(2 pow k >= 100) and (2 pow (k-1) < 100)
In this case, k evaluates to 7 (After the first copy)

Answer 2

Let there be n servers to which the files to be copied. Your approach is correct if copying can be done in parallel, i.e. after the initial round of copying there will be k servers with a copy of the file. If copying from these k servers to the remaining n-k servers can be done in parallel then your approach is ideal.
You can find the value of k as follows,
Select k such that k2 ≤ n and (k+1)2 > n.

Answer 3

One opinion is to multicast file on a network. This way first server will only send file once(and other servers receive the file all simultaneously). It can get really tricky, but I imagine this would be the fastest way. You probably need to devise your own custom protocol, what to do when one computer loses packet.
https://github.com/windsurfer/C-Sharp-Multicast

Answer 4

I know for the interview it may be too late, but for the record perhaps
you could consider something like this:
https://code.google.com/p/castcopy/
or some other multicast copy tool. No need to repeat the packets for each
or some of the receiving clients. You just send one copy of the packet and
all others listen at the same time!
Pan

Answer 5

If you use bittorrent to distribute the file over your lan then the torrent software will take care of load-balancing for you i.e. you don't need to precompute "k." I recommend using utorrent for your clients, but any client will do. Here is a tutorial for setting up the tracker etc
An advantage of using bittorrent is that the recipient servers can start distributing chunks of the file before they have the entire file.

Answer 6

Under simplistic assumptions you could treat this as a dynamic programming problem: for i = 1.. k find the fastest way to produce k copies. At each step consider the time taken to produce k-t copies in previous steps and then add on 1 step to run t copy operations in parallel, where t had better be no larger than k - t.
For the case where k is a power of two, you can produce 2 copies (counting the original) in 1 step, 4 copies in 2 steps... 128 copies in 7 steps, which is quicker than it would take to do the 9 copies that are your first stage, assuming that running 9 copies out of a single machine takes 9 times as long as copying to a single destination.
But all of this assumes that the time taken by a copy depends only on the outgoing bandwidth of the source - in practice I would expect that either all your network links are close together and the same, so that multiple copies at the same time risk slowing each other down, or your network links are widely separated but different, so that copies over different links take different amounts of time.
You should also consider sneakernet - copy to removable USB or removable hard drive and carry the device to its destination for another local copy. Historically, attempting to replace relatives of sneakernet with network links without working out the effective bandwidth of the existing sneakernet have failed by not providing enough network bandwidth.

Answer 7

I can think of Divide and Conquer
100 (50,50) -> (25 , 25) -> (12 , 13) -> (6 , 6) -> (3 ,3) -> (1 , 2) ..STOP
I am assuming the copy function will try to use the local resource (e.g Server 1 to Server 2) Server 1 resource will be used.
So from Server 1 to Server 2 and 3 (total 3 servers)
Now Server 1 to 4 , 2 to 5 , 3 to 6 (total 6 Servers)
Now Server 1 to 7 , 2 to 8 , 3 to 9....6 to 12 (total 12 Servers)
So let's Say a thread manager will copy Server 1 to Server 51 , Server 2 to Server 52 ... Server 50 to Server 100

Answer 8

bzip the file to compress it as much as possible
rsync it to all the other machines
Go for lunch/ Work on the next thing in your stack.
No time limit was mentioned so why assume one. It just makes things harder for yourself.

Answer 9

Two steps:
S00 (server one, the guy having the file initially) splits the file in 100 chunks, not saving the chunks to disk, but instead sending chunks C01-C99 to S01-S99 respectively.
S00-S99 sends their chunk to their siblings, but of cause nobody sends to S00
Expect network to be saturated pretty badly!