Is BitTorrent Sync any faster than Dropbox if you only use 2 computers? Does the speed increase if you have more computers to share the file with (faster than a centralized syncing system such as Dropbox)?
P2P sync tools are synchronous. Systems relying on a server support asynchronous transfer. So they will have different performance characteristics under different conditions.
So if the two clients are never online at the same time their effective transfer speed would be zero in a P2P setting.
On the other hand the server in a centralized solution might throttle transfers and has to traverse the internet where a P2P solution might be able to exploit a local network if they're on the same LAN.
Related
I am doing research in parallel computing side in my master. we are creating a real TriBA-Network a Triple based architecture (for multi core processor network). For this i am working on network part. So i have to implement routing in this network on layer 2 level. i have done routing on layer 3 (network layer) using TCP\IP protocol. But we have to send/received frames (not packets) on layer 2.
Maybe i can use RAW socket for sending frames using network programming. but how we can received these frames in remote PC and forward.
If somebody know about lower level communication that i can use for this task kindly share here.
Advance Thank you.
Computer's are connected in that topology over LAN locally TriBA Topology Network
From the question it's not that clear why this is to be done in kernel.
However, I understand from the question that some lower-layer approach is needed. If there are some explicit latency or throughput requirements then it needs to be stated in the question.
So, if we postpone the hypothetical in-kernel approach, indeed, you may use PF_PACKET sockets to send them. The same way, using PF_PACKET sockets you may receive them at the remote side. However, typically, this still involves certain overhead since the kernel copies data between its own buffers and the socket buffers accessed by the user in the userland.
From this perspective you may consider PACKET_MMAP technique to set up direct memory mapping to access data from kernel-side buffers. This might be a really good improvement in terms of, say, throughput, but it's rather controversial whether this will fit your need of forwarding.
Another one suggestion is to use some sort of like data plane kit (please find the link on that page) which may have a set of libraries designed specially for kernel-bypass networking (similar to PACKET_MMAP, but much better) and for certain tasks like L2 forwarding.
All in all, it is likely that you have enough tools to do what you need, however, a more detailed description would be better for understanding.
hi can any one tell me how to write code to send files from one linux pc to another linux pc using udp transfer protocol? i had the code for transfering strings from one pc to another pc i want to know how to transfer files . i would be thankful if anyone helps me
thank you so much in advance..
This is a nontrivial, and also pointless exercise, as to do it efficiently, you end up reinventing TCP.
For an "easy" way of doing it, try implementing TFTP. TFTP is a very stupid protocol where each frame is individually acknowledged and the sender waits for the response before sending the next frame. This is not a good way of doing file transfer and is much worse than TCP.
TFTP I think was designed to be implemented in the minimum amount of code (rather than being efficient). It has historically been used by network boot loaders, which must sometimes live in a tiny rom (a historical limitation).
Depending on the latency and bandwidth of your network, ack'ing individual frames and waiting for the ack before sending the next one, will result in very poor performance (unless your network has extremely low latency and low bandwidth, which do not generally go hand-in-hand).
I need to expose certain monitoring statistics from my application and I'm wondering what the most widespread framework or protocol is for doing this?
SNMP is widely used and a standard protocol. It's implemented in computers, routers, hubs, printers and practically anything connected to the net. Although it's called the SImple Network Management Protocol it's not restricted to network management.
It's an open standard and consequently there are a huge array of management/monitoring solutions, from simple shell scripts and libraries up to enterprise monitoring suites (e.g. HP Openview).
You can query synchronously for data or receive events (in SNMP-speak, traps). Each device will report a common set of data (relating mainly to the network status of that device) plus enterprise-specific data (e.g. CPU usage, printer status etc.).
It runs over UDP, and message consistency is a responsibility of the implementing library. This is a little unusual, but it's designed to operate even when the network is not functioning correctly (e.g. flooded with traffic/misconfigured etc.) and decisions about retry strategies, timeouts etc. need to be taken at the application level (unlike TCP).
Has anyone an idea how many tcp-socket connections are possible on a modern standard Linux server?
(There is in general less traffic on each connection, but all the connections have to be up all the time.)
I achieved 1600k concurrent idle socket connections, and at the same time 57k req/s on a Linux desktop (16G RAM, I7 2600 CPU). It's a single thread http server written in C with epoll. Source code is on github, a blog here.
Edit:
I did 600k concurrent HTTP connections (client & server) on both the same computer, with JAVA/Clojure . detail info post, HN discussion: http://news.ycombinator.com/item?id=5127251
The cost of a connection(with epoll):
application need some RAM per connection
TCP buffer 2 * 4k ~ 10k, or more
epoll need some memory for a file descriptor, from epoll(7)
Each registered file descriptor costs roughly 90
bytes on a 32-bit kernel, and roughly 160 bytes on a 64-bit kernel.
This depends not only on the operating system in question, but also on configuration, potentially real-time configuration.
For Linux:
cat /proc/sys/fs/file-max
will show the current maximum number of file descriptors total allowed to be opened simultaneously. Check out http://www.cs.uwaterloo.ca/~brecht/servers/openfiles.html
A limit on the number of open sockets is configurable in the /proc file system
cat /proc/sys/fs/file-max
Max for incoming connections in the OS defined by integer limits.
Linux itself allows billions of open sockets.
To use the sockets you need an application listening, e.g. a web server, and that will use a certain amount of RAM per socket.
RAM and CPU will introduce the real limits. (modern 2017, think millions not billions)
1 millions is possible, not easy. Expect to use X Gigabytes of RAM to manage 1 million sockets.
Outgoing TCP connections are limited by port numbers ~65000 per IP. You can have multiple IP addresses, but not unlimited IP addresses.
This is a limit in TCP not Linux.
10,000? 70,000? is that all :)
FreeBSD is probably the server you want, Here's a little blog post about tuning it to handle 100,000 connections, its has had some interesting features like zero-copy sockets for some time now, along with kqueue to act as a completion port mechanism.
Solaris can handle 100,000 connections back in the last century!. They say linux would be better
The best description I've come across is this presentation/paper on writing a scalable webserver. He's not afraid to say it like it is :)
Same for software: the cretins on the
application layer forced great
innovations on the OS layer. Because
Lotus Notes keeps one TCP connection
per client open, IBM contributed major
optimizations for the ”one process,
100.000 open connections” case to Linux
And the O(1) scheduler was originally
created to score well on some
irrelevant Java benchmark. The bottom
line is that this bloat benefits all of
us.
On Linux you should be looking at using epoll for async I/O. It might also be worth fine-tuning socket-buffers to not waste too much kernel space per connection.
I would guess that you should be able to reach 100k connections on a reasonable machine.
depends on the application. if there is only a few packages from each client, 100K is very easy for linux. A engineer of my team had done a test years ago, the result shows : when there is no package from client after connection established, linux epoll can watch 400k fd for readablity at cpu usage level under 50%.
Which operating system?
For windows machines, if you're writing a server to scale well, and therefore using I/O Completion Ports and async I/O, then the main limitation is the amount of non-paged pool that you're using for each active connection. This translates directly into a limit based on the amount of memory that your machine has installed (non-paged pool is a finite, fixed size amount that is based on the total memory installed).
For connections that don't see much traffic you can reduce make them more efficient by posting 'zero byte reads' which don't use non-paged pool and don't affect the locked pages limit (another potentially limited resource that may prevent you having lots of socket connections open).
Apart from that, well, you will need to profile but I've managed to get more than 70,000 concurrent connections on a modestly specified (760MB memory) server; see here http://www.lenholgate.com/blog/2005/11/windows-tcpip-server-performance.html for more details.
Obviously if you're using a less efficient architecture such as 'thread per connection' or 'select' then you should expect to achieve less impressive figures; but, IMHO, there's simply no reason to select such architectures for windows socket servers.
Edit: see here http://blogs.technet.com/markrussinovich/archive/2009/03/26/3211216.aspx; the way that the amount of non-paged pool is calculated has changed in Vista and Server 2008 and there's now much more available.
Realistically for an application, more then 4000-5000 open sockets on a single machine becomes impractical. Just checking for activity on all the sockets and managing them starts to become a performance issue - especially in real-time environments.
We have an embedded device that needs to interact with an enterprise software system.
The enterprise system currently uses many different mechanisms for communication between its components: ODBC, RPC, proprietary protocol over TCP/IP, and is moving to .Net-implmented web services.
The embedded device runs a flavor of *nix, so we're looking at what the best interaction mechanism is.
The requirements for the communication are:
Must run over TCP/IP.
Must also run over RS-232 or USB.
Must be secure (e.g. HTTPS or SSL).
Must be capable of transferring ~32MB of data.
Our current best option is gSOAP.
Does anyone out there in SO-land have any other suggestions?
Edit: Steven's answer gave me the most new pointers. Thanks to all!
You can define RESTful services the use HTTPS (which uses TCP/IP by definition) and is capable of transferring any amount of data.
The advantage of REST over SOAP is that REST is simpler. It can use JSON instead of XML which is simpler.
It has less overhead than the SOAP protocol.
Can't you just use SSL over TCP?
If you have some kind of *nix (may I guess? It's either QNX or embedded linux, right?) it should work pretty much out of the box via Ethernet, USB and RS232. Keep thing simple.
32mb is plenty of memory for this task. I would allocate between 2 and 4 mb of memory for networking & encryption (code + data).
It's not real clear why you want to tie this to a remote-procedure-call protocol like SOAP. Are there other requirements you aren't mentioning?
In general, though, this sort of thing is handled very easily using normal web-based services. You can get very lightweight http processors written in C; see this Wikipedia article for comparisons of a number of them. Then a REST interface will work fine. There are network interfaces that treat USB as a TCP connection, as well.
If you must be able to run over RS232, you might want to look elsewhere; in that case, something like sftp might do better. Or write a simple application-layer protocol that you can run over an encrypted connection.
If you are going to connect your application using RS232, I assume that you will be using PPP to connect the device to the internet. The amount of data that you are proposing to transfer is somewhat worrisome, however. Most RS232 connections are limited to 115200 baud which, ignoring the overhead required for TCP/IP/PPP framing is going to yield a transfer rate of at most 11,000 bytes per second. This implies that it will take a minimum of approximately 2800 seconds or 46 minutes to make whatever transfer that you intend.