As a way to familiarize myself with Rust & networking in general I started writing a very basic telnet chat server. Everything appears to be going well, but right now I end up with blocks of unsafe code, and I wanted to know if there was a better way to do things.
I spawn a task to listen for connections similar to this: Example TCP server written in Rust Once a user connects and I get a TcpStream I put it in a Connection struct. A Connection struct uses channels to communicate with two tasks- one for reading from its TcpStream, and one for writing. The reading task blocks on the TcpStream's read() method and sends any input back to the Connection struct. The writer blocks on a Port's recv() method and writes anything it receives to the TcpStream. In this way, the main loop of the program can simply maintain a vector of Connection structs to check for user input and write to them at will. The issue is that with this implementation the TcpStream must be be shared by both the read & write tasks, and the mutable write() method called whilst the mutable read() method is still blocking in the other task. In 0.8 I did this with an Rc and unsafe_borrow_mut's, but I'd like to do it in a better fashion if I can- the objections of Rust's type system in this case may be completely valid, for all I know. Any comments on overall design would be welcome as well. Thanks.
There is currently no idiomatic way to do this because Rust's IO API currently does not support parallel read or writes.
See this issue for more info: https://github.com/mozilla/rust/issues/11165
There will be a redesign of the TcpStream API to allow such things in one or the other way (see issue) and which will then provide an "idiomatic way" to do this.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 10 months ago.
Improve this question
I have the following problem:
I need to build a high-performing, multi-threaded HTTP server that can process large amounts of data with very low latency.
The overall data set is very very large (10+ GB) but most requests will only require access to a subset of that data. Waiting for DB access will be too slow, the data must be held in memory.
Each web request will only be performing read-operations on the data, however there will be a background worker thread that is responsible for managing updates to the data periodically.
My basic approach:
I've chosen actix web server as it has a good feature set and seems to perform best on the benchmarking I've looked at.
The main idea I have is to load all the data on boot into some shared state, into a data structure that is heavily optimised for the read operations.
Then I want to provide some kind of interface that each request handler can use to query that data and get immutable references to different parts of it depending on what it needs.
This should avoid race-conditions (as there is only the worker-thread that has write-access) as well as avoiding expensive data copying operations.
Architecture A
My original approach was to create this data inside a module:
static mut DATA: ProgramData;
Then expose public methods for accessing it, but after reading enough warnings about static memory I have abandoned that approach.
Architecture B
This is what I have currently working. I create an empty struct like this (where ProgramData is a custom struct) in the program's main function:
struct ProgramDataWrapper {
data_loaded: bool,
data: ProgramData,
}
Then I pass an Arc<RwLock> smart pointer to a DataService (which is responsible for async loading it, and managing data refreshes over time), and another copy of the Arc pointer is the Actix web state.
So this data should persist through the lifetime of the program because the main method always has a reference to it and it should never be dropped.
I have implemented public methods on this struct to enable querying the data and to get back different parts of it depending on the input parameters to the HTTP Request.
I then pass an Arc<RwLock> into the Actix web state so that every handler has read-only access to it and can query the data using the public functions (the internal data is not public).
The route handler does this by dereferencing the Arc then obtaining a read lock from RwLock, then calling some method like is_ready().
So then, for example, I have an endpoint /ready that will return true/false to the load-balancer to communicate the data is in memory and this instance is ready to start receiving requests.
I've noticed though that when the worker thread gets a write-lock on the data structure that no other route handlers can access it as they are blocked and the entire application freezes until the data is updated. This is because the entire ProgramDataWrapper struct is locked, including its public methods.
I think I could get around this by putting the RwLock on the ProgramData object itself, so that while the worker thread is assembling the new data other parts of the data can still get read locks on the ProgramDataWrapper objects and access the public interface.
Then it should be a short amount of time, once the data is ready, to get a write-lock on the data and only copy in the new bits of data then release it immediately.
Architecture C
The other idea I had is to use mpsc channels.
When I create the DataService, it can create a send-receive pair, keep the recv end and pass back the send half to the main method. This can then clone the send channel into the Actix web state, so that every route handler has a way to send data to the Data Service.
What I was thinking then, is to create a struct like this:
TwoWayData<T, U> {
query: T,
callback: std::sync::misc::Sender<U>,
}
Then in the route handler, I can create a Send-Receive pair of the above type.
I can send a message to the data service (because I have access to a pointer to the clone of the sender from the top main function), and include as that payload the object to send data back to the route handler.
Something like:
#[get("/stuff")]
pub async fn data_ready(data: web::Data<Arc<Sender<TwoWayData<DataQuery, DataResponse>>>>) -> impl Responder {
let (sx, rx): (Sender<TwoWayData<DataQuery, DataResponse>>, Receiver<TwoWayData<DataQuery, DataResponse>>) = channel();
data.send(TwoWayData {
query: "Get me some data",
callback: sx.clone(),
});
}
Then the data service can just listen to incoming messages, extract the query and process it, and send the result back down the channel it has just received.
My Question
If you're still with me, I really appreciate that.
My questions are this:
Is there a large overhead to the mspc channel that will slow down my program communicating large amounts of data over mspc channels?
Is it even possible to send the callback in the way I want to allow two-way communication? And if not, what is the accepted way of doing this?
I know this is a matter of opinion, but which of these two approaches is a more standard way of solving this type of problem, or does it just come down to personal preference / the technical requirements of the problem?
A. I would disregard static mut entirely since it is unsafe and easy to get wrong. The only way I would consider it is as static DATA: RwLock<ProgramData>, but then it is the same as option B except it is less flexible to testing, discrete data sets, etc.
B. Using an Arc<RwLock> is a very common and understandable pattern and I would consider it my first option when sharing mutable data across threads. It is also a very performant option if you keep your writing critical section small. You may reach for some other concurrent data-structure if its infeasible to clone the whole dataset for each update and in-place updates are long and/or non-trivial. At 10+ GB of data, I'd have to take a good look at your data, access, and update patterns to decide on a "best" course of action. Perhaps you can use many smaller locks within your structure, or use a DashMap, or combination thereof. There are many tools available and you may need to make something custom if you're striving for the lowest latency.
C. This looks a bit convoluted but glossing over the specifics is pretty much an "actor model", or at least based on the principles of message passing. If you wanted the data to behave as a separate "service" that can govern itself and provides more control over how the queries are processed then you can use an actor framework like Actix (originally built for Actix-Web but they've since drifted apart enough that there's no longer any meaningful relation). I personally don't use actors since they tend to be an obscuring layer of abstraction, but its up to you. It will likely be slower than accessing the data directly and you'll still need to internally decide on a concurrency mechanism as mentioned above.
I am building a multithreaded async HTTP server on top of mio.
How should I handle events on client TcpStream connections?
For is_readable it's pretty straightforward: I read the request and write the response.
For other kinds I am not sure. Should I shutdown() the stream if I get is_error or is_hup? What about is_none()?
All things that you mention have very precise meaning and map directly to POSIX/BSD Socket API. It's up to you to decide.
is_hup on Read mean the other side hanged-up it's sending side. Meaning it won't send you anything again. However it might have kept the reading open, and you might still want to send some data to it.
shutdown closes Reading/Writing/Both https://doc.rust-lang.org/std/net/enum.Shutdown.html , so it's up to you what and when you want to do.
TcpStream internally holds FileDesc and that will close the fd when you drop it, so if you don't shutdown manually everything will be closed anyway, as soon as you remove given TcpStream from usage. https://github.com/rust-lang/rust/blob/master/src/libstd/sys/unix/fd.rs#L217
So I want to implement a simple comms protocol where reads & writes are completely asynchronous. That means that client sends some data and then server may or may not respond with an answer. So I can't just call reader.read() because that blocks until at least something is returned. And I may have something more to send in the mean time.
So is there a way to check if reader has something to read?
(please note that I'm talking specifically about the streams version: I'm fully aware that protocols version has separate handlers for reading and writing and does not suffer from this issue)
There is no way to ask reader has incoming data or not.
I guess to create asyncio.Task for reading data from asyncio stream reader in loop.
If you need to write data asynchronously feel free to call StreamWriter.write() from any task that have some outgoing data.
I strongly dont recommend to use protocols directly -- they are low-level abstraction useful for flow control but for application code is better to use high-level streams.
is it possible to have multiple threads sending on the same socket? will there be interleaving of the streams or will the socket block on the first thread (assuming tcp)? the majority of opinions i've found seems to warn against doing this for obvious fears of interleaving, but i've also found a few comments that state the opposite. are interleaving fears a carryover from winsock1 and are they well-founded for winsock2? is there a way to setup a winsock2 socket that would allow for lack of local synchronization?
two of the contrary opinions below... who's right?
comment 1
"Winsock 2 implementations should be completely thread safe. Simultaneous reads / writes on different threads should succeed, or fail with WSAEINPROGRESS, depending on the setting of the overlapped flag when the socket is created. Anyway by default, overlapped sockets are created; so you don't have to worry about it. Make sure you don't use NT SP6, if ur on SP6a, you should be ok !"
source
comment 2
"The same DLL doesn't get accessed by multiple processes as of the introduction of Windows 95. Each process gets its own copy of the writable data segment for the DLL. The "all processes share" model was the old Win16 model, which is luckily quite dead and buried by now ;-)"
source
looking forward to your comments!
jim
~edit1~
to clarify what i mean by interleaving. thread 1 sends the msg "Hello" thread 2 sends the msg "world!". recipient receives: "Hwoel lorld!". this assumes both messages were NOT sent in a while loop. is this possible?
I'd really advice against doing this in any case. The send functions might send less than you tell it to for various very legit reasons, and if another thread might enter and try to also send something, you're just messing up your data.
Now, you can certainly write to a socket from several threads, but you've no longer any control over what gets on the wire unless you've proper locking at the application level.
consider sending some data:
WSASend(sock,buf,buflen,&sent,0,0,0:
the sent parameter will hold the no. of bytes actually sent - similar to the return value of the send()function. To send all the data in buf you will have to loop doing a WSASend until all all the data actually get sent.
If, say, the first WSASend sends all but the last 4 bytes, another thread might go and send something while you loop back and try to send the last 4 bytes.
With proper locking to ensure that can't happen, it should e no problem sending from several threads - I wouldn't do it anyway just for the pure hell it will be to debug when something does go wrong.
is it possible to have multiple threads sending on the same socket?
Yes - although, depending on implementation this can be more or less visible. First, I'll clarify where I am coming from:
C# / .Net 3.5
System.Net.Sockets.Socket
The overall visibility (i.e. required management) of threading and the headaches incurred will be directly dependent on how the socket is implemented (synchronously or asynchronously). If you go the synchronous route then you have a lot of work to manually manage connecting, sending, and receiving over multiple threads. I highly recommend that this implementation be avoided. The efforts to correctly and efficiently perform the synchronous methods in a threaded model simply are not worth the comparable efforts to implement the asynchronous methods.
I have implemented an asynchronous Tcp server in less time than it took for me to implement the threaded synchronous version. Async is much easier to debug - and if you are intent on Tcp (my favorite choice) then you really have few worries in lost messages, missing data, or whatever.
will there be interleaving of the streams or will the socket block on the first thread (assuming tcp)?
I had to research interleaved streams (from wiki) to ensure that I was accurate in my understanding of what you are asking. To further understand interleaving and mixed messages, refer to these links on wiki:
Real Time Messaging Protocol
Transmission Control Protocol
Specifically, the power of Tcp is best described in the following section:
Due to network congestion, traffic load balancing, or other unpredictable network behavior, IP packets can be
lost, duplicated, or delivered out of order. TCP detects these problems, requests retransmission of lost
packets, rearranges out-of-order packets, and even helps minimize network congestion to reduce the
occurrence of the other problems. Once the TCP receiver has finally reassembled a perfect copy of the data
originally transmitted, it passes that datagram to the application program. Thus, TCP abstracts the application's
communication from the underlying networking details.
What this means is that interleaved messages will be re-ordered into their respective messages as sent by the sender. It is expected that threading is or would be involved in developing a performance-driven Tcp client/server mechanism - whether through async or sync methods.
In order to keep a socket from blocking, you can set it's Blocking property to false.
I hope this gives you some good information to work with. Heck, I even learned a little bit...
From my understanding the XMPP protocol is based on an always-on connection where you have no, immediate, indication of when an XML message ends.
This means you have to evaluate the stream as it comes. This also means that, probably, you have to deal with asynchronous connections since the socket can block in the middle of an XML message, either due to message length or a connection being slow.
I would appreciate one source per answer so we can mod them up and see what's the favourite.
Are you wanting to deal with multiple connections at once? Good asynch socket processing is a must in that case, to avoid one thread per connection.
Otherwise, you just need an XML parser that can deal with a chunk of bytes at a time. Expat is the canonical example; if you're in Java, try XP. These types of XML parsers will fire events as possible, and buffer partial stanzas until the rest arrives.
Now, to address your assertion that there is no notification when a stanza ends, that's not really true. The important thing is not to process the XML stream as if it is a sequence of documents. Use the following pseudo-code:
stanza = null
while parser has more:
switch on token type:
START_TAG:
elem = create element from parser state
if stanza is not null:
add elem as child of stanza
stanza = elem
END_TAG:
parent = parent of stanza
if parent is not null:
fire OnStanza event
stanza = parent
This approach should work with an event-based or pull parser. It only requires holding on to one pointer worth of state. Obviously, you'll also need to handle attributes, character data, entity references (like & and the like), and special-purpose the stream:stream tag, but this should get you started.
Igniterealtime.org provides an open source XMPP-server and client written in java
ejabberd is written in Erlang. I don't know the details of the ejabberd implementation, but one advantage of using Erlang is really inexpensive threads. I'll speculate they start a thread per XMPP connection. In Erlang terminology these would be called processes, but these are not protected-memory address spaces they are lightweight user-space threads.