Given a server that receives connections from clients, does some work, then after the nth connection is shutdown, it either shuts down itself or starts ignoring connections by doing nothing when the clients connect. I want the thread that established the server to wait for the server to communicate that information (that it has closed its nth connection) to continue its execution.
Expected behaviour:
let t = Thread.create (Unix.establish_server loop) addr in
f1 ();
wait_for_loop_signal ();
f2 ();
I tried doing that with Thread.join in that case I would need to kill/stop the thread t to continue:
let t = Thread.create (Unix.establish_server loop) addr in
f1 ();
Thread.join ();
f2 ();
But it doesn't work, because loop is executed on a separate thread after each connection. So the code of loop is executed on child threads spawned by establish_server. And the thread on which establish_server is executed in not accessible from loop unless there is a way for a thread to kill its parent.
Using Event, by having loop send a message through a channel provided by the initial thread, with Event.sync (Event.send channel ()):
let channel = Event.new_channel () in
let t = Thread.create (Unix.establish_server (loop channel)) addr in
f1 ();
let _ = Event.sync (Event.receive channel) in
f2 ();
But in this case, the execution blocks at let _ = Event.sync (Event.receive channel) in. The send is called correctly and get executed inside loop. But the message that was sent through the channel never gets read by receive. I am not sure why.
The function Unix.establish_server launches a new process and not a thread for each connection. Since the server forks new processes with separate memory, the only way to communicate is trough an inter-process communication mechanism, for instance another socket. Neither Thread.join nor Events can be used meaningfully with establish_server. Similarly, there is no shared memory between the various connections.
In other words, if you want more control on the server, the function establish_server is not the one that you are looking for.
Related
Given a basic setup of a WebSocket server with Actix, how can I launch a daemon inside my message handler?
I've extended the example starter code linked above to call daemon(false, true) using the fork crate.
use actix::{Actor, StreamHandler};
use actix_web::{web, App, Error, HttpRequest, HttpResponse, HttpServer};
use actix_web_actors::ws;
use fork::{daemon, Fork};
/// Define HTTP actor
struct MyWs;
impl Actor for MyWs {
type Context = ws::WebsocketContext<Self>;
}
/// Handler for ws::Message message
impl StreamHandler<Result<ws::Message, ws::ProtocolError>> for MyWs {
fn handle(
&mut self,
msg: Result<ws::Message, ws::ProtocolError>,
ctx: &mut Self::Context,
) {
match msg {
Ok(ws::Message::Ping(msg)) => ctx.pong(&msg),
Ok(ws::Message::Text(text)) => {
println!("text message received");
if let Ok(Fork::Child) = daemon(false, true) {
println!("from daemon: this print but then the websocket crashes!");
};
ctx.text(text)
},
Ok(ws::Message::Binary(bin)) => ctx.binary(bin),
_ => (),
}
}
}
async fn index(req: HttpRequest, stream: web::Payload) -> Result<HttpResponse, Error> {
let resp = ws::start(MyWs {}, &req, stream);
println!("{:?}", resp);
resp
}
#[actix_web::main]
async fn main() -> std::io::Result<()> {
HttpServer::new(|| App::new().route("/ws/", web::get().to(index)))
.bind("127.0.0.1:8080")?
.run()
.await
}
The above code starts the server but when I send it a message, I receive a Panic in Arbiter thread.
text message received
from daemon: this print but then the websocket crashes!
thread 'actix-rt:worker:0' panicked at 'failed to park', /Users/xxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.25/src/runtime/basic_scheduler.rs:158:56
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Panic in Arbiter thread.
The issue with your application is that the actix-web runtime (i.e. Tokio) is multi-threaded. This is a problem because the fork() call (used internaly by daemon()) only replicates the thread that called fork().
Even if your parent process has N threads, your child process will have only 1. If your parent process has any mutexes locked by those threads, their state will be replicated in the child process, but as those threads do not exist there, they will remain locked for forever.
If you have an Rc/Arc it will never de-allocate its memory, because it will never be dropped, thus its internal count will never reach zero. The same applies for any pointers and shared state.
Or said more simply - your forked child will end up in undefined state.
This is best explained in Calling fork() in a Multithreaded Environment:
The fork( ) system call creates an exact duplicate of the address
space from which it is called, resulting in two address spaces
executing the same code. Problems can occur if the forking address
space has multiple threads executing at the time of the fork( ). When
multithreading is a result of library invocation, threads are not
necessarily aware of each other's presence, purpose, actions, and so
on. Suppose that one of the other threads (any thread other than the
one doing the fork( )) has the job of deducting money from your
checking account. Clearly, you do not want this to happen twice as a
result of some other thread's decision to call fork( ).
Because of these types of problems, which in general are problems of
threads modifying persistent state, POSIX defined the behavior of
fork( ) in the presence of threads to propagate only the forking
thread. This solves the problem of improper changes being made to
persistent state. However, it causes other problems, as discussed in
the next paragraph.
In the POSIX model, only the forking thread is propagated. All the
other threads are eliminated without any form of notice; no cancels
are sent and no handlers are run. However, all the other portions of
the address space are cloned, including all the mutex state. If the
other thread has a mutex locked, the mutex will be locked in the child
process, but the lock owner will not exist to unlock it. Therefore,
the resource protected by the lock will be permanently unavailable.
Here you can find a more reputable source with more details
To answer your other question:
"how can I launch a daemon inside my message handler?"
I assume you want to implement the classical unix "fork() on accept()" model.
In that case you are out of luck, because servers such as actix-web, and async/await
in general are not designed with that in mind. Even if you have a
single-threaded async/await server, then:
When a child is forked it inherits all file descriptors from the parent. So it's
common after a fork, the child to close its listening socket in order to avoid a
resource leak - but there is no way to do that on any of the async/await based servers,
not because it's impossible to do, but because it's not implemented.
And even more important reason to do that is to prevent the child process
from accepting new connections - because even if you run a single threaded
server, it's still capable of processing many tasks concurrently - i.e.
when your handler calls .await on something, the acceptor would be free to
accept a new connection (by stealing it from the socket's queue) and start processing it.
Your parent server may have already spawned a lot of tasks and those would be
replicated in each forked child, thus executing the very same thing multiple times,
independently in each process
And well... there is no way to prevent any of that on any of the async/await
based servers I'm familiar with. You would need a custom server that:
Checks in its acceptor task if it's a child and if it detects that it's the child
it should close the listening socket and drop the acceptor.
It should not execute any other task that was forked from the parent,
but there is no way to achieve that.
In other words - async/await and "fork() on accept()" are two different and
incompatible models for processing tasks concurrently.
A possible solution would be to have a non-async acceptor daemon that only
accepts connections and forks itself. Then spawns a web-server in the child
then feeding it the accepted socket. But although possible, none of the servers
currently have support for that.
As described in the other answer, the async runtime you're relying on may completely break if you touch it in the child process. Touching anything can completely break assumptions the actix or tokio devs made. Wacky stuff will happen if you so much as return from the function.
See this response by one of the key authors of tokio to someone doing something similar (calling fork() in the context of a threadpool with hyper):
Threads + fork is bad news... you can fork if you immediately exec and do not allocate memory or perform any other operation that may have been corrupted by the fork.
Going back to your question:
The objective is for my websocket to respond to messages and be able to launch isolated long-running processes that launch successfully and do not exit when the websocket exits.
I don't think you want to manually fork() at all. Utility functions provided by actix/tokio should integrate well with their runtimes. You may:
Run blocking or CPU-heavy code in a dedicated thread with actix_web::block
Spawn a future with actix::AsyncContext::spawn. You would ideally want to use e.g. tokio::process::Command rather than the std version to avoid blocking in an async context.
If all you're doing in the child process is running Command::new() and later Command::spawn(), I'm pretty sure you can just call it directly. There's no need to fork; it does that internally.
My program like this:
func handle(conn net.Conn) {
msg := "hello, world!"
for i:= 0; i< 100000; i++ {
go func() {
err := write(conn, msg)
}
}
}
func write(conn net.Conn, msg string) error {
mlen := fmt.Sprintf("%04d", len(msg))
_, err := conn.Write([]byte(mlen + msg))
return err
}
The program will run 100000 goroutines at same time, and all goroutines will send message to the same connection。
I am doubt that server will receive error message like "hellohelloworldworld", but there is no problem when the program run in my Ubuntu 14.04LTS.
So, Do multiple goroutine will invoke a method on a Conn simultaneously?
=========================================================================
How can I keep the Write method atomic?
The documentation states:
Multiple goroutines may invoke methods on a Conn simultaneously.
There is no mention of whether each individual write is atomic. While the current implementation may ensure that each call to Write happens completely before the next call can begin, there is no guarantee in the language specification.
This answer implies writes are atomic.
Specifically implementors of the io.Write interface are required to return an error if a partial write occurs. net.Conn handles this on unix by acquiring a lock and calling write in a loop until the whole buffer is written. On Windows it calls WSASend which guarantees to send the whole buffer unless an error occurs. But the docs do have this warning:
The order of calls made to WSASend is also the order in which the
buffers are transmitted to the transport layer. WSASend should not be
called on the same stream-oriented socket concurrently from different
threads, because some Winsock providers may split a large send request
into multiple transmissions, and this may lead to unintended data
interleaving from multiple concurrent send requests on the same
stream-oriented socket.
Which means it wouldn't necessarily be atomic, unless Go acquires a mutex - which it does.
So basically it is atomic in practice. It is conceivable that an implementation could define thread-safety as just not crashing and allow interleaved writes by unlocking the mutex around calls to write (or not acquiring it at all on windows.) That doesn't make sense to me though, and the developers have clearly shown the opposite intent.
I have a process ProcessA that starts 2 threads ThreadA and ThreadB. Both threads send and recv data from ProcessB using the same socket descriptor.
So essentially:
int s;
void thread_fnA(void*)
{
while(1) {
sendto(s);
recvfrom(s);
}
}
void thread_fnB(void*)
{
while(1) {
sendto(s);
recvfrom(s);
}
}
int main()
{
s = socket(AF_UNIX, SOCK_DGRAM, 0);
bind(s);
dispatch_thread(A);
dispatch_thread(B);
}
Is there a possibility that the message to be received by thread B could be received in thread A.
So sequence of events:
Thread A prepares a message and calls sendto();
Thread B starts executing and prepares a message and calls sendto();
Thread B calls recvfrom() simultaneously with Thread A.
However the message content expected by both threads are different.
Can the messages be exchanged, ThreadB destined message be received by ThreadA.
Should the send and receive be involved in some locks. (Mutex)
I would suggest another design, in where you have a single thread doing the sending and receiving, and message queues for the other threads.
When the send/receive thread receives a message it check what kind of message it is, and ad it to the (protected) queue of the correct processing thread. The processing threads (your current treads A and B) gets the messages from its respective message queue, and process the messages in any way it pleases. Then if thread A or B wants to send a message, it passes it to the send/receive thread using another queue, which the send/receive thread polls.
Alternatively, the processing threads (A and B in your example) could send directly over the socket. Or each have a different socket used only for sending.
Since you are using the same socket in both threads it is possible that one thread reads the message that is destined to the other thread. Even if you use mutex, the design would be very difficult. You can open two sockets (or even pipes):
One socket is for communication in the direction A->B
The second socket in the direction B->A
A second possibility is having one socket with one writer (thread A) and one reader (thread B). The reader, when it receives a datagram, it decides, maybe based on datagram payload, what task to do. Or it can also send a task to other set of workers that will process the datagram.
I have a thread with a TCP Socket that connects to a server and waits for data in a while loop, so the thread never ends. When the socket receives data, it is parsed, and based on the opcode of the packet, should call x function. Whats the fastest/best way to go about that?
I read around that doing some kind of task/message queue system is a way of doing it, but not sure if there is any better options.
Should mention that I can not use boost:
Edit: Sorry, half asleep haha.
Here is the loop from thread x:
while (Running)
{
if (client.IsConnected())
{
Recieve();
}
FPlatformProcess::Sleep(0.01);
}
In the Receive function, it parses the data, and based on the packet opcode, I need to be able to call a function on the main thread (the GUI thread), because a lot of the packets are to spawn GUI objects, and I can't create GUI objects from any other thread than the main one.
So basically: I have a main thread, that spawns a new thread that enters a loop, listens for data, and I need to be able to call a function from the 2nd thread that runs on the main thread.
I want to ask a question about Application architecture1. There will be the main GUI thread for providing user interaction2. A Receive thread based on UDP socket that will receive UDP packets as they arrive (want this to be blocking.3. Another thread for sending event based as well as periodic UDP packets.How do I implement this architecture in Qt, basically i have following questions:1. For the Receive Thread, how do I make it blocking ?I know about readyRead() signal, and I can connect it to some slot that will process the datagram, but how do i loop this so that this thread does this forever. 2. In send Thread I can generate a signal form the GUI thread which will be received by the Sending Thread and a slot here will write some data on the socket, but again how will this thread survive when it has nothing to send, I mean loop, poll over something what ?
Use event loops in the secondary threads.
QThread::exec() starts the thread's event loop which will run until QThread::quit() is called. That should solve your "how to wait until something happens" problem. The default implementation of QThread::run() just calls exec(), so I'd go with that. You could set everything up in your main() method, e.g. for the sender thread:
//Create UI
MainWindow mainWindow;
mainWindow.show();
//set up sender thread and the `QObject` doing the actual work (Sender)
QThread senderThread;
Sender sender; //the object doing the actual sending
sender.moveToThread(&sender); //move sender to its thread
senderThread.start(); //starts the thread which will then enter the event loop
//connect UI to sender thread
QObject::connect(&mainWindow, SIGNAL(sendMessage(QString)), &sender, SLOT(sendMessage(QString)), Qt::QueuedConnection);
...
const int ret = app.exec(); // enter main event loop
`senderThread.quit();` //tell sender thread to quit its event loop
`senderThread.wait();` //wait until senderThread is done
`return ret;` // leave main
Sender would just be a QObject with a sendMessage() slot doing the sending, a QTimer plus another slot for the periodic UDP packages, etc.