Stop channels when ws not able to connect - multithreading

I have the following code which works ok, the issue is that when the socket.Connect() fails to connect I want to stop the process, I’ve tried with the following code
but it’s not working, I.e. if the socket connect fails to connect the program still runs.
What I want to happen is that if the connect fails, the process stops and the channe…what am I missing here?
func run (appName string) (err error) {
done = make(chan bool)
defer close(done)
serviceURL, e := GetContext().getServiceURL(appName)
if e != nil {
err = errors.New("process failed" + err.Error())
LogDebug("Exiting %v func[err =%v]", methodName, err)
return err
}
url := "wss://" + serviceURL + route
socket := gowebsocket.New(url)
addPass(&socket, user, pass)
socket.OnConnectError = OnConnectErrorHandler
socket.OnConnected = OnConnectedHandler
socket.OnTextMessage = socketTextMessageHandler
socket.OnDisconnected = OnDisconnectedHandler
LogDebug("In %v func connecting to URL %v", methodName, url)
socket.Connect()
jsonBytes, e := json.Marshal(payload)
if e != nil {
err = errors.New("build process failed" + e.Error())
LogDebug("Exiting %v func[err =%v]", methodName, err)
return err
}
jsonStr := string(jsonBytes)
LogDebug("In %v Connecting to payload JSON is %v", methodName, jsonStr)
socket.SendText(jsonStr)
<-done
LogDebug("Exiting %v func[err =%v]", methodName, err)
return err
}
func OnConnectErrorHandler(err error, socket gowebsocket.Socket) {
methodName := "OnConnectErrorHandler"
LogDebug("Starting %v parameters [err = %v , socket = %v]", methodName, err, socket)
LogInfo("Disconnected from server ")
done <- true
}
The process should open one ws connection for process that runs about 60-90 sec (like execute npm install) and get the logs of the process via web socket and when it finish , and of course handle the issue that could happen like network issue or some error running the process

So, #Slabgorb is correct - if you look here (https://github.com/sacOO7/GoWebsocket/blob/master/gowebsocket.go#L87) you will see that the OnConnectErrorHandler is called synchronously during the execution of your call to Connect(). The Connect() function doesn't kick off a separate goroutine to handle the websocket until after the connection is fully established and the OnConnected callback has completed. So when you try to write to the unbuffered channel done, you are blocking the same goroutine that called into the run() function to begin with, and you deadlock yourself, because no goroutine will ever be able to read from the channel to unblock you.
So you could go with his solution and turn it into a buffered channel, and that will work, but my suggestion would be not to write to a channel for this sort of one-time flag behavior, but use close signaling instead. Define a channel for each condition you want to terminate run(), and in the appropriate websocket handler function, close the channel when that condition happens. At the bottom of run(), you can select on all the channels, and exit when the first one closes. It would look something like this:
package main
import "errors"
func run(appName string) (err error) {
// first, define one channel per socket-closing-reason (DO NOT defer close these channels.)
connectErrorChan := make(chan struct{})
successDoneChan := make(chan struct{})
surpriseDisconnectChan := make(chan struct{})
// next, wrap calls to your handlers in a closure `https://gobyexample.com/closures`
// that captures a reference to the channel you care about
OnConnectErrorHandler := func(err error, socket gowebsocket.Socket) {
MyOnConnectErrorHandler(connectErrorChan, err, socket)
}
OnDisconnectedHandler := func(err error, socket gowebsocket.Socket) {
MyOnDisconectedHandler(surpriseDisconnectChan, err, socket)
}
// ... declare any other handlers that might close the connection here
// Do your setup logic here
// serviceURL, e := GetContext().getServiceURL(appName)
// . . .
// socket := gowebsocket.New(url)
socket.OnConnectError = OnConnectErrorHandler
socket.OnConnected = OnConnectedHandler
socket.OnTextMessage = socketTextMessageHandler
socket.OnDisconnected = OnDisconnectedHandler
// Prepare and send your message here...
// LogDebug("In %v func connecting to URL %v", methodName, url)
// . . .
// socket.SendText(jsonStr)
// now wait for one of your signalling channels to close.
select { // this will block until one of the handlers signals an exit
case <-connectError:
err = errors.New("never connected :( ")
case <-successDone:
socket.Close()
LogDebug("mission accomplished! :) ")
case <-surpriseDisconnect:
err = errors.New("somebody cut the wires! :O ")
}
if err != nil {
LogDebug(err)
}
return err
}
// *Your* connect error handler will take an extra channel as a parameter
func MyOnConnectErrorHandler(done chan struct{}, err error, socket gowebsocket.Socket) {
methodName := "OnConnectErrorHandler"
LogDebug("Starting %v parameters [err = %v , socket = %v]", methodName, err, socket)
LogInfo("Disconnected from server ")
close(done) // signal we are done.
}
This has a few advantages:
1) You don't need to guess which callbacks happen in-process and which happen in background goroutines (and you don't have to make all your channels buffered 'just in case')
2) Selecting on the multiple channels lets you find out why you are exiting and maybe handle cleanup or logging differently.
Note 1: If you choose to use close signaling, you have to use different channels for each source in order to avoid race conditions that might cause a channel to get closed twice from different goroutines (e.g. a timeout happens just as you get back a response, and both handlers fire; the second handler to close the same channel causes a panic.) This is also why you don't want to defer close all the channel at the top of the function.
Note 2: Not directly relevant to your question, but -- you don't need to close every channel - once all the handles to it go out of scope, the channel will get garbage collected whether or not it has been closed.

Ok, what is happening is the channel is blocking when you try to add something to it. Try initializing the done channel with a buffer (I used 1) like this:
done = make(chan bool, 1)

Related

Cancelling golang os.Stdin read using context

I want to be able to cancel an os.Stdin read using context which is not possible. Usually, you close the file handle to accomplish cancel, but I do not want to close os.Stdin
Possible solutions could be:
Can it be determined if Stdin.Read will block?
Can the thread be terminated like in pthreads?
Should os.Stdin be forwarded to another file handle that can be closed?
Here’s what I got, the ugliness is that scannerThread is left running on context cancelation:
// Keystrokes emits keystroke events
// on g0.Context() shutdown, scannerThread is left running until the next newline
// the lines channel is never closed
func Keystrokes(lines chan<- string, g0 parl.Go) {
var err error
defer g0.Done(err)
defer parl.Recover(parl.Annotation(), &err, parl.NoOnError)
// stdio.Scan cannot be terminated, so let that thread terminate whenever
var scanLines parl.NBChan[string]
go scannerThread(scanLines.Send, g0.Context())
// consume scannerThread output
scannerCh := scanLines.Ch()
done := g0.Context().Done()
for {
select {
case line := <-scannerCh:
lines <- line
case <-done:
return // canceled by context exit
}
}
}
// scannerThread reads from os.Stdin and therefore cannot be cancelled.
// send is a non-blocking send function.
// ctx indicates shutdown effective on next os.Stdin newline.
func scannerThread(send func(string), ctx context.Context) {
var err error
defer parl.Recover(parl.Annotation(), &err, parl.Infallible)
scanner := bufio.NewScanner(os.Stdin)
for scanner.Scan() { // scanner is typically stuck here
if ctx.Err() != nil {
return // terminated via context
}
send(scanner.Text())
}
// scanner had error
err = scanner.Err()
}
The answer is to read os.Stdin non-blocking, works on Linux macOS
os.Stdin has a block against closing it, I remember
Non-blocking os.Stdin.Read() returns bytes once a complete line is available, otherwise returns n==0
Here is how to set non-blocking:
err = unix.SetNonblock(int(os.Stdin.Fd()), true)
The state of non-blocking can also be read:
flags, err := unix.FcntlInt(os.Stdin.Fd(), unix.F_GETFL, 0)
wasSet = flags&unix.O_NONBLOCK != 0

Is retry handled from go client library for azure service bus?

I'm referencing to the client library in https://github.com/Azure/azure-service-bus-go. I was able to write a simple client which listens to a subscription and read the messages. If I drop the network, after a few seconds I could see the receive loop exiting with the error message "context canceled". I was hoping the client library will do some kind of retry mechanism and handle any connection issues or any server timeouts etc. Do we need to handle this our selves ? Also do we get any exceptions which we could identify and use it for this retrying mechanism ? Any sample code would be highly appreciated.
below is the sample code I tried (only including the vital parts).
err = subscription.Receive(ctx, servicebus.HandlerFunc(func(ctx context.Context, message *servicebus.Message) error {
fmt.Println(string(message.Data))
return message.Complete(ctx)
}))
if err != nil {
fmt.Println("FATAL: ", err)
return
}
Unfortunately, the older library doesn't do a good job of retrying, so you'll need to wrap the code in your own retry loop.
func demoReceiveWithRecovery(ns *servicebus.Namespace) {
parentCtx := context.TODO()
for {
q, err := ns.NewQueue(queueName)
if err != nil {
panic(err)
}
defer q.Close(parentCtx)
handler := servicebus.HandlerFunc(func(c context.Context, m *servicebus.Message) error {
log.Printf("Received message")
return m.Complete(c)
})
err = q.Receive(parentCtx, handler)
// check for potential recoverable situation
if err != nil && errors.Is(err, context.Canceled) {
// This workaround distinguishes between the handler cancelling because the
// library has disconnected vs the parent context (passed in by you) being cancelled.
if parentCtx.Err() != nil {
// our parent context cancelled, which caused the entire operation to be canceled
log.Printf("Cancelled by parent context")
return
} else {
// cancelled internally due to an error, we can restart the queue
log.Printf("Error occurred, restarting")
_ = q.Close(parentCtx)
continue
}
} else {
log.Printf("Other error, closing client and restarting: %s", err.Error())
_ = q.Close(parentCtx)
}
}
}
NOTE: This library was deprecated recently, but it was deprecated after you posted your question. The new package is here if you want to try it - azservicebus with a migration guide here to make things easier: migrationguide.md
The new package should recover properly in this scenario for receiving. There is an open bug in the new package I'm working on for the sending side to do recovery #16695.
If you want to ask further questions or have some feature requests you can submit those in the github issues for https://github.com/Azure/azure-sdk-for-go.

Multithreading socket connections in Golang

Background
I'm creating a chatroom server using Golang. Every time a client connects to the server, I'm starting the Client function as a new thread, so that each client gets a thread that is listened to. I was able to create a simple client server connection based on this tutorial, but now I am trying to create connections for multiple clients so that they can send messages to chatrooms.
Code explanation
By following this tutorial, it looks like I can create a thread by using go func() and wait for a connection to be made by including a channel (<-newClient). I pass a bool value into the channel before calling the user function, because the user function will run forever and I want to make other client connections. Each client connection will have the user function running for it.
Problem
I don't know how to pass the user connection variable to my other functions. I'm placing conn net in the arguments for my functions, but this is just a placeholder because I'm not sure the proper way to do it.
Also, my go func() with the call to user() after the channel implementation is my best attempt at multithreading, but I'm not sure that I'm thinking about it right.
Server.go
package main
import (
"net"
"fmt"
"bufio"
"strings"
"container/list"
"time"
)
type chatRoom struct {
name string
messages list.List
users list.List
lastUsed time.Time
}
var chatRooms *list.List //A list of chatrooms where each client can post messages, where messages can be seen by all clients in the chatroom
var conn net.Conn
func user(conn net) {
for {
message, _ := bufio.NewReader(conn).ReadString('\n') // will listen for message to process ending in newline (\n)
fmt.Print("Message Received:", string(message)) // output message received
s := strings.Split(string(message), ":")
if strings.Compare(s[0],"create") == 0{ //create a chat room
create(conn, s[1])
}else if strings.Compare(s[0],"list") == 0 { //List the current chatrooms
msg = listRooms(conn)
}else if strings.Compare(s[0],"join") == 0 { //Join the user to a chat room
join(conn, s[1])
}else if strings.Compare(s[0],"leave") == 0 { //Remove the user from a chatroom
leave(conn, s[1])
}else if strings.Compare(s[0],"message") == 0{ //Send a message to a chatroom
message(conn, s[1], s[2])
}
}
}
func main() {
fmt.Println("Launching server...")
this.userList = list.New()
this.chatRooms = list.New();
ln, _ := net.Listen("tcp", ":8081") // listen on all interfaces
conn, _ := ln.Accept() // accept connection on port
for { // run loop forever (or until ctrl-c)
go func(){
newClient := make(chan bool)
ln, _ := net.Listen("tcp", ":8081") // listen on all interfaces
conn, _ := ln.Accept() // accept connection on port
newClient <- true
user(conn)
}
<-newClient
}
}
It is very important to understand a distinction here: this is concurrency, not multithreading, and these are goroutines, not threads. These concepts are not interchangeable. As to your core issue, there are some significant issues with your implementation:
You're starting a goroutine closing over variables that are shared across loop iterations, crucially conn. Every time you accept a connection, you overwrite the conn which is shared by every goroutine. You should instead pass the conn to your goroutine so it has its own local copy, or create a new conn variable in each loop iteration instead of re-using one.
You're starting a new listener in every loop iteration, which is going to fail because the old listener is still using that port. You don't need a new listener. Just keep calling ln.Accept on the existing listener to continue accepting new connections. Take a look at the introduction of the documentation for the net package, or check any code that uses listeners in Go for an example.
You're creating newClient inside the goroutine, then trying to reference it outside the goroutine. This won't even compile, and it's unclear what you are trying to do with this channel in the first place.
Take a look at some existing networking code - in the net or net/http libraries or some popular projects on GitHub - to see good examples of how to write a network application. Do some web searches for blog posts or tutorials or how-to's, there are tons out there. And definitely read the documentation for the packages you're using, it will help you a lot.

Troubles with gitlab scraping via golang

I'm newbie in programming and I need help. Trying to write gitlab scraper on golang.
Something goes wrong when i'm trying to get information about projects in multithreading mode.
Here is the code:
func (g *Gitlab) getAPIResponce(url string, structure interface{}) error {
responce, responce_error := http.Get(url)
if responce_error != nil {
return responce_error
}
ret, _ := ioutil.ReadAll(responce.Body)
if string(ret) != "[]" {
err := json.Unmarshal(ret, structure)
return err
}
return errors.New(error_emptypage)
}
...
func (g *Gitlab) GetProjects() {
projects_chan := make(chan Project, g.LatestProjectID)
var waitGroup sync.WaitGroup
queue := make(chan struct{}, 50)
for i := g.LatestProjectID; i > 0; i-- {
url := g.BaseURL + projects_url + "/" + strconv.Itoa(i) + g.Token
waitGroup.Add(1)
go func(url string, channel chan Project) {
queue <- struct{}{}
defer waitGroup.Done()
var oneProject Project
err := g.getAPIResponce(url, &oneProject)
if err != nil {
fmt.Println(err.Error())
}
fmt.Printf(".")
channel <- oneProject
<-queue
}(url, projects_chan)
}
go func() {
waitGroup.Wait()
close(projects_chan)
}()
for project := range projects_chan {
if project.ID != 0 {
g.Projects = append(g.Projects, project)
}
}
}
And here is the output:
$ ./gitlab-auditor
latest project = 1532
Gathering projects...
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Get https://gitlab.example.com/api/v4/projects/563&private_token=SeCrEt_ToKeN: unexpected EOF
Get https://gitlab.example.com/api/v4/projects/558&private_token=SeCrEt_ToKeN: unexpected EOF
..Get https://gitlab.example.com/api/v4/projects/531&private_token=SeCrEt_ToKeN: unexpected EOF
Get https://gitlab.example.com/api/v4/projects/571&private_token=SeCrEt_ToKeN: unexpected EOF
.Get https://gitlab.example.com/api/v4/projects/570&private_token=SeCrEt_ToKeN: unexpected EOF
..Get https://gitlab.example.com/api/v4/projects/467&private_token=SeCrEt_ToKeN: unexpected EOF
Get https://gitlab.example.com/api/v4/projects/573&private_token=SeCrEt_ToKeN: unexpected EOF
................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Every time it's different projects, but it's id is around 550.
When I'm trying to curl links from output, i'm getting normal JSON. When I'm trying to run this code with queue := make(chan struct{}, 1) (in single thread) - everything is fine.
What can it be?
i would say this not a very clear way to achieve concurrency.
what seems to be happening here is
you create a buffered channel that has a size of 50.
then you fire up 1532 goroutines
the first 50 of them enqueue themselves and start processing. by the time they <-queue and free up somespace a random one from the next manages to get on the queue.
as people say in the comments most certainly you hit some limits around the time it the blast has made it around id 550. Then gitlab's API is angry at you and rate limits.
then another goroutine is fired that will close the channel to notify the main goroutine
the main goroutine reads messages.
the talk go concurrency patterns
as well as this blog post concurrency in go might help.
personally i rarely use buffered channels. for your problem i would go like:
define a number of workers
have the main goroutine fire up the workers with a func listening on a channel of ints , doing the api call, writing to a channel of projects
have the main goroutine send to a channel of ints the project number to be fetched and read from the channel of projects.
maybe ratelimit by firing a ticker and have main read from it before it sends the next request?
main closes the number channel to notify the others to die.

go websockets eof

I'm trying to make a simple command forwarder to connect my home computer to a server I own, so that I can push commands to my server and my home pc gets it. Those commands are simple pause/resume for my downloader. My design is, that on a server, I run a hub instance, which creates a window for passing commands and a window for backend to pass those commands to my pc. I'm bounding those two "windows" with a channel, they run a server. When a client connects and sends a message to the hub, it gets streamed through a channel to backend window and then to the real backend (on my home pc). When backend responds to the backend window on the hub, the hub prints the result back to the client.
With this approach, only the first message passes and works with my downloader. I have to reconnect the backend from my home pc with the hub each time I get a message to get this working properly. I don't think that's the proper way with websockets, so here I am. After one successful request (when the backend finishes it's work and replies the result), it gets looped forever with EOF error.
The important parts of the code are:
main executable
hub handlers
backend connector
If you put the source in your GOPATH (i'm developing it for the tip version of go to support modern websockets), to compile it:
go build gosab/cmd, to run it:
./cmd -mode="hub" hub
./cmd -mode="backend" --address="localhost:8082" backend
To pass messages to the hub, use this javascript:
var s = new WebSocket("ws://localhost:8082")
s.send("1 5")
So how do I handle it? Are channels a good way to communicate between two different requests?
I'm surprised you haven't received an answer to this.
What you need to do is something like the code below. When you receive an incoming websocket connection, a new goroutine is spawned for that connection. If you let that goroutine end, it'll disconnect the websocket client.
I'm making an assumption that you're not necessarily going to be running the client and server on the same computer. If you always are, then it'd be better to do the communication internally via channels or such instead of using websockets or a network port. I only mention this because I'm not completely sure what you're using this for. I just hope I answered the right part of your question.
package main
import (
"code.google.com/p/go.net/websocket"
"flag"
"fmt"
"net/http"
"os"
"time"
)
type Message struct {
RequestID int
Command string
SomeOtherThing string
Success bool
}
var mode *string = flag.String("mode", "<nil>", "Mode: server or client")
var address *string = flag.String("address", "localhost:8080", "Bind address:port")
func main() {
flag.Parse()
switch *mode {
case "server":
RunServer()
case "client":
RunClient()
default:
flag.Usage()
}
}
func RunServer() {
http.Handle("/", http.FileServer(http.Dir("www")))
http.Handle("/server", websocket.Handler(WSHandler))
fmt.Println("Starting Server")
err := http.ListenAndServe(*address, nil)
if err != nil {
fmt.Printf("HTTP failed: %s\n", err.Error())
os.Exit(1)
}
}
func WSHandler(ws *websocket.Conn) {
defer ws.Close()
fmt.Println("Client Connected")
for {
var message Message
err := websocket.JSON.Receive(ws, &message)
if err != nil {
fmt.Printf("Error: %s\n", err.Error())
return
}
fmt.Println(message)
// do something useful here...
response := new(Message)
response.RequestID = message.RequestID
response.Success = true
response.SomeOtherThing = "The hot dog left the castle as requested."
err = websocket.JSON.Send(ws, response)
if err != nil {
fmt.Printf("Send failed: %s\n", err.Error())
os.Exit(1)
}
}
}
func RunClient() {
fmt.Println("Starting Client")
ws, err := websocket.Dial(fmt.Sprintf("ws://%s/server", *address), "", fmt.Sprintf("http://%s/", *address))
if err != nil {
fmt.Printf("Dial failed: %s\n", err.Error())
os.Exit(1)
}
incomingMessages := make(chan Message)
go readClientMessages(ws, incomingMessages)
i := 0
for {
select {
case <-time.After(time.Duration(2e9)):
i++
response := new(Message)
response.RequestID = i
response.Command = "Eject the hot dog."
err = websocket.JSON.Send(ws, response)
if err != nil {
fmt.Printf("Send failed: %s\n", err.Error())
os.Exit(1)
}
case message := <-incomingMessages:
fmt.Println(message)
}
}
}
func readClientMessages(ws *websocket.Conn, incomingMessages chan Message) {
for {
var message Message
err := websocket.JSON.Receive(ws, &message)
if err != nil {
fmt.Printf("Error: %s\n", err.Error())
return
}
incomingMessages <- message
}
}

Resources