Golang newbie here. Can somebody explain why the following code generates a deadlock?
I am aware of sending true to boolean <- done channel but I don't want to use it.
package main
import (
"fmt"
"sync"
"time"
)
var wg2 sync.WaitGroup
func producer2(c chan<- int) {
for i := 0; i < 5; i++ {
time.Sleep(time.Second * 10)
fmt.Println("Producer Writing to chan %d", i)
c <- i
}
}
func consumer2(c <-chan int) {
defer wg2.Done()
fmt.Println("Consumer Got value %d", <-c)
}
func main() {
c := make(chan int)
wg2.Add(5)
fmt.Println("Starting .... 1")
go producer2(c)
go consumer2(c)
fmt.Println("Starting .... 2")
wg2.Wait()
}
Following is my understanding and I know that it is wrong:
The channel will be blocked the moment 0 is written to it within the
loop of producer function
So I expect channel to be emptied by the
consumer afterwards.
As the channel is emptied in the step 2,
producer function can again put in another value and then get
blocked and steps 2 repeats again.
Your original deadlock is caused by wg2.Add(5), you were waiting for 5 goroutines to finish, but only one did; you called wg2.Done() once. Change this to wg2.Add(1), and your program will run without error.
However, I suspect that you intended to consume all the values in the channel not just one as you do. If you change consumer function to:
func consumer2(c <-chan int) {
defer wg2.Done()
for i := range c {
fmt.Printf("Consumer Got value %d\n", i)
}
}
You will get another deadlock because the channel is not closed in producer function, and consumer is waiting for more values that never arrive. Adding close(c) to the producer function will fix it.
Why it error?
Running your code gets the following error:
➜ gochannel go run dl.go
Starting .... 1
Starting .... 2
Producer Writing to chan 0
Consumer Got value 0
Producer Writing to chan 1
fatal error: all goroutines are asleep - deadlock!
Here is why:
There are three goroutines in your code: main,producer2 and consumer2. When it runs,
producer2 sends a number 0 to the channel
consumer2 recives 0 from the channel, and exits
producer2 sends 1 to the channel, but no one is consuming, since consumer2 already exits
producer2 is waiting
main executes wg2.Wait(), but not all waitgroup are closed. So main is waiting
Two goroutines are waiting here, does nothing, and nothing will be done no matter how long you wait. It is a deadlock! Golang detects it and panic.
There are two concepts you are confused here:
how waitgourp works
how to receive all values from a channel
I'll explain them here briefly, there are alreay many articles out there on the internet.
how waitgroup works
WaitGroup if a way to wait for all groutine to finish. When running goroutines in the background, it's important to know when all of them quits, then certain action can be conducted.
In your case, we run two goroutines, so at the beginning we should set wg2.Add(2), and each goroutine should add wg2.Done() to notify it is done.
Receive data from a channel
When receiving data from a channel. If you know exactly how many data it will send, use for loop this way:
for i:=0; i<N; i++ {
data = <-c
process(data)
}
Otherwise use it this way:
for data := range c {
process(data)
}
Also, Don't forget to close channel when there is no more data to send.
How to fix it?
With the above explanation, the code can be fixed as:
package main
import (
"fmt"
"sync"
"time"
)
var wg2 sync.WaitGroup
func producer2(c chan<- int) {
defer wg2.Done()
for i := 0; i < 5; i++ {
time.Sleep(time.Second * 1)
fmt.Printf("Producer Writing to chan %d\n", i)
c <- i
}
close(c)
}
func consumer2(c <-chan int) {
defer wg2.Done()
for i := range c {
fmt.Printf("Consumer Got value %d\n", i)
}
}
func main() {
c := make(chan int)
wg2.Add(2)
fmt.Println("Starting .... 1")
go producer2(c)
go consumer2(c)
fmt.Println("Starting .... 2")
wg2.Wait()
}
Here is another possible way to fix it.
The expected output
Fixed code gives the following output:
➜ gochannel go run dl.go
Starting .... 1
Starting .... 2
Producer Writing to chan 0
Consumer Got value 0
Producer Writing to chan 1
Consumer Got value 1
Producer Writing to chan 2
Consumer Got value 2
Producer Writing to chan 3
Consumer Got value 3
Producer Writing to chan 4
Consumer Got value 4
Related
I have situation where in, the main go routines will create "x" go routines. but it is interested only in "y" ( y < x ) go routines to finish.
I was hoping to use Waitgroup. But Waitgroup only allows me to wait on all go routines. I cannot, for example do this,
1. wg.Add (y)
2 create "x" go routines. These routines will call wg.Done() when finished.
3. wg. Wait()
This panics when the y+1 go routine calls wg.Done() because the wg counter goes negative.
I sure can use channels to solve this but I am interested if Waitgroup solves this.
As noted in Adrian's answer, sync.WaitGroup is a simple counter whose Wait method will block until the counter value reaches zero. It is intended to allow you to block (or join) on a number of goroutines before allowing a main flow of execution to proceed.
The interface of WaitGroup is not sufficiently expressive for your usecase, nor is it designed to be. In particular, you cannot use it naïvely by simply calling wg.Add(y) (where y < x). The call to wg.Done by the (y+1)th goroutine will cause a panic, as it is an error for a wait group to have a negative internal value. Furthermore, we cannot be "smart" by observing the internal counter value of the WaitGroup; this would break an abstraction and, in any event, its internal state is not exported.
Implement your own!
You can implement the relevant logic yourself using some channels per the code below (playground link). Observe from the console that 10 goroutines are started, but after two have completed, we fallthrough to continue execution in the main method.
package main
import (
"fmt"
"time"
)
// Set goroutine counts here
const (
// The number of goroutines to spawn
x = 10
// The number of goroutines to wait for completion
// (y <= x) must hold.
y = 2
)
func doSomeWork() {
// do something meaningful
time.Sleep(time.Second)
}
func main() {
// Accumulator channel, used by each goroutine to signal completion.
// It is buffered to ensure the [y+1, ..., x) goroutines do not block
// when sending to the channel, which would cause a leak. It will be
// garbage collected when all goroutines end and the channel falls
// out of scope. We receive y values, so only need capacity to receive
// (x-y) remaining values.
accChan := make(chan struct{}, x-y)
// Spawn "x" goroutines
for i := 0; i < x; i += 1 {
// Wrap our work function with the local signalling logic
go func(id int, doneChan chan<- struct{}) {
fmt.Printf("starting goroutine #%d\n", id)
doSomeWork()
fmt.Printf("goroutine #%d completed\n", id)
// Communicate completion of goroutine
doneChan <- struct{}{}
}(i, accChan)
}
for doneCount := 0; doneCount < y; doneCount += 1 {
<-accChan
}
// Continue working
fmt.Println("Carrying on without waiting for more goroutines")
}
Avoid leaking resources
As this does not wait for the [y+1, ..., x) goroutines to complete, you should take special care in the doSomeWork function to remove or minimize the risk that the work can block indefinitely, which would also cause a leak. Remove, where possible, the feasibility of indefinite blocking on I/O (including channel operations) or falling into infinite loops.
You could use a context to signal to the additional goroutines when their results are no longer required to have them break out of execution.
WaitGroup doesn't actually wait on goroutines, it waits until its internal counter reaches zero. If you only Add() the number of goroutines you care about, and you only call Done() in those goroutines you care about, then Wait() will only block until those goroutines you care about have finished. You are in complete control of the logic and flow, there are no restrictions on what WaitGroup "allows".
Are these y specific go-routines that you are trying to track, or any y out of the x? What are the criteria?
Update:
1. If you hve control over any criteria to pick matching y go-routines:
You can do wp.wg.Add(1) and wp.wg.Done() from inside the goroutine based on your condition by passing it as a pointer argument into the goroutine, if your condition can't be checked outside the goroutine.
Something like below sample code. Will be able to be more specific if you provide more details of what you are trying to do.
func sampleGoroutine(z int, b string, wg *sync.WaitGroup){
defer func(){
if contition1{
wg.Done()
}
}
if contition1 {
wg.Add(1)
//do stuff
}
}
func main() {
wg := sync.WaitGroup{}
for i := 0; i < x; i++ {
go sampleGoroutine(1, "one", &wg)
}
wg.Wait()
}
2. If you have no control over which ones, and just want the first y:
Based on your comment, that you have no control/desire to pick any specific goroutines, but the ones that finish first. If you would want to do it in a generic way, you can use the below custom waitGroup implementation that fits your use case. (It's not copy-safe, though. Also doesn't have/need wg.Add(int) method)
type CountedWait struct {
wait chan struct{}
limit int
}
func NewCountedWait(limit int) *CountedWait {
return &CountedWait{
wait: make(chan struct{}, limit),
limit: limit,
}
}
func (cwg *CountedWait) Done() {
cwg.wait <- struct{}{}
}
func (cwg *CountedWait) Wait() {
count := 0
for count < cwg.limit {
<-cwg.wait
count += 1
}
}
Which can be used as follows:
func sampleGoroutine(z int, b string, wg *CountedWait) {
success := false
defer func() {
if success == true {
fmt.Printf("goroutine %d finished successfully\n", z)
wg.Done()
}
}()
fmt.Printf("goroutine %d started\n", z)
time.Sleep(time.Second)
if rand.Intn(10)%2 == 0 {
success = true
}
}
func main() {
x := 10
y := 3
wg := NewCountedWait(y)
for i := 0; i < x; i += 1 {
// Wrap our work function with the local signalling logic
go sampleGoroutine(i, "something", wg)
}
wg.Wait()
fmt.Printf("%d out of %d goroutines finished successfully.\n", y, x)
}
3. You can also club in context with 2 to ensure that the remaining goroutines don't leak
You may not be able to run this on play.golang, as it has some long sleeps.
Below is a sample output:
(note that, there may be more than y=3 goroutines marking Done, but you are only waiting till 3 finish)
goroutine 9 started
goroutine 0 started
goroutine 1 started
goroutine 2 started
goroutine 3 started
goroutine 4 started
goroutine 5 started
goroutine 5 marking done
goroutine 6 started
goroutine 7 started
goroutine 7 marking done
goroutine 8 started
goroutine 3 marking done
continuing after 3 out of 10 goroutines finished successfully.
goroutine 9 will be killed, bcz cancel
goroutine 8 will be killed, bcz cancel
goroutine 6 will be killed, bcz cancel
goroutine 1 will be killed, bcz cancel
goroutine 0 will be killed, bcz cancel
goroutine 4 will be killed, bcz cancel
goroutine 2 will be killed, bcz cancel
Play links
https://play.golang.org/p/l5i6X3GClBq
https://play.golang.org/p/Bcns0l9OdFg
https://play.golang.org/p/rkGSLyclgje
I am new to go and I am trying to understand the way channels in goroutines work. To my understanding, the keyword range could be used to iterate over a the values of the channel up until the channel is closed or the buffer runs out; hence, a for range c will repeatedly loops until the buffer runs out.
I have the following simple function that adds value to a channel:
func main() {
c := make(chan int)
go printchannel(c)
for i:=0; i<10 ; i++ {
c <- i
}
}
I have two implementations of printchannel and I am not sure why the behaviour is different.
Implementation 1:
func printchannel(c chan int) {
for range c {
fmt.Println(<-c)
}
}
output: 1 3 5 7
Implementation 2:
func printchannel(c chan int) {
for i:=range c {
fmt.Println(i)
}
}
output: 0 1 2 3 4 5 6 7 8
And I was expecting neither of those outputs!
Wanted output: 0 1 2 3 4 5 6 7 8 9
Shouldnt the main function and the printchannel function run on two threads in parallel, one adding values to the channel and the other reading the values up until the channel is closed? I might be missing some fundamental go/thread concept here and pointers to that would be helpful.
Feedback on this (and my understanding to channels manipulation in goroutines) is greatly appreciated!
Implementation 1. You're reading from the channel twice - range c and <-c are both reading from the channel.
Implementation 2. That's the correct approach. The reason you might not see 9 printed is that two goroutines might run in parallel threads. In that case it might go like this:
main goroutine sends 9 to the channel and blocks until it's read
second goroutine receives 9 from the channel
main goroutine unblocks and exits. That terminates whole program which doesn't give second goroutine a chance to print 9
In case like that you have to synchronize your goroutines. For example, like so
func printchannel(c chan int, wg *sync.WaitGroup) {
for i:=range c {
fmt.Println(i)
}
wg.Done() //notify that we're done here
}
func main() {
c := make(chan int)
wg := sync.WaitGroup{}
wg.Add(1) //increase by one to wait for one goroutine to finish
//very important to do it here and not in the goroutine
//otherwise you get race condition
go printchannel(c, &wg) //very important to pass wg by reference
//sync.WaitGroup is a structure, passing it
//by value would produce incorrect results
for i:=0; i<10 ; i++ {
c <- i
}
close(c) //close the channel to terminate the range loop
wg.Wait() //wait for the goroutine to finish
}
As to goroutines vs threads. You shouldn't confuse them and probably should understand the difference between them. Goroutines are green threads. There're countless blog posts, lectures and stackoverflow answers on that topic.
In implementation 1, range reads into channel once, then again in Println. Hence you're skipping over 2, 4, 6, 8.
In both implementations, once the final i (9) has been sent to goroutine, the program exits. Thus goroutine does not have the time to print out 9. To solve it, use a WaitGroup as has been mentioned in the other answer, or a done channel to avoid semaphore/mutex.
func main() {
c := make(chan int)
done := make(chan bool)
go printchannel(c, done)
for i:=0; i<10 ; i++ {
c <- i
}
close(c)
<- done
}
func printchannel(c chan int, done chan bool) {
for i := range c {
fmt.Println(i)
}
done <- true
}
The reason your first implementation only returns every other number is because you are, in effect "taking" from c twice each time the loop runs: first with range, then again with <-. It just happens that you're not actually binding or using the first value taken off the channel, so all you end up printing is every other one.
An alternative approach to your first implementation would be to not use range at all, e.g.:
func printchannel(c chan int) {
for {
fmt.Println(<-c)
}
}
I could not replicate the behavior of your second implementation, on my machine, but the reason for that is that both of your implementations are racy - they will terminate whenever main ends, regardless of what data may be pending in a channel or however many goroutines may be active.
As a closing note, I'd warn you not to think about goroutines as explicitly being "threads", though they have a similar mental model and interface. In a simple program like this it's not at all unlikely that Go might just do it all using a single OS thread.
Your first loop does not work as you have 2 blocking channel receivers and they do not execute at the same time.
When you call the goroutine the loop starts, and it waits for the first value to be sent to the channel. Effectively think of it as <-c .
When the for loop in the main function runs it sends 0 on the Chan. At this point the range c recieves the value and stops blocking the execution of the loop.
Then it is blocked by the reciever at fmt.println(<-c) . When 1 is sent on the second iteration of the loop in main the recieved at fmt.println(<-c) reads from the channel, allowing fmt.println to execute thus finishing the loop and waiting for a value at the for range c .
Your second implementation of the looping mechanism is the correct one.
The reason it exits before printing to 9 is that after the for loop in main finishes the program goes ahead and completes execution of main.
In Go func main is launched as a goroutine itself while executing. Thus when the for loop in main completes it goes ahead and exits, and as the print is within a parallel goroutine that is closed, it is never executed. There is no time for it to print as there is nothing to block main from completing and exiting the program.
One way to solve this is to use wait groups http://www.golangprograms.com/go-language/concurrency.html
In order to get the expected result you need to have a blocking process running in main that provides enough time or waits for confirmation of the execution of the goroutine before allowing the program to continue.
I'm rather new in golang, and trying to understand main principles and write gouroutines based code, using chanels.
In others langs that i was using there were no such instruments, and i wonder getting such an errors like panic...
my code:
package main
import "fmt"
import (
"time"
)
type Work struct {
x,y,z int
}
func worker(in <-chan *Work, out chan<- *Work){
for w := range in {
w.z = w.x + w.y
time.Sleep(time.Duration(w.z))
out <-w
}
}
func sendWork(in chan <- *Work){
var wo *Work
wo.x, wo.y, wo.z = 1,2,3
in <- wo
in <- wo
in <- wo
in <- wo
in <- wo
}
func receiveWork(out <-chan *Work ) []*Work{
var slice []*Work
for el := range out {
slice = append(slice, el)
}
return slice
}
func main() {
in, out := make(chan *Work), make(chan *Work)
for i := 0; i<3; i++{
go worker(in, out)
}
go sendWork(in)
data := receiveWork(out)
fmt.Printf("%v", data)
}
But at the terminal i got this :
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x1 addr=0x0 pc=0x401130]
goroutine 8 [running]:
main.sendWork(0xc0820101e0)
C:/temp/gocode/src/helloA/helloA.go:21 +0x20
created by main.main
C:/temp/gocode/src/helloA/helloA.go:43 +0xe4
goroutine 1 [chan receive]:
main.receiveWork(0xc082010240, 0x0, 0x0, 0x0)
C:/temp/gocode/src/helloA/helloA.go:31 +0x80
main.main()
C:/temp/gocode/src/helloA/helloA.go:45 +0xf2
goroutine 5 [chan receive]:
main.worker(0xc0820101e0, 0xc082010240)
C:/temp/gocode/src/helloA/helloA.go:12 +0x55
created by main.main
C:/temp/gocode/src/helloA/helloA.go:40 +0xaf
goroutine 6 [runnable]:
main.worker(0xc0820101e0, 0xc082010240)
C:/temp/gocode/src/helloA/helloA.go:11
created by main.main
C:/temp/gocode/src/helloA/helloA.go:40 +0xaf
goroutine 7 [runnable]:
main.worker(0xc0820101e0, 0xc082010240)
C:/temp/gocode/src/helloA/helloA.go:11
created by main.main
C:/temp/gocode/src/helloA/helloA.go:40 +0xaf
How can i determine where is the problem, and how can i close the gouroutines nicely, not to left them as processes...
p.s. Forgive me my noob questions. please
The nil dereference:
You are trying to access a struct referenced by a pointer, but that pointer has not been set to an instance of that struct. You have to declare a struct that you can point the pointer to.
The error first appears here:
wo.x, wo.y, wo.z = 1,2,3
where you try to write to the object pointed to by wo. But the pointer is nil here; it's not actually pointing to an instance of Work. We have to create that instance, so we can point to it.
The nil value for a pointer to a struct is nil. If you don't declare a struct instance for it to point to, it points to nil.
var wo *Work
declares wo as a pointer of type Work to nil.
var wo = &Work{}
declares wo as a pointer of type Work to a new instance of Work.
Or you can use the shorter syntax:
wo := &Work{}
As for the deadlock:
When we close a channel, range loops over that channel will exit. In func worker we range over a channel. When this channel is closed, the worker(s) will exit.
In order to wait for all the workers to finish processing, we use a sync.WaitGroup. This is an easy way to wait for a group of goroutines to finish running before continuing.
First you tell the waitgroup how many goroutines it should wait for.
wg.Add(3)
Then you wait:
wg.Wait()
until all of the goroutines have called
wg.Done()
which they do when they are done executing.
In this case, we need to close the output channel when all workers are done executing, so that the func receiveWork can exit its for range loop. We can do this by starting a new goroutine for this task:
go func() {
wg.Wait()
close(out)
}()
This is the whole file, after these edits:
package main
import (
"fmt"
"sync"
"time"
)
type Work struct {
x, y, z int
}
func worker(in <-chan *Work, out chan<- *Work, wg *sync.WaitGroup) {
for w := range in {
w.z = w.x + w.y
time.Sleep(time.Duration(w.z))
out <- w
}
wg.Done() // this worker is now done; let the WaitGroup know.
}
func sendWork(in chan<- *Work) {
wo := &Work{x: 1, y: 2, z: 3} // more compact way of initializing the struct
in <- wo
in <- wo
in <- wo
in <- wo
in <- wo
close(in) // we are done sending to this channel; close it
}
func receiveWork(out <-chan *Work) []*Work {
var slice []*Work
for el := range out {
slice = append(slice, el)
}
return slice
}
func main() {
var wg sync.WaitGroup
in, out := make(chan *Work), make(chan *Work)
wg.Add(3) // number of workers
for i := 0; i < 3; i++ {
go worker(in, out, &wg)
}
go sendWork(in)
go func() {
wg.Wait()
close(out)
}()
data := receiveWork(out)
fmt.Printf("%v", data)
}
Which outputs:
[0x104382f0 0x104382f0 0x104382f0 0x104382f0 0x104382f0]
which is probably not what you expected. It does however highlight one problem with this code. More on that later.
If you want to print the contents of the structs, you could either stop using pointers to Work, or loop over the elements of the slice and print them one-by-one, like this:
for _, w := range data {
fmt.Printf("%v", w)
}
which outputs:
&{1 2 3}&{1 2 3}&{1 2 3}&{1 2 3}&{1 2 3}
Go doesn't follow pointers more than one step down when printing, to avoid infinite recursion, so you have to do this manually.
Race condition:
Since you are sending pointers to the same instance of *Work multiple times down the channel, that same instance is being accessed by multiple goroutines at the same time without synchronization. What you probably want is to stop using pointers, and use values. Work instead of *Work.
If you want to use pointers, maybe because Work is actually really big, you probably want to make multiple instances of *Work so you only ever send it to one goroutine.
Here's what the go race detector has to say about the code:
C:/Go\bin\go.exe run -race C:/gopath/src/github.com/drathier/scratchpad/go/main/main.go
[0xc0820403c0 0xc0820403c0 0xc0820403c0 0xc0820403c0 0xc0820403c0]==================
WARNING: DATA RACE
Write by goroutine 6:
main.worker()
C:/gopath/src/github.com/drathier/scratchpad/go/main/main.go:15 +0x8a
Previous write by goroutine 8:
main.worker()
C:/gopath/src/github.com/drathier/scratchpad/go/main/main.go:15 +0x8a
Goroutine 6 (running) created at:
main.main()
C:/gopath/src/github.com/drathier/scratchpad/go/main/main.go:45 +0x10c
Goroutine 8 (running) created at:
main.main()
C:/gopath/src/github.com/drathier/scratchpad/go/main/main.go:45 +0x10c
==================
Found 1 data race(s)
exit status 66
At this row:
w.z = w.x + w.y
all goroutines are modifying w.z concurrently, so if they try to write different values to w.z, there's no telling what value actually end up in there. Once again, this is easily fixed by creating multiple instances of *Work, or by using values instead of pointers: Work.
TL;DR: Please just go to the last part and tell me how you would solve this problem.
I've begun using Go this morning coming from Python. I want to call a closed-source executable from Go several times, with a bit of concurrency, with different command line arguments. My resulting code is working just well but I'd like to get your input in order to improve it. Since I'm at an early learning stage, I'll also explain my workflow.
For the sake of simplicity, assume here that this "external closed-source program" is zenity, a Linux command line tool that can display graphical message boxes from the command line.
Calling an executable file from Go
So, in Go, I would go like this:
package main
import "os/exec"
func main() {
cmd := exec.Command("zenity", "--info", "--text='Hello World'")
cmd.Run()
}
This should be working just right. Note that .Run() is a functional equivalent to .Start() followed by .Wait(). This is great, but if I wanted to execute this program just once, the whole programming stuff would not be worth it. So let's just do that multiple times.
Calling an executable multiple times
Now that I had this working, I'd like to call my program multiple times, with custom command line arguments (here just i for the sake of simplicity).
package main
import (
"os/exec"
"strconv"
)
func main() {
NumEl := 8 // Number of times the external program is called
for i:=0; i<NumEl; i++ {
cmd := exec.Command("zenity", "--info", "--text='Hello from iteration n." + strconv.Itoa(i) + "'")
cmd.Run()
}
}
Ok, we did it! But I still can't see the advantage of Go over Python … This piece of code is actually executed in a serial fashion. I have a multiple-core CPU and I'd like to take advantage of it. So let's add some concurrency with goroutines.
Goroutines, or a way to make my program parallel
a) First attempt: just add "go"s everywhere
Let's rewrite our code to make things easier to call and reuse and add the famous go keyword:
package main
import (
"os/exec"
"strconv"
)
func main() {
NumEl := 8
for i:=0; i<NumEl; i++ {
go callProg(i) // <--- There!
}
}
func callProg(i int) {
cmd := exec.Command("zenity", "--info", "--text='Hello from iteration n." + strconv.Itoa(i) + "'")
cmd.Run()
}
Nothing! What is the problem? All the goroutines are executed at once. I don't really know why zenity is not executed but AFAIK, the Go program exited before the zenity external program could even be initialized. This was confirmed by the use of time.Sleep: waiting for a couple of seconds was enough to let the 8 instance of zenity launch themselves. I don't know if this can be considered a bug though.
To make it worse, the real program I'd actually like to call takes a while to execute itself. If I execute 8 instances of this program in parallel on my 4-core CPU, it's gonna waste some time doing a lot of context switching … I don't know how plain Go goroutines behave, but exec.Command will launch zenity 8 times in 8 different threads. To make it even worse, I want to execute this program more than 100,000 times. Doing all of that at once in goroutines won't be efficient at all. Still, I'd like to leverage my 4-core CPU!
b) Second attempt: use pools of goroutines
The online resources tend to recommend the use of sync.WaitGroup for this kind of work. The problem with that approach is that you are basically working with batches of goroutines: if I create of WaitGroup of 4 members, the Go program will wait for all the 4 external programs to finish before calling a new batch of 4 programs. This is not efficient: CPU is wasted, once again.
Some other resources recommended the use of a buffered channel to do the work:
package main
import (
"os/exec"
"strconv"
)
func main() {
NumEl := 8 // Number of times the external program is called
NumCore := 4 // Number of available cores
c := make(chan bool, NumCore - 1)
for i:=0; i<NumEl; i++ {
go callProg(i, c)
c <- true // At the NumCoreth iteration, c is blocking
}
}
func callProg(i int, c chan bool) {
defer func () {<- c}()
cmd := exec.Command("zenity", "--info", "--text='Hello from iteration n." + strconv.Itoa(i) + "'")
cmd.Run()
}
This seems ugly. Channels were not intended for this purpose: I'm exploiting a side-effect. I love the concept of defer but I hate having to declare a function (even a lambda) to pop a value out of the dummy channel that I created. Oh, and of course, using a dummy channel is, by itself, ugly.
c) Third attempt: die when all the children are dead
Now we are nearly finished. I have just to take into account yet another side effect: the Go program closes before all the zenity pop-ups are closed. This is because when the loop is finised (at the 8th iteration), nothing prevents the program from finishing. This time, sync.WaitGroup will be useful.
package main
import (
"os/exec"
"strconv"
"sync"
)
func main() {
NumEl := 8 // Number of times the external program is called
NumCore := 4 // Number of available cores
c := make(chan bool, NumCore - 1)
wg := new(sync.WaitGroup)
wg.Add(NumEl) // Set the number of goroutines to (0 + NumEl)
for i:=0; i<NumEl; i++ {
go callProg(i, c, wg)
c <- true // At the NumCoreth iteration, c is blocking
}
wg.Wait() // Wait for all the children to die
close(c)
}
func callProg(i int, c chan bool, wg *sync.WaitGroup) {
defer func () {
<- c
wg.Done() // Decrease the number of alive goroutines
}()
cmd := exec.Command("zenity", "--info", "--text='Hello from iteration n." + strconv.Itoa(i) + "'")
cmd.Run()
}
Done.
My questions
Do you know any other proper way to limit the number of goroutines executed at once?
I don't mean threads; how Go manages goroutines internally is not relevant. I really mean limiting the number of goroutines launched at once: exec.Command creates a new thread each time it is called, so I should control the number of time it is called.
Does that code look fine to you?
Do you know how to avoid the use of a dummy channel in that case?
I can't convince myself that such dummy channels are the way to go.
I would spawn 4 worker goroutines that read the tasks from a common channel. Goroutines that are faster than others (because they are scheduled differently or happen to get simple tasks) will receive more task from this channel than others. In addition to that, I would use a sync.WaitGroup to wait for all workers to finish. The remaining part is just the creation of the tasks. You can see an example implementation of that approach here:
package main
import (
"os/exec"
"strconv"
"sync"
)
func main() {
tasks := make(chan *exec.Cmd, 64)
// spawn four worker goroutines
var wg sync.WaitGroup
for i := 0; i < 4; i++ {
wg.Add(1)
go func() {
for cmd := range tasks {
cmd.Run()
}
wg.Done()
}()
}
// generate some tasks
for i := 0; i < 10; i++ {
tasks <- exec.Command("zenity", "--info", "--text='Hello from iteration n."+strconv.Itoa(i)+"'")
}
close(tasks)
// wait for the workers to finish
wg.Wait()
}
There are probably other possible approaches, but I think this is a very clean solution that is easy to understand.
A simple approach to throttling (execute f() N times but maximum maxConcurrency concurrently), just a scheme:
package main
import (
"sync"
)
const maxConcurrency = 4 // for example
var throttle = make(chan int, maxConcurrency)
func main() {
const N = 100 // for example
var wg sync.WaitGroup
for i := 0; i < N; i++ {
throttle <- 1 // whatever number
wg.Add(1)
go f(i, &wg, throttle)
}
wg.Wait()
}
func f(i int, wg *sync.WaitGroup, throttle chan int) {
defer wg.Done()
// whatever processing
println(i)
<-throttle
}
Playground
I wouldn't probably call the throttle channel "dummy". IMHO it's an elegant way (it's not my invention of course), how to limit concurrency.
BTW: Please note that you're ignoring the returned error from cmd.Run().
🧩 Modules
Golang Concurrency Manager
📃 Template
package main
import (
"fmt"
"github.com/zenthangplus/goccm"
"math/rand"
"runtime"
)
func main() {
semaphore := goccm.New(runtime.NumCPU())
for {
semaphore.Wait()
go func() {
fmt.Println(rand.Int())
semaphore.Done()
}()
}
semaphore.WaitAllDone()
}
🎰 Optimal routine quantity
If the operation is CPU bounded: runtime.NumCPU()
Otherwise test with: time go run *.go
🔨 Configure
export GOPATH="$(pwd)/gopath"
go mod init *.go
go mod tidy
🧹 CleanUp
find "${GOPATH}" -exec chmod +w {} \;
rm --recursive --force "${GOPATH}"
try this:
https://github.com/korovkin/limiter
limiter := NewConcurrencyLimiter(10)
limiter.Execute(func() {
zenity(...)
})
limiter.Wait()
You could use Worker Pool pattern described here in this post.
This is how an implementation would look like ...
package main
import (
"os/exec"
"strconv"
)
func main() {
NumEl := 8
pool := 4
intChan := make(chan int)
for i:=0; i<pool; i++ {
go callProg(intChan) // <--- launch the worker routines
}
for i:=0;i<NumEl;i++{
intChan <- i // <--- push data which will be received by workers
}
close(intChan) // <--- will safely close the channel & terminate worker routines
}
func callProg(intChan chan int) {
for i := range intChan{
cmd := exec.Command("zenity", "--info", "--text='Hello from iteration n." + strconv.Itoa(i) + "'")
cmd.Run()
}
}
I'm pretty new to Go and there is one thing in my code which I don't understand.
I wrote a simple bubblesort algorithm (I know it's not really efficient ;)).
Now I want to start 3 GoRoutines. Each thread should sort his array independent from the other ones. When finished, the func. should print a "done"-Message.
Here is my Code:
package main
import (
"fmt"
"time" //for time functions e.g. Now()
"math/rand" //for pseudo random numbers
)
/* Simple bubblesort algorithm*/
func bubblesort(str string, a []int) []int {
for n:=len(a); n>1; n-- {
for i:=0; i<n-1; i++ {
if a[i] > a[i+1] {
a[i], a[i+1] = a[i+1], a[i] //swap
}
}
}
fmt.Println(str+" done") //done message
return a
}
/*fill slice with pseudo numbers*/
func random_fill(a []int) []int {
for i:=0; i<len(a); i++ {
a[i] = rand.Int()
}
return a
}
func main() {
rand.Seed( time.Now().UTC().UnixNano()) //set seed for rand.
a1 := make([]int, 34589) //create slice
a2 := make([]int, 42) //create slice
a3 := make([]int, 9999) //create slice
a1 = random_fill(a1) //fill slice
a2 = random_fill(a2) //fill slice
a3 = random_fill(a3) //fill slice
fmt.Println("Slices filled ...")
go bubblesort("Thread 1", a1) //1. Routine Start
go bubblesort("Thread 2", a2) //2. Routine Start
go bubblesort("Thread 3", a3) //3. Routine Start
fmt.Println("Main working ...")
time.Sleep(1*60*1e9) //Wait 1 minute for the "done" messages
}
This is what I get:
Slices filled ...
Main working ...
Thread 1 done
Thread 2 done
Thread 3 done
Should'nt Thread 2 finish first, since his slice is the smallest?
It seems that all the threads are waiting for the others to finish, because the "done"-messages appear at the same time, no matter how big the slices are..
Where is my brainbug? =)
Thanks in advance.
*Edit:
When putting "time.Sleep(1)" in the for-loop in the bubblesort func. it seems to work.. but I want to clock the duration on different machines with this code (I know, i have to change the random thing), so sleep would falsify the results.
Indeed, there is no garantee regarding the order in which your goroutines will be executed.
However if you force the true parallel processing by explicitly letting 2 processor cores run :
import (
"fmt"
"time" //for time functions e.g. Now()
"math/rand" //for pseudo random numbers
"runtime"
)
...
func main() {
runtime.GOMAXPROCS(2)
rand.Seed( time.Now().UTC().UnixNano()) //set seed for rand.
...
Then you will get the expected result :
Slices filled ...
Main working ...
Thread 2 done
Thread 3 done
Thread 1 done
Best regards
The important thing is the ability to "yield" the processor to other processes, before the whole potentialy long-running workload is finished. This holds true as well in single-core context or multi-core context (because Concurrency is not the same as Parallelism).
This is exactly what the runtime.Gosched() function does :
Gosched yields the processor, allowing other goroutines to run. It
does not suspend the current goroutine, so execution resumes
automatically.
Be aware that a "context switch" is not free : it costs a little time each time.
On my machine without yielding, your program runs in 5.1s.
If you yield in the outer loop (for n:=len(a); n>1; n--), it runs in 5.2s : small overhead.
If you yield in the inner loop (for i:=0; i<n-1; i++), it runs in 61.7s : huge overhead !!
Here is the modified program correctly yielding, with the small overhead :
package main
import (
"fmt"
"math/rand"
"runtime"
"time"
)
/* Simple bubblesort algorithm*/
func bubblesort(str string, a []int, ch chan []int) {
for n := len(a); n > 1; n-- {
for i := 0; i < n-1; i++ {
if a[i] > a[i+1] {
a[i], a[i+1] = a[i+1], a[i] //swap
}
}
runtime.Gosched() // yield after part of the workload
}
fmt.Println(str + " done") //done message
ch <- a
}
/*fill slice with pseudo numbers*/
func random_fill(a []int) []int {
for i := 0; i < len(a); i++ {
a[i] = rand.Int()
}
return a
}
func main() {
rand.Seed(time.Now().UTC().UnixNano()) //set seed for rand.
a1 := make([]int, 34589) //create slice
a2 := make([]int, 42) //create slice
a3 := make([]int, 9999) //create slice
a1 = random_fill(a1) //fill slice
a2 = random_fill(a2) //fill slice
a3 = random_fill(a3) //fill slice
fmt.Println("Slices filled ...")
ch1 := make(chan []int) //create channel of result
ch2 := make(chan []int) //create channel of result
ch3 := make(chan []int) //create channel of result
go bubblesort("Thread 1", a1, ch1) //1. Routine Start
go bubblesort("Thread 2", a2, ch2) //2. Routine Start
go bubblesort("Thread 3", a3, ch3) //3. Routine Start
fmt.Println("Main working ...")
<-ch1 // Wait for result 1
<-ch2 // Wait for result 2
<-ch3 // Wait for result 3
}
Output :
Slices filled ...
Main working ...
Thread 2 done
Thread 3 done
Thread 1 done
I also used channels to implement the rendez-vous, as suggested in my previous comment.
Best regards :)
Since the release of Go 1.2, the original program now works may work fine without modification. You may try it in Playground.
This is explained in the Go 1.2 release notes :
In prior releases, a goroutine that was looping forever could starve
out other goroutines on the same thread, a serious problem when
GOMAXPROCS provided only one user thread. In Go 1.2, this is partially
addressed: The scheduler is invoked occasionally upon entry to a
function.