In Go, it can be specified which direction a channel can send. I am trying to create an example about it, look at the following code:
package main
import (
"fmt"
"time"
)
func main() {
ic_send_only := make(<-chan int) //a channel that can only send data - arrow going out is sending
ic_recv_only := make(chan<- int) //a channel that can only receive a data - arrow going in is receiving
go func() {
ic_recv_only <- 4555
}()
go func() {
ic_send_only <- ic_recv_only
}()
fmt.Println(ic_recv_only)
time.Sleep(10000)
}
I get the compiler error
# command-line-arguments
.\send_receive.go:19: invalid operation: ic_send_only <- ic_recv_only (send to receive-only type <-chan int)
[Finished in 0.2s with exit code 2]
How can I use channel direction in the right way?
Or does anyone have a better sample than me?
Three issues:
You have the send and receive operations reversed (which is the error you're seeing)
Creating recv-only or send-only channels make no sense, as you cannot use them
The notation you're using is trying to send the channel itself, not the result. You need to receive and send, which requires two arrows.
ic_recv_only <- <-ic_send_only
You may be confused because you have the terminology reversed. <-ch is a "receive operation", and ch <- is a send operation. Note that in your example, everything would be deadlocked because you can't complete the corresponding sends and receives to pass something through either channel.
Here is a complete example:
// This receives an int from a channel. The channel is receive-only
func consumer(ch <-chan int) int {
return <-ch
}
// This sends an int over a channel. The channel is send-only
func producer(i int, ch chan<- int) {
ch <- i
}
func main() {
ch := make(chan int)
go producer(42, ch)
result := consumer(ch)
fmt.Println("received", result)
}
The key point that JimB has made, to summarise, is that
You create channels using make
Every channel has two ends.
You communicate via the ends of channels via <-. The ends are important.
There is a sending end and a receiving end; channels are unidirectional.
Note also that each end can be safely accessed concurrently by more than one goroutine.
Also note that JimB's example producer(i int, ch chan<- int) and consumer(ch <-chan int) functions have parameters that specify which end they use via <-chan and chan<-, instead of just chan. Although this is optional, it is good practice because the compiler will help you fix silly mistakes if you do this.
Related
My objective is to create a program that counts every unique word's occurrence in a text file in a parallellised fashion, all occurrences have to be presented in a single map.
What I do here is dividing the textfile into string and then to an array. That array is then divided into two slices of equal length and fed concurrently to the mapper function.
func WordCount(text string) (map[string]int) {
wg := new(sync.WaitGroup)
s := strings.Fields(newText)
freq := make(map[string]int,len(s))
channel := make(chan map[string]int,2)
wg.Add(1)
go mappers(s[0:(len(s)/2)], freq, channel,wg)
wg.Add(1)
go mappers(s[(len(s)/2):], freq, channel,wg)
wg.Wait()
actualMap := <-channel
return actualMap
func mappers(slice []string, occurrences map[string]int, ch chan map[string]int, wg *sync.WaitGroup) {
var l = sync.Mutex{}
for _, word := range slice {
l.Lock()
occurrences[word]++
l.Unlock()
}
ch <- occurrences
wg.Done()
}
The bottom line is, is that I get a huge multiline error that starts with
fatal error: concurrent map writes
When I run the code. Which I thought I guarded for through mutual exclusion
l.Lock()
occurrences[word]++
l.Unlock()
What am I doing wrong here? And furthermore. How can I combine all the maps in a channel? And with combine I mean same key's values get summed in the new map.
The main problem is that you use a separate lock in each goroutine. That doesn't do any help to serialize access to the map. The same lock has to be used in each goroutine.
And since you use the same map in each goroutine, you don't have to merge them, and you don't need a channel to deliver the result.
Even if you use the same mutex in each goroutine, since you use a single map, this probably won't help in performance, the goroutines will have to compete with each other for the map's lock.
You should create a separate map in each goroutine, use that to count locally, and then deliver the result map on the channel. This might give you a performance boost.
But then you don't need a lock, since each goroutine will have its own map which it can read/write without a mutex.
But then you'll do have to deliver the result on the channel, and then merge it.
And since goroutines deliver results on the channel, the waitgroup becomes unnecessary.
func WordCount(text string) map[string]int {
s := strings.Fields(text)
channel := make(chan map[string]int, 2)
go mappers(s[0:(len(s)/2)], channel)
go mappers(s[(len(s)/2):], channel)
total := map[string]int{}
for i := 0; i < 2; i++ {
m := <-channel
for k, v := range m {
total[k] += v
}
}
return total
}
func mappers(slice []string, ch chan map[string]int) {
occurrences := map[string]int{}
for _, word := range slice {
occurrences[word]++
}
ch <- occurrences
}
Example testing it:
fmt.Println(WordCount("aa ab cd cd de ef a x cd aa"))
Output (try it on the Go Playground):
map[a:1 aa:2 ab:1 cd:3 de:1 ef:1 x:1]
Also note that in theory this looks "good", but in practice you may still not achieve any performance boost, as the goroutines do too "little" work, and launching them and merging the results requires effort which may outweight the benefits.
I am new to go and I am trying to understand the way channels in goroutines work. To my understanding, the keyword range could be used to iterate over a the values of the channel up until the channel is closed or the buffer runs out; hence, a for range c will repeatedly loops until the buffer runs out.
I have the following simple function that adds value to a channel:
func main() {
c := make(chan int)
go printchannel(c)
for i:=0; i<10 ; i++ {
c <- i
}
}
I have two implementations of printchannel and I am not sure why the behaviour is different.
Implementation 1:
func printchannel(c chan int) {
for range c {
fmt.Println(<-c)
}
}
output: 1 3 5 7
Implementation 2:
func printchannel(c chan int) {
for i:=range c {
fmt.Println(i)
}
}
output: 0 1 2 3 4 5 6 7 8
And I was expecting neither of those outputs!
Wanted output: 0 1 2 3 4 5 6 7 8 9
Shouldnt the main function and the printchannel function run on two threads in parallel, one adding values to the channel and the other reading the values up until the channel is closed? I might be missing some fundamental go/thread concept here and pointers to that would be helpful.
Feedback on this (and my understanding to channels manipulation in goroutines) is greatly appreciated!
Implementation 1. You're reading from the channel twice - range c and <-c are both reading from the channel.
Implementation 2. That's the correct approach. The reason you might not see 9 printed is that two goroutines might run in parallel threads. In that case it might go like this:
main goroutine sends 9 to the channel and blocks until it's read
second goroutine receives 9 from the channel
main goroutine unblocks and exits. That terminates whole program which doesn't give second goroutine a chance to print 9
In case like that you have to synchronize your goroutines. For example, like so
func printchannel(c chan int, wg *sync.WaitGroup) {
for i:=range c {
fmt.Println(i)
}
wg.Done() //notify that we're done here
}
func main() {
c := make(chan int)
wg := sync.WaitGroup{}
wg.Add(1) //increase by one to wait for one goroutine to finish
//very important to do it here and not in the goroutine
//otherwise you get race condition
go printchannel(c, &wg) //very important to pass wg by reference
//sync.WaitGroup is a structure, passing it
//by value would produce incorrect results
for i:=0; i<10 ; i++ {
c <- i
}
close(c) //close the channel to terminate the range loop
wg.Wait() //wait for the goroutine to finish
}
As to goroutines vs threads. You shouldn't confuse them and probably should understand the difference between them. Goroutines are green threads. There're countless blog posts, lectures and stackoverflow answers on that topic.
In implementation 1, range reads into channel once, then again in Println. Hence you're skipping over 2, 4, 6, 8.
In both implementations, once the final i (9) has been sent to goroutine, the program exits. Thus goroutine does not have the time to print out 9. To solve it, use a WaitGroup as has been mentioned in the other answer, or a done channel to avoid semaphore/mutex.
func main() {
c := make(chan int)
done := make(chan bool)
go printchannel(c, done)
for i:=0; i<10 ; i++ {
c <- i
}
close(c)
<- done
}
func printchannel(c chan int, done chan bool) {
for i := range c {
fmt.Println(i)
}
done <- true
}
The reason your first implementation only returns every other number is because you are, in effect "taking" from c twice each time the loop runs: first with range, then again with <-. It just happens that you're not actually binding or using the first value taken off the channel, so all you end up printing is every other one.
An alternative approach to your first implementation would be to not use range at all, e.g.:
func printchannel(c chan int) {
for {
fmt.Println(<-c)
}
}
I could not replicate the behavior of your second implementation, on my machine, but the reason for that is that both of your implementations are racy - they will terminate whenever main ends, regardless of what data may be pending in a channel or however many goroutines may be active.
As a closing note, I'd warn you not to think about goroutines as explicitly being "threads", though they have a similar mental model and interface. In a simple program like this it's not at all unlikely that Go might just do it all using a single OS thread.
Your first loop does not work as you have 2 blocking channel receivers and they do not execute at the same time.
When you call the goroutine the loop starts, and it waits for the first value to be sent to the channel. Effectively think of it as <-c .
When the for loop in the main function runs it sends 0 on the Chan. At this point the range c recieves the value and stops blocking the execution of the loop.
Then it is blocked by the reciever at fmt.println(<-c) . When 1 is sent on the second iteration of the loop in main the recieved at fmt.println(<-c) reads from the channel, allowing fmt.println to execute thus finishing the loop and waiting for a value at the for range c .
Your second implementation of the looping mechanism is the correct one.
The reason it exits before printing to 9 is that after the for loop in main finishes the program goes ahead and completes execution of main.
In Go func main is launched as a goroutine itself while executing. Thus when the for loop in main completes it goes ahead and exits, and as the print is within a parallel goroutine that is closed, it is never executed. There is no time for it to print as there is nothing to block main from completing and exiting the program.
One way to solve this is to use wait groups http://www.golangprograms.com/go-language/concurrency.html
In order to get the expected result you need to have a blocking process running in main that provides enough time or waits for confirmation of the execution of the goroutine before allowing the program to continue.
This question already has answers here:
Multiple goroutines listening on one channel
(7 answers)
Closed 5 years ago.
I understand the regular behavior of a channel is that it empties after a read. Is there a way to keep an unbuffered channel value for multiple reads without the value been removed from the channel?
For example, I have a goroutine that generates a single data for multiple down stream go routines to use. I don't want to have to create multiple channels or use a buffered channel which would require me to duplicate the source data (I don't even know how many copies I will need). Effectively, I want to be able to do something like the following:
main{
ch := make(ch chan dType)
ch <- sourceDataGenerator()
for _,_ := range DynamicRange{
go TargetGoRoutine(ch)
}
close(ch) // would want this to remove the value and the channel
}
func(ch chan dType) TargetGoRoutine{
targetCollection <- ch // want to keep the channel value after read
}
EDIT
Some feel this is a duplicate question. Perhaps, but not sure. The solution here seems simple in the end as n-canter pointed out. All it needs is for every go routine to "recycle" the data by putting it back to the channel after use. None of the supposedly "duplicates" provided this solution. Here is a sample:
package main
import (
"fmt"
"sync"
)
func main() {
c := make(chan string)
var wg sync.WaitGroup
wg.Add(5)
for i := 0; i < 5; i++ {
go func(i int) {
wg.Done()
msg := <-c
fmt.Printf("Data:%s, From go:%d\n", msg, i)
c <-msg
}(i)
}
c <- "Original"
wg.Wait()
fmt.Println(<-c)
}
https://play.golang.org/p/EXBbf1_icG
You may readd value back to the channel after reading, but then all your gouroutines will read shared value sequentially and also you'll need some synchronization primitives for last goroutine not to block.
As far as I know the only case when you can use the single channel for broadcasting is closing it. In this case all readers will be notified.
If you don't want to duplicate large data, maybe you'd better use some global variable. But use it carefully, because it violates golang rule: "Don't communicate by sharing memory; share memory by communicating."
Also look at this question How to broadcast message using channel
Golang newbie here. Can somebody explain why the following code generates a deadlock?
I am aware of sending true to boolean <- done channel but I don't want to use it.
package main
import (
"fmt"
"sync"
"time"
)
var wg2 sync.WaitGroup
func producer2(c chan<- int) {
for i := 0; i < 5; i++ {
time.Sleep(time.Second * 10)
fmt.Println("Producer Writing to chan %d", i)
c <- i
}
}
func consumer2(c <-chan int) {
defer wg2.Done()
fmt.Println("Consumer Got value %d", <-c)
}
func main() {
c := make(chan int)
wg2.Add(5)
fmt.Println("Starting .... 1")
go producer2(c)
go consumer2(c)
fmt.Println("Starting .... 2")
wg2.Wait()
}
Following is my understanding and I know that it is wrong:
The channel will be blocked the moment 0 is written to it within the
loop of producer function
So I expect channel to be emptied by the
consumer afterwards.
As the channel is emptied in the step 2,
producer function can again put in another value and then get
blocked and steps 2 repeats again.
Your original deadlock is caused by wg2.Add(5), you were waiting for 5 goroutines to finish, but only one did; you called wg2.Done() once. Change this to wg2.Add(1), and your program will run without error.
However, I suspect that you intended to consume all the values in the channel not just one as you do. If you change consumer function to:
func consumer2(c <-chan int) {
defer wg2.Done()
for i := range c {
fmt.Printf("Consumer Got value %d\n", i)
}
}
You will get another deadlock because the channel is not closed in producer function, and consumer is waiting for more values that never arrive. Adding close(c) to the producer function will fix it.
Why it error?
Running your code gets the following error:
➜ gochannel go run dl.go
Starting .... 1
Starting .... 2
Producer Writing to chan 0
Consumer Got value 0
Producer Writing to chan 1
fatal error: all goroutines are asleep - deadlock!
Here is why:
There are three goroutines in your code: main,producer2 and consumer2. When it runs,
producer2 sends a number 0 to the channel
consumer2 recives 0 from the channel, and exits
producer2 sends 1 to the channel, but no one is consuming, since consumer2 already exits
producer2 is waiting
main executes wg2.Wait(), but not all waitgroup are closed. So main is waiting
Two goroutines are waiting here, does nothing, and nothing will be done no matter how long you wait. It is a deadlock! Golang detects it and panic.
There are two concepts you are confused here:
how waitgourp works
how to receive all values from a channel
I'll explain them here briefly, there are alreay many articles out there on the internet.
how waitgroup works
WaitGroup if a way to wait for all groutine to finish. When running goroutines in the background, it's important to know when all of them quits, then certain action can be conducted.
In your case, we run two goroutines, so at the beginning we should set wg2.Add(2), and each goroutine should add wg2.Done() to notify it is done.
Receive data from a channel
When receiving data from a channel. If you know exactly how many data it will send, use for loop this way:
for i:=0; i<N; i++ {
data = <-c
process(data)
}
Otherwise use it this way:
for data := range c {
process(data)
}
Also, Don't forget to close channel when there is no more data to send.
How to fix it?
With the above explanation, the code can be fixed as:
package main
import (
"fmt"
"sync"
"time"
)
var wg2 sync.WaitGroup
func producer2(c chan<- int) {
defer wg2.Done()
for i := 0; i < 5; i++ {
time.Sleep(time.Second * 1)
fmt.Printf("Producer Writing to chan %d\n", i)
c <- i
}
close(c)
}
func consumer2(c <-chan int) {
defer wg2.Done()
for i := range c {
fmt.Printf("Consumer Got value %d\n", i)
}
}
func main() {
c := make(chan int)
wg2.Add(2)
fmt.Println("Starting .... 1")
go producer2(c)
go consumer2(c)
fmt.Println("Starting .... 2")
wg2.Wait()
}
Here is another possible way to fix it.
The expected output
Fixed code gives the following output:
➜ gochannel go run dl.go
Starting .... 1
Starting .... 2
Producer Writing to chan 0
Consumer Got value 0
Producer Writing to chan 1
Consumer Got value 1
Producer Writing to chan 2
Consumer Got value 2
Producer Writing to chan 3
Consumer Got value 3
Producer Writing to chan 4
Consumer Got value 4
I'm rather new in golang, and trying to understand main principles and write gouroutines based code, using chanels.
In others langs that i was using there were no such instruments, and i wonder getting such an errors like panic...
my code:
package main
import "fmt"
import (
"time"
)
type Work struct {
x,y,z int
}
func worker(in <-chan *Work, out chan<- *Work){
for w := range in {
w.z = w.x + w.y
time.Sleep(time.Duration(w.z))
out <-w
}
}
func sendWork(in chan <- *Work){
var wo *Work
wo.x, wo.y, wo.z = 1,2,3
in <- wo
in <- wo
in <- wo
in <- wo
in <- wo
}
func receiveWork(out <-chan *Work ) []*Work{
var slice []*Work
for el := range out {
slice = append(slice, el)
}
return slice
}
func main() {
in, out := make(chan *Work), make(chan *Work)
for i := 0; i<3; i++{
go worker(in, out)
}
go sendWork(in)
data := receiveWork(out)
fmt.Printf("%v", data)
}
But at the terminal i got this :
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x1 addr=0x0 pc=0x401130]
goroutine 8 [running]:
main.sendWork(0xc0820101e0)
C:/temp/gocode/src/helloA/helloA.go:21 +0x20
created by main.main
C:/temp/gocode/src/helloA/helloA.go:43 +0xe4
goroutine 1 [chan receive]:
main.receiveWork(0xc082010240, 0x0, 0x0, 0x0)
C:/temp/gocode/src/helloA/helloA.go:31 +0x80
main.main()
C:/temp/gocode/src/helloA/helloA.go:45 +0xf2
goroutine 5 [chan receive]:
main.worker(0xc0820101e0, 0xc082010240)
C:/temp/gocode/src/helloA/helloA.go:12 +0x55
created by main.main
C:/temp/gocode/src/helloA/helloA.go:40 +0xaf
goroutine 6 [runnable]:
main.worker(0xc0820101e0, 0xc082010240)
C:/temp/gocode/src/helloA/helloA.go:11
created by main.main
C:/temp/gocode/src/helloA/helloA.go:40 +0xaf
goroutine 7 [runnable]:
main.worker(0xc0820101e0, 0xc082010240)
C:/temp/gocode/src/helloA/helloA.go:11
created by main.main
C:/temp/gocode/src/helloA/helloA.go:40 +0xaf
How can i determine where is the problem, and how can i close the gouroutines nicely, not to left them as processes...
p.s. Forgive me my noob questions. please
The nil dereference:
You are trying to access a struct referenced by a pointer, but that pointer has not been set to an instance of that struct. You have to declare a struct that you can point the pointer to.
The error first appears here:
wo.x, wo.y, wo.z = 1,2,3
where you try to write to the object pointed to by wo. But the pointer is nil here; it's not actually pointing to an instance of Work. We have to create that instance, so we can point to it.
The nil value for a pointer to a struct is nil. If you don't declare a struct instance for it to point to, it points to nil.
var wo *Work
declares wo as a pointer of type Work to nil.
var wo = &Work{}
declares wo as a pointer of type Work to a new instance of Work.
Or you can use the shorter syntax:
wo := &Work{}
As for the deadlock:
When we close a channel, range loops over that channel will exit. In func worker we range over a channel. When this channel is closed, the worker(s) will exit.
In order to wait for all the workers to finish processing, we use a sync.WaitGroup. This is an easy way to wait for a group of goroutines to finish running before continuing.
First you tell the waitgroup how many goroutines it should wait for.
wg.Add(3)
Then you wait:
wg.Wait()
until all of the goroutines have called
wg.Done()
which they do when they are done executing.
In this case, we need to close the output channel when all workers are done executing, so that the func receiveWork can exit its for range loop. We can do this by starting a new goroutine for this task:
go func() {
wg.Wait()
close(out)
}()
This is the whole file, after these edits:
package main
import (
"fmt"
"sync"
"time"
)
type Work struct {
x, y, z int
}
func worker(in <-chan *Work, out chan<- *Work, wg *sync.WaitGroup) {
for w := range in {
w.z = w.x + w.y
time.Sleep(time.Duration(w.z))
out <- w
}
wg.Done() // this worker is now done; let the WaitGroup know.
}
func sendWork(in chan<- *Work) {
wo := &Work{x: 1, y: 2, z: 3} // more compact way of initializing the struct
in <- wo
in <- wo
in <- wo
in <- wo
in <- wo
close(in) // we are done sending to this channel; close it
}
func receiveWork(out <-chan *Work) []*Work {
var slice []*Work
for el := range out {
slice = append(slice, el)
}
return slice
}
func main() {
var wg sync.WaitGroup
in, out := make(chan *Work), make(chan *Work)
wg.Add(3) // number of workers
for i := 0; i < 3; i++ {
go worker(in, out, &wg)
}
go sendWork(in)
go func() {
wg.Wait()
close(out)
}()
data := receiveWork(out)
fmt.Printf("%v", data)
}
Which outputs:
[0x104382f0 0x104382f0 0x104382f0 0x104382f0 0x104382f0]
which is probably not what you expected. It does however highlight one problem with this code. More on that later.
If you want to print the contents of the structs, you could either stop using pointers to Work, or loop over the elements of the slice and print them one-by-one, like this:
for _, w := range data {
fmt.Printf("%v", w)
}
which outputs:
&{1 2 3}&{1 2 3}&{1 2 3}&{1 2 3}&{1 2 3}
Go doesn't follow pointers more than one step down when printing, to avoid infinite recursion, so you have to do this manually.
Race condition:
Since you are sending pointers to the same instance of *Work multiple times down the channel, that same instance is being accessed by multiple goroutines at the same time without synchronization. What you probably want is to stop using pointers, and use values. Work instead of *Work.
If you want to use pointers, maybe because Work is actually really big, you probably want to make multiple instances of *Work so you only ever send it to one goroutine.
Here's what the go race detector has to say about the code:
C:/Go\bin\go.exe run -race C:/gopath/src/github.com/drathier/scratchpad/go/main/main.go
[0xc0820403c0 0xc0820403c0 0xc0820403c0 0xc0820403c0 0xc0820403c0]==================
WARNING: DATA RACE
Write by goroutine 6:
main.worker()
C:/gopath/src/github.com/drathier/scratchpad/go/main/main.go:15 +0x8a
Previous write by goroutine 8:
main.worker()
C:/gopath/src/github.com/drathier/scratchpad/go/main/main.go:15 +0x8a
Goroutine 6 (running) created at:
main.main()
C:/gopath/src/github.com/drathier/scratchpad/go/main/main.go:45 +0x10c
Goroutine 8 (running) created at:
main.main()
C:/gopath/src/github.com/drathier/scratchpad/go/main/main.go:45 +0x10c
==================
Found 1 data race(s)
exit status 66
At this row:
w.z = w.x + w.y
all goroutines are modifying w.z concurrently, so if they try to write different values to w.z, there's no telling what value actually end up in there. Once again, this is easily fixed by creating multiple instances of *Work, or by using values instead of pointers: Work.