I would need help to understand why the following code does not work. I am building a pipeline and trying to have a step that synchronize values from two source channels. My source/producer code looks something like below (in my real code i read the text from a file). The sources are sorted but are values are not guaranteed to be in both sources.
func Source() <-chan int{
out := make(chan int, 5)
go func() {
defer reader.Close()
out <- 1
out <- 2
out <- 3
out <- 4
out <- 5
out <- 7
close(out)
}()
return out
}
and the synchronization code looks like this:
func Sync(a, b <-chan int) <-chan int {
out := make(chan int)
go func() {
av, ak:= <-a
bv, bk:= <-b
for ak || bk {
if !ak || av < bv {
out <- bv
bv, bk = <-b
continue
}
if !bk|| bv > av {
out <- av
av, ak = <-a
continue
}
out <- av
av, ak = <-a
bv, bk = <-b
}
close(out)
}()
return out
}
and my program looks something like this:
func main() {
os := Source()
ns := Source()
for val := range Sync(ns, os) {
fmt.Printf("[SYNCED] %v \n", val)
}
}
The expected behaviour is that my both sources buffer values into the channel and my sync first reads value from the first source. Then from the second. Compare them and if they are equal continues to the next in the both channels. If the differ we will send out the value that is behind and replace it with a new one and make the same comparison again.
What happends is that it looks like the sync code is run several times for the values and I will get things like [SYNCED] 1 several times. Why?
Please help me get this fixed!
Regarding http://play.golang.org/p/uhd3EWrwEo and http://play.golang.org/p/Dqq7-cPaFq -
Actually, the code for ints will fail with the similar test case as well:
os := Source([]int{1, 2, 3})
ns := Source([]int{1, 3, 4})
puts the ints version to infinite loop.
This happens because when !aok || avalue > bvalue is checked, it does not take into account that if aok is true (some elements still are in a) and bok is false (no more elements in b), then avalue > "" is always true. So it tries to take another item from b (which is empty) and goes to infinite loop. Fixed code: http://play.golang.org/p/vYhuOZxRMl
Related
I'm parsing a cron string from AWS that looks like this cron(0 7 13 November ? 2019). Is there a clean way to go from November back to 11 using Go's built in types? The time.Month type allows mapping int to string, but there doesn't seem to be a way to do the reverse. Am I missing something? For now, I've written this to get a map[string]int that I'm using like this: monthi := getMonths()[monthName].
func getMonths() map[string]int {
m := make(map[string]int)
for i := 1; i < 13; i++ {
month := time.Month(i).String()
m[month] = i
}
return m
}
Foreword: I released this utility in github.com/icza/gox, see timex.ParseMonth().
That is currently the best approach.
Best would be to use a package level variable and populate the map only once.
And it's much cleaner and safer to do the population this way:
var months = map[string]time.Month{}
func init() {
for i := time.January; i <= time.December; i++ {
months[i.String()] = i
}
}
Testing it:
for _, s := range []string{"January", "December", "invalid"} {
m := months[s]
fmt.Println(int(m), m)
}
Output (try it on the Go Playground):
1 January
12 December
0 %!Month(0)
Note that this map has the flexibility that you may add short month names, mapping to the same month. E.g. you may also add months["Jan"] = time.January, so if your input is "Jan", you would also be able to get time.January. This could easily be done by slicing the long name, in the same loop, for example:
for i := time.January; i <= time.December; i++ {
name := i.String()
months[name] = i
months[name[:3]] = i
}
Also note that it's possible to use time.Parse() to do the parsing where the layout string is "January":
for _, s := range []string{"January", "December", "invalid"} {
t, err := time.Parse("January", s)
m := t.Month()
fmt.Println(int(m), m, err)
}
Which outputs (try it on the Go Playground):
1 January <nil>
12 December <nil>
1 January parsing time "invalid" as "January": cannot parse "invalid" as "January"
But the simple map lookup is superior to this in performance.
See similar question: Parse Weekday string into time.Weekday
My questions is how to schedule running independent non-blocking functions every interval N.
My initial approach is to use go channels within a select statement to receive the values in a non-blocking manner and use time.Sleep(N) in each function to schedule the call.
In the code snippet below, this only for the first run; however, after the first call, it keeps calling computeY() repeatedly without respecting the time.Sleep() call.
package main
import (
"fmt"
"time"
)
var (
x string = ""
y string = ""
)
func computeY(c chan string) {
time.Sleep(10 * time.Second)
fmt.Println("I'm in Y")
y = "this is Y value"
c <- y
}
func computeX(c chan string) {
time.Sleep(1 * time.Second)
x = "this is X value"
c <- x
}
func main() {
xC := make(chan string)
yC := make(chan string)
for {
go computeX(xC)
go computeY(yC)
select {
case x := <-xC:
fmt.Println(fmt.Sprintf("X: %v, Y: %v", x, y))
case y := <-yC:
fmt.Println(fmt.Sprintf("X: %v, Y: %v", x, y))
}
}
}
You are calling both computeX and computeY every iteration of the loop.
Since computeX takes 1s, the for loop iterates once per second and an extra time when yC gets a value.
This means that you're running go computeY at t=0s, t=1s, t=2s, etc....
The first one terminates at t=10s, the second at t=11s, etc...
If you want to make sure you only schedule one computeX and computeY at a time, you need to change your main to something along the lines of:
go computeX(xC)
go computeY(yC)
for {
select {
case x = <-xC:
fmt.Printf("Finished computeX: X: %v, Y: %v\n", x, y)
go computeX(xC)
case y = <-yC:
fmt.Printf("Finished computeY: X: %v, Y: %v\n", x, y)
go computeY(yC)
}
}
A few other things to note about your code:
x and y are global and assigned in computeX and computeY
your channel reads shadow x and y
fmt.Println(fmt.Sprintf("...")) is just fmt.Printf("...\n")
you don't need to initialize strings to "", that's the default value
While #Marc's answer explains your code's problem and shows how to fix it, I will try to give you some patterns on scheduling functions.
Pattern 1: time.Ticker
A Ticker holds a channel that delivers `ticks' of a clock at intervals.
Example:
func Schedule(interval time.Duration,f func()) {
t:=time.NewTimer(interval)
go func() {
for {
<-t.C
f()
}
}()
}
Pattern 2: time.AfterFunc
AfterFunc waits for the duration to elapse and then calls f in its own goroutine. It returns a Timer that can be used to cancel the call using its Stop method.
Example:
func Schedule(interval time.Duration,f func()) {
var wrap func()
wrap = func() {
f()
time.AfterFunc(wrap)
}
time.AfterFunc(f)
}
Pattern 1 is more readable and expressive while pattern 2 is more efficient on memory.
I'm creating a simple channel that takes string values. But apparently I'm pushing each letter in the string instead of the whole string in each loop.
I'm probably missing something very fundamental. What am I doing wrong ?
https://play.golang.org/p/-6E-f7ALbD
Code:
func doStuff(s string, ch chan string) {
ch <- s
}
func main() {
c := make(chan string)
loops := [5]int{1, 2, 3, 4, 5}
for i := 0; i < len(loops); i++ {
go doStuff("helloooo", c)
}
results := <-c
fmt.Println("channel size = ", len(results))
// print the items in channel
for _, r := range results {
fmt.Println(string(r))
}
}
Your code sends strings on the channel properly:
func doStuff(s string, ch chan string){
ch <- s
}
The problem is at the receiver side:
results := <- c
fmt.Println("channel size = ", len(results))
// print the items in channel
for _,r := range results {
fmt.Println(string(r))
}
results will be a single value received from the channel (the first value sent on it). And you print the length of this string.
Then you loop over this string (results) using a for range which loops over its runes, and you print those.
What you want is loop over the values of the channel:
// print the items in channel
for s := range c {
fmt.Println(s)
}
This when run will result in a runtime panic:
fatal error: all goroutines are asleep - deadlock!
Because you never close the channel, and a for range on a channel runs until the channel is closed. So you have to close the channel sometime.
For example let's wait 1 second, then close it:
go func() {
time.Sleep(time.Second)
close(c)
}()
This way your app will run and quit after 1 second. Try it on the Go Playground.
Another, nicer solution is to use sync.WaitGroup: this waits until all goroutines are done doing their work (sending a value on the channel), then it closes the channel (so there is no unnecessary wait / delay).
var wg = sync.WaitGroup{}
func doStuff(s string, ch chan string) {
ch <- s
wg.Done()
}
// And in main():
for i := 0; i < len(loops); i++ {
wg.Add(1)
go doStuff("helloooo", c)
}
go func() {
wg.Wait()
close(c)
}()
Try this one on the Go Playground.
Notes:
To repeat something 5 times, you don't need that ugly loops array. Simply do:
for i := 0; i < 5; i++ {
// Do something
}
The reason you are getting back the letters instead of string is that you are assigning the channel result to a variable and iterating over the result of the channel assigned to this variable which in your case is a string, and in Go you can iterate over a string with a for range loop to get the runes.
You can simply print the channel without to iterate over the channel result.
package main
import (
"fmt"
)
func doStuff(s string, ch chan string){
ch <- s
}
func main() {
c := make(chan string)
loops := [5]int{1,2,3,4,5}
for i := 0; i < len(loops) ; i++ {
go doStuff("helloooo", c)
}
results := <- c
fmt.Println("channel size = ", len(results))
fmt.Println(results) // will print helloooo
}
I am writing a program that calculates a Riemann sum based on user input. The program will split the function into 1000 rectangles (yes I know I haven't gotten that math in there yet) and sum them up and return the answer. I am using go routines to compute the 1000 rectangles but am getting an
fatal error: all go routines are asleep - deadlock!
What is the correct way to handle multiple go routines? I have been looking around and haven't seen an example that resembles my case? I'm new and want to adhere to standards. Here is my code (it is runnable if you'd like to see what a typical use case of this is - however it does break)
package main
import "fmt"
import "time"
//Data type to hold 'part' of function; ie. "4x^2"
type Pair struct {
coef, exp int
}
//Calculates the y-value of a 'part' of the function and writes this to the channel
func calc(c *chan float32, p Pair, x float32) {
val := x
//Raise our x value to the power, contained in 'p'
for i := 1; i < p.exp; i++ {
val = val * val
}
//Read existing answer from channel
ans := <-*c
//Write new value to the channel
*c <- float32(ans + (val * float32(p.coef)))
}
var c chan float32 //Channel
var m map[string]Pair //Map to hold function 'parts'
func main() {
c = make(chan float32, 1001) //Buffered at 1001
m = make(map[string]Pair)
var counter int
var temp_coef, temp_exp int
var check string
var up_bound, low_bound float32
var delta float32
counter = 1
check = "default"
//Loop through as long as we have no more function 'parts'
for check != "n" {
fmt.Print("Enter the coefficient for term ", counter, ": ")
fmt.Scanln(&temp_coef)
fmt.Print("Enter the exponent for term ", counter, ": ")
fmt.Scanln(&temp_exp)
fmt.Print("Do you have more terms to enter (y or n): ")
fmt.Scanln(&check)
fmt.Println("")
//Put data into our map
m[string(counter)] = Pair{temp_coef, temp_exp}
counter++
}
fmt.Print("Enter the lower bound: ")
fmt.Scanln(&low_bound)
fmt.Print("Enter the upper bound: ")
fmt.Scanln(&up_bound)
//Calculate the delta; ie. our x delta for the riemann sum
delta = (float32(up_bound) - float32(low_bound)) / float32(1000)
//Make our go routines here to add
for i := low_bound; i < up_bound; i = i + delta {
//'counter' is indicative of the number of function 'parts' we have
for j := 1; j < counter; j++ {
//Go routines made here
go calc(&c, m[string(j)], i)
}
}
//Wait for the go routines to finish
time.Sleep(5000 * time.Millisecond)
//Read the result?
ans := <-c
fmt.Print("Answer: ", ans)
}
It dead locks because both the calc() and the main() function reads from the channel before anyone gets to write to it.
So you will end up having every (non-main) go routine blocking at:
ans := <-*c
waiting for someone other go routine to enter a value into the channel. There fore none of them gets to the next line where they actually write to the channel. And the main() routine will block at:
ans := <-c
Everyone is waiting = deadlock
Using buffered channels
Your solution should have the calc() function only writing to the channel, while the main() could read from it in a for-range loop, suming up the values coming from the go-routines.
You will also need to add a way for main() to know when there will be no more values arriving, perhaps by using a sync.WaitGroup (maybe not the best, since main isn't suppose to wait but rather sum things up) or an ordinary counter.
Using shared memory
Sometimes it is not necessarily a channel you need. Having a shared value that you update with the sync/atomic package (atomic add doesn't work on floats) lock with a sync.Mutex works fine too.
First function
ReadF2C
takes a filename and channel, reads from file and inputs in channel.
Second function
WriteC2F
takes 2 channels and filename, takes value of each channel and saves the lower value in the output file. I'm sure there is a few syntax errors but i'm new to GO
package main
import (
"fmt"
"bufio"
"os"
"strconv"
)
func main() {
fmt.Println("Hello World!\n\n")
cs1 := make (chan int)
var nameinput string = "input.txt"
readF2C(nameinput,cs1)
cs2 := make (chan int)
cs3 := make (chan int)
cs2 <- 10
cs2 <- 16
cs2 <- 7
cs2 <- 2
cs2 <- 5
cs3 <- 8
cs3 <- 15
cs3 <- 14
cs3 <- 1
cs3 <- 6
var nameoutput string = "output.txt"
writeC2F (nameoutput,cs2,cs3)
}
func readF2C (fn string, ch chan int){
f,err := os.Open(fn)
r := bufio.NewReader(f)
for err != nil { // not end of file
fmt.Println(r.ReadString('\n'))
ch <- r.ReadString('\n')
}
if err != nil {
fmt.Println(r.ReadString('\n'))
ch <- -1
}
}
func writeC2F(fn string, // output text file
ch1 chan int, // first input channel
ch2 chan int){
var j int = 0
var channel1temp int
var channel2temp int
f,_ := os.Create(fn)
w := bufio.NewWriter(f)
channel1temp = <-ch1
channel2temp = <-ch2
for j := 1; j <= 5; j++ {
if (channel2temp < channel1temp){
n4, err := w.WriteString(strconv.Itoa(channel1temp))
} else{
n4, err := w.WriteString(strconv.Itoa(channel2temp))
}
w.flush()
}
}
This is the error messages I get:
prog.go:38: multiple-value r.ReadString() in single-value context
prog.go:65: w.flush undefined (cannot refer to unexported field or method bufio.(*Writer)."".flush)
There are multiple errors:
1)
Unlike C, Go enforces you to have your curly braces directly after your statements. So for an if case (and the same for func), instead of doing it like this:
if (channel2temp < channel1temp)
{
use this
if channel2temp < channel1temp {
2)
There is no while in Go. Use for
for {
...
}
or
for channel1temp != null || channel2temp != null {
...
}
3)
Usage of non-declared variables. Often easy to fix by making a short variable declaration the first time you initialize the variable. So instead of:
r = bufio.NewReader(file)
use
r := bufio.NewReader(file)
4)
Trying to a assign multi-value return into a single variable. If a function returns two values and you only need one, the variable you don't want can be discarded by assigning it to _. So instead of:
file := os.Open(fn)
use
file, _ := os.Open(fn)
but best practice would be to catch that error:
file, err := os.Open(fn)
if err != nil {
panic(err)
}
There are more errors on top of this, but maybe it will get you started.
I also suggest reading Effective Go since it will explain many of the things I've just mentioned.
Edit:
And there are help online for sure. It might be a new language, but the online material is really useful. Below is a few that I used when learning Go:
Effective Go: Good document on how to write idiomatic Go code
The Go programming language Tour: Online tour of Go with interactive examples.
Go By Example: Interactive examples of Go programs, starting with Hello World.
Go Specification: Surprisingly readable for being a specification. Maybe not a start point, but very useful.