Dining philosophers problem in Go fails unit test - multithreading

I'm taking a Go course, that has an assignment as follows:
Implement the dining philosopher's problem with the following
constraints/modifications.
There should be 5 philosophers sharing chopsticks, with one chopstick between each adjacent pair of philosophers.
Each philosopher should eat only 3 times (not in an infinite loop as we did in lecture).
The philosophers pick up the chopsticks in any order, not lowest-numbered first (which we did in lecture).
In order to eat, a philosopher must get permission from a host which executes in its own goroutine.
The host allows no more than 2 philosophers to eat concurrently.
Each philosopher is numbered, 1 through 5.
When a philosopher starts eating (after it has obtained necessary
locks) it prints "starting to eat " on a line by itself, where
is the number of the philosopher.
When a philosopher finishes eating (before it has released its locks)
it prints "finishing eating " on a line by itself, where
is the number of the philosopher.
My implementation:
package main
import (
"fmt"
"io"
"math/rand"
"os"
"sync"
"time"
)
const (
NumPhilosophers = 5
NumEatMaxTimes = 3
NumMaxAllowedToEat = 2
)
type chopstick struct{ sync.Mutex }
type philosopher struct {
num int
cs []*chopstick
}
func setTable() []*philosopher {
cs := make([]*chopstick, NumPhilosophers)
for i := 0; i < NumPhilosophers; i++ {
cs[i] = new(chopstick)
}
ph := make([]*philosopher, NumPhilosophers)
for i := 0; i < NumPhilosophers; i++ {
ph[i] = &philosopher{i + 1, []*chopstick{cs[i], cs[(i+1)%NumPhilosophers]}}
}
return ph
}
func (ph philosopher) eat(sem chan int, wg *sync.WaitGroup, w io.Writer) {
for i := 0; i < NumEatMaxTimes; i++ {
/* Ask host for permission to eat */
sem <- 1
/*
Pick any of the left or right chopsticks.
Notice how the methods on the Mutex can be called directly on a chopstick due to embedding.
*/
firstCS := rand.Intn(2)
secondCS := (firstCS + 1) % 2
ph.cs[firstCS].Lock()
ph.cs[secondCS].Lock()
fmt.Fprintf(w, "Starting to eat %d\n", ph.num)
x := rand.Intn(NumEatMaxTimes)
time.Sleep(time.Duration(x) * time.Second)
fmt.Fprintf(w, "Finishing eating %d\n", ph.num)
ph.cs[secondCS].Unlock()
ph.cs[firstCS].Unlock()
<-sem
}
wg.Done()
}
func main() {
run(os.Stdout)
}
func run(w io.Writer) {
var sem = make(chan int, NumMaxAllowedToEat)
rand.Seed(time.Now().UnixNano())
var wg sync.WaitGroup
allPh := setTable()
wg.Add(len(allPh))
for _, ph := range allPh {
go ph.eat(sem, &wg, w)
}
wg.Wait()
}
Unit test:
func TestRun(t *testing.T) {
var out bytes.Buffer
run(&out)
lines := strings.Split(strings.ReplaceAll(out.String(), "\r\n", "\n"), "\n")
eating := make(map[int]bool)
timesEaten := make(map[int]int)
for _, line := range lines {
if line == "" {
continue
}
fmt.Println(line)
tokens := strings.Fields(line)
i, err := strconv.Atoi(tokens[len(tokens)-1])
if err != nil {
t.Errorf("Bad line: %s", line)
}
s := strings.ToLower(tokens[0])
if s == "starting" {
if len(eating) > (NumMaxAllowedToEat - 1) {
t.Errorf("%v are eating at the same time", eating)
}
_, ok := eating[i]
if ok {
t.Errorf("%d started before finishing", i)
}
eating[i] = true
} else if s == "finishing" {
_, ok := eating[i]
if !ok {
t.Errorf("%d finished without starting", i)
}
delete(eating, i)
timesEaten[i] = timesEaten[i] + 1
}
}
for k, v := range timesEaten {
if v > NumEatMaxTimes {
t.Errorf("%d ate %d times", k, v)
}
}
if len(timesEaten) != NumPhilosophers {
t.Error("One or more didn't get to eat")
}
}
The problem is, the test randomly fails. Below is one execution (line numbers added):
1. Starting to eat 5
2. Starting to eat 2
3. Finishing eating 2
4. Finishing eating 5
5. Starting to eat 3
6. Starting to eat 1
7. Finishing eating 1
8. Finishing eating 3
9. Starting to eat 2
10. Starting to eat 4
11. Finishing eating 4
12. Starting to eat 5
13. Finishing eating 2
14. Finishing eating 5
15. Starting to eat 3
16. Finishing eating 3
17. Starting to eat 1
18. Finishing eating 4
19. Finishing eating 1
20. Starting to eat 5
21. Finishing eating 5
22. Starting to eat 3
23. Finishing eating 3
24. Starting to eat 4
25. Starting to eat 2
26. Finishing eating 2
27. Starting to eat 1
28. Finishing eating 4
29. Finishing eating 1
--- FAIL: TestRun (12.01s)
main_test.go:43: 4 finished without starting
Philosopher 4 has started on lines 10, and 24 and finished on lines 11, 18, and 28. Line 28 is unmatched, so the test correctly complains. However, I'm having a hard time finding the bug. Can you help?

Answering my own question, it turned out that byes.Buffer is not thread-safe. I ended up using go-fakeio library for the test as shown below.
s, err := fakeio.Stderr().Stdout().Do(run)
if err != nil {
t.Errorf("%v", err)
}
The rest of the test remains the same. main.run function no longer needs an io.Writer since the fakeio library replaces stderr and stdout.

Related

Go program slowing down when increasing number of goroutines

I'm doing a small project for my parallelism course and I have tried it with buffered channels, unbuffered channels, without channels using pointers to slices etc. Also, tried to optimize it as much as possible (not the current state) but I still get the same result: increasing number of goroutines (even by 1) slows down the whole program. Can someone please tell me what I'm doing wrong and is even parallelism enhancement possible in this situation?
Here is part of the code:
func main() {
rand.Seed(time.Now().UnixMicro())
numAgents := 2
fmt.Println("Please pick a number of goroutines: ")
fmt.Scanf("%d", &numAgents)
numFiles := 4
fmt.Println("How many files do you want?")
fmt.Scanf("%d", &numFiles)
start := time.Now()
numAssist := numFiles
channel := make(chan []File, numAgents)
files := make([]File, 0)
for i := 0; i < numAgents; i++ {
if i == numAgents-1 {
go generateFiles(numAssist, channel)
} else {
go generateFiles(numFiles/numAgents, channel)
numAssist -= numFiles / numAgents
}
}
for i := 0; i < numAgents; i++ {
files = append(files, <-channel...)
}
elapsed := time.Since(start)
fmt.Printf("Function took %s\n", elapsed)
}
func generateFiles(numFiles int, channel chan []File) {
magicNumbersMap := getMap()
files := make([]File, 0)
for i := 0; i < numFiles; i++ {
content := randElementFromMap(&magicNumbersMap)
length := rand.Intn(400) + 100
hexSlice := getHex()
for j := 0; j < length; j++ {
content = content + hexSlice[rand.Intn(len(hexSlice))]
}
hash := getSHA1Hash([]byte(content))
file := File{
content: content,
hash: hash,
}
files = append(files, file)
}
channel <- files
}
Expectation was that by increasing goroutines the program would run faster but to a certain number of goroutines and at that point by increasing goroutines I would get the same execution time or a little bit slower.
EDIT: All the functions that are used:
import (
"crypto/sha1"
"encoding/base64"
"fmt"
"math/rand"
"time"
)
type File struct {
content string
hash string
}
func getMap() map[string]string {
return map[string]string{
"D4C3B2A1": "Libcap file format",
"EDABEEDB": "RedHat Package Manager (RPM) package",
"4C5A4950": "lzip compressed file",
}
}
func getHex() []string {
return []string{
"0", "1", "2", "3", "4", "5",
"6", "7", "8", "9", "A", "B",
"C", "D", "E", "F",
}
}
func randElementFromMap(m *map[string]string) string {
x := rand.Intn(len(*m))
for k := range *m {
if x == 0 {
return k
}
x--
}
return "Error"
}
func getSHA1Hash(content []byte) string {
h := sha1.New()
h.Write(content)
return base64.URLEncoding.EncodeToString(h.Sum(nil))
}
Simply speaking - the files generation code is not complex enough to justify parallel execution. All the context switching and moving data through the channel eats all benefit of parallel processing.
If you add something like time.Sleep(time.Millisecond * 10) inside the loop in your generateFiles function as if it was doing something more complex, you'll see what you expected to see - more goroutines work faster. But again, only until certain level, when extra work to do parallel processing overweights the benefit.
Note also, the execution time of the last bit of your program:
for i := 0; i < numAgents; i++ {
files = append(files, <-channel...)
}
directly depends on number of goroutines. Since all goroutines finish approximately at the same time, this loop almost never executed in parallel with your workers and the time it takes to run is simply added to the total time.
Next, when you append to files slice multiple times, it has to grow several times and copy the data over to the new location. You can avoid this by initially creating a slice that will fil all your resulting elements (luckily, you know exactly how many you'll need).

Adding Map Elements to Slice in Sequential Orde

I'm working on a text wrapping function. I want it to break a long line of text into string slices of a maximum length of characters. I've got it mostly working. However, sometimes the words are placed out of order.
This happens when there is a long word followed by a short word. I believe the program sees the longer word will not fit on the line so it skips that word and adds in the next word that will fit.
As this is text, the words must stay in the correct order. How can I force the loop to only add words in the correct order?
Actual Output:
[]string{" Go back out of the hotel entrance and your is", " room on lower ground a private street", " entrance."}
Expected Output:
[]string{" Go back out of the hotel entrance and your", " room is on lower ground a private street", " entrance."}
This is what I have so far.
Link: https://play.golang.org/p/YsCWoM9hQJV
package main
import (
"fmt"
"strings"
)
func main() {
directions := "Go back out of the hotel entrance and your room is on the lower ground a private street entrance."
ws := strings.Split(directions, " ")
neededSlices := strings.Count(directions, "") / 48
if strings.Count(directions, "")%48 != 0 {
neededSlices++
}
ls := make([]string, neededSlices, neededSlices)
keys := make(map[string]bool)
for i := 0; i < len(ls); i++ {
for _, v := range ws {
if _, ok := keys[v]; !ok {
if strings.Count(ls[i], "")+strings.Count(v, "") <= 48 {
ls[i] = ls[i] + " " + v
keys[v] = true
}
}
}
}
fmt.Printf("%#v", ls)
}
I think this is simple implementation of what you need
package main
import (
"fmt"
"strings"
)
func main() {
directions := "Go back out of the hotel entrance and your room is on the lower ground a private street entrance."
ws := strings.Split(directions, " ")
sl := []string{""}
i := 0
for _,word := range ws {
if (len(sl[i]) + len(word) + 1) >=48 {
i++
sl = append(sl, "")
}
sl[i] += " " + word
}
fmt.Printf("%#v", sl)
}
Link: https://play.golang.org/p/7R2TS6lv4Tm
The first problem I notice is your usage of a map. A map can only contain a key once. Due to this, your code will only contain each word once in one of the output slices.
The second problem is that you iterate over the whole ws array again for each iteration of the ls slice. I guess you tried to work around this issue with the map?
The solution would be to iterate only once over ws and assign the words to the index in ls.
Also note that strings.Count returns the number of characters (unicode points) in the string plus 1.
Your code btw also adds a space at the beginning of each string in the slice. I am not sure if this is intended (your expected output matches this). In my example solution I deviate from that so my output does not 100% match your stated expected output but I think it gives a more expected result.
package main
import (
"fmt"
"strings"
)
func main() {
directions := "Go back out of the hotel entrance and your hotel room is on the lower ground a private street entrance."
ws := strings.Split(directions, " ")
ls := []string{}
l := 0 // current line length
i := -1
for _, v := range ws {
lv := strings.Count(v, "")-1
if l == 0 || l+lv+1 > 48 {
i++
ls = append(ls, v)
l = lv
continue
}
ls[i] += " " + v
l += lv+1
}
fmt.Printf("%#v", ls)
}
Go Playground: https://play.golang.org/p/HhdX8RudiXn

Go channel takes each letter as string instead of the whole string

I'm creating a simple channel that takes string values. But apparently I'm pushing each letter in the string instead of the whole string in each loop.
I'm probably missing something very fundamental. What am I doing wrong ?
https://play.golang.org/p/-6E-f7ALbD
Code:
func doStuff(s string, ch chan string) {
ch <- s
}
func main() {
c := make(chan string)
loops := [5]int{1, 2, 3, 4, 5}
for i := 0; i < len(loops); i++ {
go doStuff("helloooo", c)
}
results := <-c
fmt.Println("channel size = ", len(results))
// print the items in channel
for _, r := range results {
fmt.Println(string(r))
}
}
Your code sends strings on the channel properly:
func doStuff(s string, ch chan string){
ch <- s
}
The problem is at the receiver side:
results := <- c
fmt.Println("channel size = ", len(results))
// print the items in channel
for _,r := range results {
fmt.Println(string(r))
}
results will be a single value received from the channel (the first value sent on it). And you print the length of this string.
Then you loop over this string (results) using a for range which loops over its runes, and you print those.
What you want is loop over the values of the channel:
// print the items in channel
for s := range c {
fmt.Println(s)
}
This when run will result in a runtime panic:
fatal error: all goroutines are asleep - deadlock!
Because you never close the channel, and a for range on a channel runs until the channel is closed. So you have to close the channel sometime.
For example let's wait 1 second, then close it:
go func() {
time.Sleep(time.Second)
close(c)
}()
This way your app will run and quit after 1 second. Try it on the Go Playground.
Another, nicer solution is to use sync.WaitGroup: this waits until all goroutines are done doing their work (sending a value on the channel), then it closes the channel (so there is no unnecessary wait / delay).
var wg = sync.WaitGroup{}
func doStuff(s string, ch chan string) {
ch <- s
wg.Done()
}
// And in main():
for i := 0; i < len(loops); i++ {
wg.Add(1)
go doStuff("helloooo", c)
}
go func() {
wg.Wait()
close(c)
}()
Try this one on the Go Playground.
Notes:
To repeat something 5 times, you don't need that ugly loops array. Simply do:
for i := 0; i < 5; i++ {
// Do something
}
The reason you are getting back the letters instead of string is that you are assigning the channel result to a variable and iterating over the result of the channel assigned to this variable which in your case is a string, and in Go you can iterate over a string with a for range loop to get the runes.
You can simply print the channel without to iterate over the channel result.
package main
import (
"fmt"
)
func doStuff(s string, ch chan string){
ch <- s
}
func main() {
c := make(chan string)
loops := [5]int{1,2,3,4,5}
for i := 0; i < len(loops) ; i++ {
go doStuff("helloooo", c)
}
results := <- c
fmt.Println("channel size = ", len(results))
fmt.Println(results) // will print helloooo
}

How to sync goroutines in pipeline

I would need help to understand why the following code does not work. I am building a pipeline and trying to have a step that synchronize values from two source channels. My source/producer code looks something like below (in my real code i read the text from a file). The sources are sorted but are values are not guaranteed to be in both sources.
func Source() <-chan int{
out := make(chan int, 5)
go func() {
defer reader.Close()
out <- 1
out <- 2
out <- 3
out <- 4
out <- 5
out <- 7
close(out)
}()
return out
}
and the synchronization code looks like this:
func Sync(a, b <-chan int) <-chan int {
out := make(chan int)
go func() {
av, ak:= <-a
bv, bk:= <-b
for ak || bk {
if !ak || av < bv {
out <- bv
bv, bk = <-b
continue
}
if !bk|| bv > av {
out <- av
av, ak = <-a
continue
}
out <- av
av, ak = <-a
bv, bk = <-b
}
close(out)
}()
return out
}
and my program looks something like this:
func main() {
os := Source()
ns := Source()
for val := range Sync(ns, os) {
fmt.Printf("[SYNCED] %v \n", val)
}
}
The expected behaviour is that my both sources buffer values into the channel and my sync first reads value from the first source. Then from the second. Compare them and if they are equal continues to the next in the both channels. If the differ we will send out the value that is behind and replace it with a new one and make the same comparison again.
What happends is that it looks like the sync code is run several times for the values and I will get things like [SYNCED] 1 several times. Why?
Please help me get this fixed!
Regarding http://play.golang.org/p/uhd3EWrwEo and http://play.golang.org/p/Dqq7-cPaFq -
Actually, the code for ints will fail with the similar test case as well:
os := Source([]int{1, 2, 3})
ns := Source([]int{1, 3, 4})
puts the ints version to infinite loop.
This happens because when !aok || avalue > bvalue is checked, it does not take into account that if aok is true (some elements still are in a) and bok is false (no more elements in b), then avalue > "" is always true. So it tries to take another item from b (which is empty) and goes to infinite loop. Fixed code: http://play.golang.org/p/vYhuOZxRMl

How to get CPU usage

My Go program needs to know the current cpu usage percentage of all system and user processes.
How can I obtain that?
Check out this package http://github.com/c9s/goprocinfo, goprocinfo package does the parsing stuff for you.
stat, err := linuxproc.ReadStat("/proc/stat")
if err != nil {
t.Fatal("stat read fail")
}
for _, s := range stat.CPUStats {
// s.User
// s.Nice
// s.System
// s.Idle
// s.IOWait
}
I had a similar issue and never found a lightweight implementation. Here is a slimmed down version of my solution that answers your specific question. I sample the /proc/stat file just like tylerl recommends. You'll notice that I wait 3 seconds between samples to match top's output, but I have also had good results with 1 or 2 seconds. I run similar code in a loop within a go routine, then I access the cpu usage when I need it from other go routines.
You can also parse the output of top -n1 | grep -i cpu to get the cpu usage, but it only samples for half a second on my linux box and it was way off during heavy load. Regular top seemed to match very closely when I synchronized it and the following program:
package main
import (
"fmt"
"io/ioutil"
"strconv"
"strings"
"time"
)
func getCPUSample() (idle, total uint64) {
contents, err := ioutil.ReadFile("/proc/stat")
if err != nil {
return
}
lines := strings.Split(string(contents), "\n")
for _, line := range(lines) {
fields := strings.Fields(line)
if fields[0] == "cpu" {
numFields := len(fields)
for i := 1; i < numFields; i++ {
val, err := strconv.ParseUint(fields[i], 10, 64)
if err != nil {
fmt.Println("Error: ", i, fields[i], err)
}
total += val // tally up all the numbers to get total ticks
if i == 4 { // idle is the 5th field in the cpu line
idle = val
}
}
return
}
}
return
}
func main() {
idle0, total0 := getCPUSample()
time.Sleep(3 * time.Second)
idle1, total1 := getCPUSample()
idleTicks := float64(idle1 - idle0)
totalTicks := float64(total1 - total0)
cpuUsage := 100 * (totalTicks - idleTicks) / totalTicks
fmt.Printf("CPU usage is %f%% [busy: %f, total: %f]\n", cpuUsage, totalTicks-idleTicks, totalTicks)
}
It seems like I'm allowed to link to the full implementation that I wrote on bitbucket; if it's not, feel free to delete this. It only works on linux so far, though: systemstat.go
The mechanism for getting CPU usage is OS-dependent, since the numbers mean slightly different things to different OS kernels.
On Linux, you can query the kernel to get the latest stats by reading the pseudo-files in the /proc/ filesystem. These are generated on-the-fly when you read them to reflect the current state of the machine.
Specifically, the /proc/<pid>/stat file for each process contains the associated process accounting information. It's documented in proc(5). You're interested specifically in fields utime, stime, cutime and cstime (starting at the 14th field).
You can calculate the percentage easily enough: just read the numbers, wait some time interval, and read them again. Take the difference, divide by the amount of time you waited, and there's your average. This is precisely what the top program does (as well as all other programs that perform the same service). Bear in mind that you can have over 100% cpu usage if you have more than 1 CPU.
If you just want a system-wide summary, that's reported in /proc/stat -- calculate your average using the same technique, but you only have to read one file.
You can use the os.exec package to execute the ps command and get the result.
Here is a program issuing the ps aux command, parsing the result and printing the CPU usage of all processes on linux :
package main
import (
"bytes"
"log"
"os/exec"
"strconv"
"strings"
)
type Process struct {
pid int
cpu float64
}
func main() {
cmd := exec.Command("ps", "aux")
var out bytes.Buffer
cmd.Stdout = &out
err := cmd.Run()
if err != nil {
log.Fatal(err)
}
processes := make([]*Process, 0)
for {
line, err := out.ReadString('\n')
if err!=nil {
break;
}
tokens := strings.Split(line, " ")
ft := make([]string, 0)
for _, t := range(tokens) {
if t!="" && t!="\t" {
ft = append(ft, t)
}
}
log.Println(len(ft), ft)
pid, err := strconv.Atoi(ft[1])
if err!=nil {
continue
}
cpu, err := strconv.ParseFloat(ft[2], 64)
if err!=nil {
log.Fatal(err)
}
processes = append(processes, &Process{pid, cpu})
}
for _, p := range(processes) {
log.Println("Process ", p.pid, " takes ", p.cpu, " % of the CPU")
}
}
Here is an OS independent solution using Cgo to harness the clock() function provided by C standard library:
//#include <time.h>
import "C"
import "time"
var startTime = time.Now()
var startTicks = C.clock()
func CpuUsagePercent() float64 {
clockSeconds := float64(C.clock()-startTicks) / float64(C.CLOCKS_PER_SEC)
realSeconds := time.Since(startTime).Seconds()
return clockSeconds / realSeconds * 100
}
I recently had to take CPU usage measurements from a Raspberry Pi (Raspbian OS) and used github.com/c9s/goprocinfo combined with what is proposed here:
Accurate calculation of CPU usage given in percentage in Linux?
The idea comes from the htop source code and is to have two measurements (previous / current) in order to calculate the CPU usage:
func calcSingleCoreUsage(curr, prev linuxproc.CPUStat) float32 {
PrevIdle := prev.Idle + prev.IOWait
Idle := curr.Idle + curr.IOWait
PrevNonIdle := prev.User + prev.Nice + prev.System + prev.IRQ + prev.SoftIRQ + prev.Steal
NonIdle := curr.User + curr.Nice + curr.System + curr.IRQ + curr.SoftIRQ + curr.Steal
PrevTotal := PrevIdle + PrevNonIdle
Total := Idle + NonIdle
// fmt.Println(PrevIdle, Idle, PrevNonIdle, NonIdle, PrevTotal, Total)
// differentiate: actual value minus the previous one
totald := Total - PrevTotal
idled := Idle - PrevIdle
CPU_Percentage := (float32(totald) - float32(idled)) / float32(totald)
return CPU_Percentage
}
For more you can also check https://github.com/tgogos/rpi_cpu_memory

Resources