I am trying to convert the following java statement of threads in go;
int num = 5;
Thread[] threads = new Thread[5];
for (int i = 0; i < num; i++)
{
threads[i] = new Thread(new NewClass(i));
threads[i].start();
}
for (int i = 0; i < numT; i++)
threads[i].join();
I wanted to know, how to convert this to go?
Thanks
Golang uses a concept called "goroutines" (inspired by "coroutines") instead of "threads" used by many other languages.
Your specific example looks like a common use for the sync.WaitGroup type:
wg := sync.WaitGroup{}
for i := 0; i < 5; i++ {
wg.Add(1) // Increment the number of routines to wait for.
go func(x int) { // Start an anonymous function as a goroutine.
defer wg.Done() // Mark this routine as complete when the function returns.
SomeFunction(x)
}(i) // Avoid capturing "i".
}
wg.Wait() // Wait for all routines to complete.
Note that SomeFunction(...) in the example can be work of any sort and will be executed concurrently with all other invocations; however, if it uses goroutines itself then it should be sure to use a similar mechanism as shown here to prevent from returning from the "SomeFunction" until it is actually done with its work.
Related
I am trying to make my application run as fast as possible. I purchased a semi-powerful container off of Google Cloud and I am just itching to see how many iterations per second I can get out of this program. However, I am new to Go and so far my implementation is showing to be very messy and not working well.
The way I have it set up now, it will start out at a high rate (around 11,000 iterations per second) but then quickly dwindle down to 2,000. My goal is for a far bigger number than even 11,000. Also, the infofunc(i) function can't seem to keep up with fast speeds and using a goroutine for that function causes overlap of the printing to the console. Also, it will on occasion reuse the WaitGroup before the Wait has returned.
I don't like to be the person to ask to be spoon-fed code, but I am at a loss as to how to implement this. There seems to be so many different methods when it comes to parallelism, multithreading, etc. and it is confusing to me.
import (
"fmt"
"math/big"
"os"
"os/exec"
"sync"
"time"
)
var found = 0
var pages_queried = 0
var start_time = time.Now()
var bignum = new(big.Int)
var foundAddresses = 0
var wg sync.WaitGroup
var set = make(map[string]bool)
var addresses = []string{"6ab42gyr", "lo08n4g6"}
func main() {
bignum.SetString("1000000000000000000000000000", 10)
pick := os.Args[1]
kpp := 128
switch pick {
case "btc":
i := new(big.Int)
i, ok := i.SetString(os.Args[2], 10)
if ok {
cmd := exec.Command("clear")
cmd.Stdout = os.Stdout
cmd.Run()
for i.Cmp(bignum) < 0 {
wg.Add(1)
go func(i *big.Int) {
defer wg.Done()
go printKeys(i.String(), kpp)
i.Add(i, big.NewInt(1))
pages_queried += 1
infofunc(i)
}(i)
wg.Wait()
}
}
}
}
func infofunc(i *big.Int) {
elapsed := time.Now().Sub(start_time)
duration, _ := time.ParseDuration(elapsed.String())
duration2 := int(duration.Seconds())
if duration2 != 0 {
fmt.Printf("\033[5;0H")
fmt.Printf("Started at %s. Found: %d. Elapsed: %s. Queried: %d pages. Current page: %s. Rate: %d/s", start_time.String(), found, elapsed.String(), pages_queried, i.String(), (pages_queried / duration2))
}
}
func printKeys(pageNumber string, keysPerPage int) {
keys := generateKeys(pageNumber, keysPerPage)
length := len(keys)
var addressesLen = len(addresses)
for i := 0; i < length; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
for ii := 0; ii < addressesLen; ii++ {
wg.Add(1)
go func(i int, ii int, keys []key) {
defer wg.Done()
for _, v := range addresses {
if set[keys[i].compressed] || set[keys[i].uncompressed] {
fmt.Print("Found an address: " + v + "!\n")
fmt.Printf("%v", keys[i])
fmt.Print("\n")
foundAddresses += 1
found += 1
}
}
}(i, ii, keys)
}
}(i)
foundAddresses = 0
}
}
I would not use a global sync.WaitGroup, it is hard to understand what is happening. Instead, just define it wherever you need.
You are calling wg.Wait() inside the loop block. That is basically blocking the loop every iteration waiting for goroutine to complete. What you really want is to spawn all the goroutines and only then wait for their completition.
if ok {
cmd := exec.Command("clear")
cmd.Stdout = os.Stdout
cmd.Run()
var wg sync.WaitGroup //I am about to spawn goroutines, I need to wait for them
for i.Cmp(bignum) < 0 {
wg.Add(1)
go func(i *big.Int) {
defer wg.Done()
go printKeys(i.String(), kpp)
i.Add(i, big.NewInt(1))
pages_queried += 1
infofunc(i)
}(i)
}
wg.Wait() //Now that all goroutines are working, let's wait
}
You cannot avoid the print overlap when you have multiple goroutines. If that's a problem you might think of using the Go's log stdlib, which will add timestamps for you. Then, you should be able to sort them in chronological order.
Anyway, split the code in more goroutines does not ensure a speed up. If the problem you are trying to solve is intrinsically sequential, then more goroutines will just add more contention and pressure on Go scheduler, leading to the opposite result. More details here. Thus, a goroutine for infofunc will not help. But it can be improved by using a logger library instead of plain fmt package.
func infofunc(i *big.Int) {
duration := time.Since(start_time).Seconds()
if duration != 0 {
log.Printf("\033[5;0H")
log.Printf("Started at %s. Found: %d. Elapsed: %s. Queried: %d pages. Current page: %s. Rate: %d/s", start_time.String(), found, elapsed.String(), pages_queried, i.String(), (pages_queried / duration2))
}
}
For printKeys, I would not create so many goroutines, they are not going to help if work they need to perform is CPU bound, which seems to be the case here.
func printKeys(pageNumber string, keysPerPage int) {
keys := generateKeys(pageNumber, keysPerPage)
length := len(keys)
var addressesLen = len(addresses)
var wg sync.WaitGroup //Local WaitGroup
for i := 0; i < length; i++ {
wg.Add(1)
go func(i int) { //This goroutine could be removed, in my opinion.
defer wg.Done()
for ii := 0; ii < addressesLen; ii++ {
for _, v := range addresses {
if set[keys[i].compressed] || set[keys[i].uncompressed] {
log.Printf("Found an address: %v\n", v)
log.Printf("%v", keys[i])
log.Printf("\n")
foundAddresses += 1
found += 1
}
}
}
}(i)
foundAddresses = 0
}
wg.Wait()
}
I would suggest to write a benchmark on these functions and then enable tracing. In this way you should get an idea where your code is spending most of the time.
I was trying to implement multithreading in golang. I am able to implement go routines but it is not working as expected. below is the sample program which i have prepared,
func test(s string, fo *os.File) {
var s1 [105]int
count :=0
for x :=1000; x<1101;x++ {
s1[count] = x;
count++
}
//fmt.Println(s1[0])
for i := range s1 {
runtime.Gosched()
sd := s + strconv.Itoa(i)
var fileMutex sync.Mutex
fileMutex.Lock()
fmt.Fprintf(fo,sd)
defer fileMutex.Unlock()
}
}
func main() {
fo,err :=os.Create("D:/Output.txt")
if err != nil {
panic(err)
}
for i := 0; i < 4; i++ {
go test("bye",fo)
}
}
OUTPUT - good0bye0bye0bye0bye0good1bye1bye1bye1bye1good2bye2bye2bye2bye2.... etc.
the above program will create a file and write "Hello" and "bye" in the file.
My problem is i am trying to create 5 thread and wanted to process different values values with different thread. if you will see the above example it is printing "bye" 4 times.
i wanted output like below using 5 thread,
good0bye0good1bye1good2bye2....etc....
any idea how can i achieve this?
First, you need to block in your main function until all other goroutines return. The mutexes in your program aren't blocking anything, and since they're re-initialized in each loop, they don't even block within their own goroutine. You can't defer an unlock if you're not returning from the function, you need to explicitly unlock in each iteration of the loop. You aren't using any of the values in your array (though you should use a slice instead), so we can drop that entirely. You also don't need runtime.GoSched in a well-behaved program, and it does nothing here.
An equivalent program that will run to completion would look like:
var wg sync.WaitGroup
var fileMutex sync.Mutex
func test(s string, fo *os.File) {
defer wg.Done()
for i := 0; i < 105; i++ {
fileMutex.Lock()
fmt.Fprintf(fo, "%s%d", s, i)
fileMutex.Unlock()
}
}
func main() {
fo, err := os.Create("D:/output.txt")
if err != nil {
log.Fatal(err)
}
for i := 0; i < 4; i++ {
wg.Add(1)
go test("bye", fo)
}
wg.Wait()
}
Finally though, there's no reason to try and write serial values to a single file from multiple goroutines, and it's less efficient to do so. If you want the values ordered over the entire file, you will need to use a single goroutine anyway.
Here is a simplified version of my situation:
void AppendToVector(std::vector<int>* my_vector) {
for (int i = 0; i < 100; i++) {
my_vector->push_back(i);
}
}
void CreateVectors(const int num_threads) {
std::vector<std::vector<int>* > my_vector_of_pointers(10);
ThreadPool pool(num_threads);
for (for int i = 0; i < 10; i++) {
my_vector_of_pointers[i] = new std::vector<int>();
pool.AddTask(AppendToVector,
&my_vector_of_pointers[i]);
}
}
My question is whether I need to put a mutex lock in AppendToVector when running this with multiple threads? My intuition tells me I do not have to because there is no possibility of two threads accessing the same data, but I am not fully confident.
Every thread is appending different std::vector (inside AppendToVector function) so there is no need for locking in your case. In case you change your code in the way more than one thread access same vector then you will need lock. Don't be confused because std::vectors you are passing to AppendToVector are them-selfs elements of main std::list, it matters only that here threads are manipulating with completely different (not shared) memory
Is it better for a thread to block than to wait? Is there a difference?
Scenario 1 is just having Thread 2 hog global variable k until it is done with it. Scenario 2 presents more of a real-world multithreaded scenario with more than 2 threads.
Scenario 1:
global_var k = 1;
Thread1()
{
//preliminary work
while (!done)
{
mutex_lock(handshake_k);
if (100 == k)
done = true;
mutex_unlock(handshake_k);
}
//continue executing
}
Thread2() {
//preliminary work
mutex_lock(handshake_k);
for (i=0; i <= 100; i++)
++k; ;
mutex_unlock(handshake_k);
}
Scenario 2:
global_var k = 1;
Thread1()
{
//preliminary work
while (!done)
{
mutex_lock(handshake_k);
if (k < 100)
{
wait_cv(handshake_monitor_k); //unlocks handshake_k
//mutex exclusively locked here
}
else
done = true;
mutex_unlock(handshake_k);
}
//continue executing
}
Thread2()
{
//preliminary work
for (i=0; i <= 100; i++)
{
mutex_lock(handshake_k);
++k;
mutex_unlock(handshake_k);
}
}
In this case, it doesn't much matter because it takes such a short time to count k to 100.
If, however, you were doing something that took some time, the 2nd would be more appropriate unless you knew, for sure, that k would have to reach 100 before anything happened.
In real life, you are not likely to know what the waiting threads will be doing while waiting. No need to hog all the CPU time in the 2nd thread in that case. Free things up now and then so the granularity of the CPU sharing is smaller. This is also useful in cases where Thread 1 is tied into some sort of GUI event handling.
How to implement barrier using posix semaphores?
void my_barrier_init(int a){
int i;
bar.number = a;
bar.counter = 0;
bar.arr = (sem_t*) malloc(sizeof(sem_t)*bar.number);
bar.cont = (sem_t*) malloc(sizeof(sem_t)*bar.number);
for(i = 0; i < bar.number; i++){
sem_init(&bar.arr[i], 0, 0);
sem_init(&bar.cont[i], 0, 0); }
}
void my_barrier_wait(){
int i;
bar.counter++;
if(bar.number == bar.counter){
for(i = 0; i < bar.number-1; i++){ sem_wait(&bar.arr[i]); }
for(i = 0; i < bar.number-1; i++){ sem_post(&bar.cont[i]); }
bar.counter = 0;
}else{
sem_post(&bar.arr[pthread_self()-2]);
sem_wait(&bar.cont[pthread_self()-2]);
}
}
When function my_barrier_wait is called, first (N-1) times it would set(+1) for semaphores in array 'arr' and go to sleep(calling sem_wait). N-th time it decrements semaphores in 'arr' and SHOULD (as I expect) wake up [0..bar.number-1] threads posting +1 for semaphores in 'cont' array. It doesn't work like barrier.
You need to look at this (PDF), The Little Book of Semaphores by Allen Downey. Specifically section 3.6.7. It's in Python, but gist of it should be clear enough.