I read this post about data races. And I understand, that access to interface variable need to be sync. But what about access to struct?
For example we have this code:
package main
import (
"fmt"
)
type s struct {
field0 uint8
field1 uint16
}
func main() {
s0 := s{
field0: 1,
field1: 2,
}
s1 := s{
field0: 3,
field1: 4,
}
var shared s
var loop0, loop1 func()
loop0 = func() {
shared = s0
go loop1()
}
loop1 = func() {
shared = s1
go loop0()
}
go loop0()
for {
fmt.Println(shared)
}
}
I build this code with -race flag and race detector no detect any errors.
But if shared is not a pointer, what really happened? We need set 2 fields (if on stack).
Questions:
is it data race?
if no, why?
UPD: start loop0
is it data race?
Yes. Your loop1 and loop2 won't race with each other, but they both will race with your for { fmt.Println() } loop, because fmt.Println reads at the same time the other functions are writing to the same value.
Related
I have a use case where I need to lock on arguments of a function.
The function itself can be accessed concurrently
Function signature is something like
func (m objectType) operate(key string) (bool) {
// get lock on "key" (return false if unable to get lock in X ms - eg: 100 ms)
// operate
// release lock on "key"
return true;
}
The data space which can be locked is in the range of millions (~10 million)
Concurrent access to operate() is in the range of thousands (1 - 5k)
Expected contention is low though possible in case of hotspots in key (hence the lock)
What is the right way to implement this ? Few options I explored using a concurrent hash map
sync.Map - this is suited for cases with append only entries and high read ratio compared to writes. Hence not applicable here
sharded hashmap where each shard is locked by RWMutex - https://github.com/orcaman/concurrent-map - While this would work, concurrency is limited by no of shards rather than actual contention between keys. Also doesn't enable the timeout scenarios when lot of contention happens for a subset of keys
Though timeout is a P1 requirement, the P0 requirement would be to increase throughput here by granular locking if possible.
Is there a good way to achieve this ?
I would do it by using a map of buffered channels:
to acquire a "mutex", try to fill a buffered channel with a value
work
when done, empty the buffered channel so that another goroutine can use it
Example:
package main
import (
"fmt"
"sync"
"time"
)
type MutexMap struct {
mut sync.RWMutex // handle concurrent access of chanMap
chanMap map[int](chan bool) // dynamic mutexes map
}
func NewMutextMap() *MutexMap {
var mut sync.RWMutex
return &MutexMap{
mut: mut,
chanMap: make(map[int](chan bool)),
}
}
// Acquire a lock, with timeout
func (mm *MutexMap) Lock(id int, timeout time.Duration) error {
// get global lock to read from map and get a channel
mm.mut.Lock()
if _, ok := mm.chanMap[id]; !ok {
mm.chanMap[id] = make(chan bool, 1)
}
ch := mm.chanMap[id]
mm.mut.Unlock()
// try to write to buffered channel, with timeout
select {
case ch <- true:
return nil
case <-time.After(timeout):
return fmt.Errorf("working on %v just timed out", id)
}
}
// release lock
func (mm *MutexMap) Release(id int) {
mm.mut.Lock()
ch := mm.chanMap[id]
mm.mut.Unlock()
<-ch
}
func work(id int, mm *MutexMap) {
// acquire lock with timeout
if err := mm.Lock(id, 100*time.Millisecond); err != nil {
fmt.Printf("ERROR: %s\n", err)
return
}
fmt.Printf("working on task %v\n", id)
// do some work...
time.Sleep(time.Second)
fmt.Printf("done working on %v\n", id)
// release lock
mm.Release(id)
}
func main() {
mm := NewMutextMap()
var wg sync.WaitGroup
for i := 0; i < 50; i++ {
wg.Add(1)
id := i % 10
go func(id int, mm *MutexMap, wg *sync.WaitGroup) {
work(id, mm)
defer wg.Done()
}(id, mm, &wg)
}
wg.Wait()
}
EDIT: different version, where we also handle the concurrent access to the chanMap itself
Is it possible to EXPORT_SYMBOL() a struct that contains a kmalloc array? If yes, what are the things that I need to keep in mind?
This is a psuedo code of what I want to do.
struct test {
int a;
...
uint64_t* data;
}
struct test foo;
EXPORT_SYMBOL(foo);
...
In module1_kthread1_func() I have:
int module1_kthread1_func(void *foo){
...
foo->data = kmalloc(SIZE, GFP_KERNEL);
...
foo->data[var] = 1243;
var++;
...
}
In module2_kthread2_func() I have:
...
extern struct test foo;
...
int module2_kthread2_func(void* foo){
...
for (i=0; i<SIZE; i++)
printk(KERN_INFO "Variable value of %d is %llu", i, foo->data[var]);
...
}
It's definitely possible, yes.
You need to be careful and make sure that code that uses it knows that some of the fields might not be available before they are allocated. That is, check if they are NULL first instead of directly accessing them.
You might want to declare the structure with an explicit initializer, just so that it is obvious what is going on:
struct test foo = {
.a = 123,
.data = NULL // Initialized by function X when Y
};
EXPORT_SYMBOL(foo);
If such fields are compulsory for the structure to be used, you might want to initialize them early on (see here).
Is there a simple program which demonstrates how queues work in Go.
I just need something like add number 1 to 10 in queue and pull those from the queue in parallel using another thread.
A queue that is safe for concurrent use is basically a language construct: channel.
A channel–by design–is safe for concurrent send and receive. This is detaild here: If I am using channels properly should I need to use mutexes? Values sent on it are received in the order they were sent.
You can read more about channels here: What are golang channels used for?
A very simple example:
c := make(chan int, 10) // buffer for 10 elements
// Producer: send elements in a new goroutine
go func() {
for i := 0; i < 10; i++ {
c <- i
}
close(c)
}()
// Consumer: receive all elements sent on it before it was closed:
for v := range c {
fmt.Println("Received:", v)
}
Output (try it on the Go Playground):
Received: 0
Received: 1
Received: 2
Received: 3
Received: 4
Received: 5
Received: 6
Received: 7
Received: 8
Received: 9
Note that the channel buffer (10 in this example) has nothing to do with the number of elements you want to send "through" it. The buffer tells how many elements the channel may "store", or in other words, how many elements you may send on it without blocking when there are nobody is receiving from it. When the channel's buffer is full, further sends will block until someone starts receiving values from it.
You could use channel(safe for concurrent use) and wait group to read from queue concurrently
package main
import (
"fmt"
"sync"
)
func main() {
queue := make(chan int)
wg := new(sync.WaitGroup)
wg.Add(1)
defer wg.Wait()
go func(wg *sync.WaitGroup) {
for {
r, ok := <-queue
if !ok {
wg.Done()
return
}
fmt.Println(r)
}
}(wg)
for i := 1; i <= 10; i++ {
queue <- i
}
close(queue)
}
Playground link: https://play.golang.org/p/A_Amqcf2gwU
Another option is to create and implement a queue interface, with a backing type of a channel for concurrency. For convenience, I've made a gist.
Here's how you can use it.
queue := GetIntConcurrentQueue()
defer queue.Close()
// queue.Enqueue(1)
// myInt, errQueueClosed := queue.DequeueBlocking()
// myInt, errIfNoInt := queue.DequeueNonBlocking()
Longer example here - https://play.golang.org/p/npb2Uj9hGn1
Full implementation below, and again here's the gist of it.
// Can be any backing type, even 'interface{}' if desired.
// See stackoverflow.com/q/11403050/3960399 for type conversion instructions.
type IntConcurrentQueue interface {
// Inserts the int into the queue
Enqueue(int)
// Will return error if there is nothing in the queue or if Close() was already called
DequeueNonBlocking() (int, error)
// Will block until there is a value in the queue to return.
// Will error if Close() was already called.
DequeueBlocking() (int, error)
// Close should be called with defer after initializing
Close()
}
func GetIntConcurrentQueue() IntConcurrentQueue {
return &intChannelQueue{c: make(chan int)}
}
type intChannelQueue struct {
c chan int
}
func (q *intChannelQueue) Enqueue(i int) {
q.c <- i
}
func (q *intChannelQueue) DequeueNonBlocking() (int, error) {
select {
case i, ok := <-q.c:
if ok {
return i, nil
} else {
return 0, fmt.Errorf("queue was closed")
}
default:
return 0, fmt.Errorf("queue has no value")
}
}
func (q *intChannelQueue) DequeueBlocking() (int, error) {
i, ok := <-q.c
if ok {
return i, nil
}
return 0, fmt.Errorf("queue was closed")
}
func (q *intChannelQueue) Close() {
close(q.c)
}
I was trying to implement multithreading in golang. I am able to implement go routines but it is not working as expected. below is the sample program which i have prepared,
func test(s string, fo *os.File) {
var s1 [105]int
count :=0
for x :=1000; x<1101;x++ {
s1[count] = x;
count++
}
//fmt.Println(s1[0])
for i := range s1 {
runtime.Gosched()
sd := s + strconv.Itoa(i)
var fileMutex sync.Mutex
fileMutex.Lock()
fmt.Fprintf(fo,sd)
defer fileMutex.Unlock()
}
}
func main() {
fo,err :=os.Create("D:/Output.txt")
if err != nil {
panic(err)
}
for i := 0; i < 4; i++ {
go test("bye",fo)
}
}
OUTPUT - good0bye0bye0bye0bye0good1bye1bye1bye1bye1good2bye2bye2bye2bye2.... etc.
the above program will create a file and write "Hello" and "bye" in the file.
My problem is i am trying to create 5 thread and wanted to process different values values with different thread. if you will see the above example it is printing "bye" 4 times.
i wanted output like below using 5 thread,
good0bye0good1bye1good2bye2....etc....
any idea how can i achieve this?
First, you need to block in your main function until all other goroutines return. The mutexes in your program aren't blocking anything, and since they're re-initialized in each loop, they don't even block within their own goroutine. You can't defer an unlock if you're not returning from the function, you need to explicitly unlock in each iteration of the loop. You aren't using any of the values in your array (though you should use a slice instead), so we can drop that entirely. You also don't need runtime.GoSched in a well-behaved program, and it does nothing here.
An equivalent program that will run to completion would look like:
var wg sync.WaitGroup
var fileMutex sync.Mutex
func test(s string, fo *os.File) {
defer wg.Done()
for i := 0; i < 105; i++ {
fileMutex.Lock()
fmt.Fprintf(fo, "%s%d", s, i)
fileMutex.Unlock()
}
}
func main() {
fo, err := os.Create("D:/output.txt")
if err != nil {
log.Fatal(err)
}
for i := 0; i < 4; i++ {
wg.Add(1)
go test("bye", fo)
}
wg.Wait()
}
Finally though, there's no reason to try and write serial values to a single file from multiple goroutines, and it's less efficient to do so. If you want the values ordered over the entire file, you will need to use a single goroutine anyway.
A class t_analyser contains a function which performs some operations on a t_data object and outputs the results.
I would like to add to t_analyser a filter capability: a small and fast function
bool filter(const t_data & d) {...}
which allows to skip the analysis (and the output) if some conditions are met for that particular data. The filter should be setted up easily from the main, so I was thinking to store a shared function pointer in t_analyser and use a lambda to initialize it.
Is this a good approach? My concerns are related to the fact that many analysers can call the same filter function simultaneously in different threads, could this be a problem? Can I simply copy the pointer in the t_analyser's copy constructor? Any hint would be much appreciated.
This could be a problem if your filter function had side effects. Its signature is simple and says that it just makes some decision reading data from t_data, so make sure that t_data is not modified in parallel thread and you'll be fine.
Consider the following program:
#include <iostream>
struct X
{
void foo1(){ std::cout << "foo1" << std::endl; }
void foo2(){ std::cout << "foo2" << std::endl; }
};
typedef void (X::*MemberFunctionPointer)();
struct ByRef
{
ByRef( MemberFunctionPointer& f )
: f_( f )
{
}
void operator()()
{
X x;
(x.*f_)();
}
MemberFunctionPointer& f_;
};
struct ByValue
{
ByValue( MemberFunctionPointer f )
: f_( f )
{
}
void operator()()
{
X x;
(x.*f_)();
}
MemberFunctionPointer f_;
};
int main()
{
MemberFunctionPointer p = &X::foo1;
ByRef byRef( p );
ByValue byValue( p );
byRef();
byValue();
p = &X::foo2;
byRef();
byValue();
return 0;
}
Output:
foo1
foo1
foo2
foo1
Press <RETURN> to close this window...
From this you will notice that in the one case the member function pointer is passed by value (and not shared), and in the other it is passed by reference (and shared). When using the syntax:
foo( void( X::*f)() )...
the pointer to member function is passed by value, and is copied (and cannot be modified) again.
You can declare the function pointer as static + thread specific:
static _declspec(thread) FUNC_TYPE filterFunc;
Each thread that modifies filterFunc works on a different copy of the pointer.