I am trying to make my application run as fast as possible. I purchased a semi-powerful container off of Google Cloud and I am just itching to see how many iterations per second I can get out of this program. However, I am new to Go and so far my implementation is showing to be very messy and not working well.
The way I have it set up now, it will start out at a high rate (around 11,000 iterations per second) but then quickly dwindle down to 2,000. My goal is for a far bigger number than even 11,000. Also, the infofunc(i) function can't seem to keep up with fast speeds and using a goroutine for that function causes overlap of the printing to the console. Also, it will on occasion reuse the WaitGroup before the Wait has returned.
I don't like to be the person to ask to be spoon-fed code, but I am at a loss as to how to implement this. There seems to be so many different methods when it comes to parallelism, multithreading, etc. and it is confusing to me.
import (
"fmt"
"math/big"
"os"
"os/exec"
"sync"
"time"
)
var found = 0
var pages_queried = 0
var start_time = time.Now()
var bignum = new(big.Int)
var foundAddresses = 0
var wg sync.WaitGroup
var set = make(map[string]bool)
var addresses = []string{"6ab42gyr", "lo08n4g6"}
func main() {
bignum.SetString("1000000000000000000000000000", 10)
pick := os.Args[1]
kpp := 128
switch pick {
case "btc":
i := new(big.Int)
i, ok := i.SetString(os.Args[2], 10)
if ok {
cmd := exec.Command("clear")
cmd.Stdout = os.Stdout
cmd.Run()
for i.Cmp(bignum) < 0 {
wg.Add(1)
go func(i *big.Int) {
defer wg.Done()
go printKeys(i.String(), kpp)
i.Add(i, big.NewInt(1))
pages_queried += 1
infofunc(i)
}(i)
wg.Wait()
}
}
}
}
func infofunc(i *big.Int) {
elapsed := time.Now().Sub(start_time)
duration, _ := time.ParseDuration(elapsed.String())
duration2 := int(duration.Seconds())
if duration2 != 0 {
fmt.Printf("\033[5;0H")
fmt.Printf("Started at %s. Found: %d. Elapsed: %s. Queried: %d pages. Current page: %s. Rate: %d/s", start_time.String(), found, elapsed.String(), pages_queried, i.String(), (pages_queried / duration2))
}
}
func printKeys(pageNumber string, keysPerPage int) {
keys := generateKeys(pageNumber, keysPerPage)
length := len(keys)
var addressesLen = len(addresses)
for i := 0; i < length; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
for ii := 0; ii < addressesLen; ii++ {
wg.Add(1)
go func(i int, ii int, keys []key) {
defer wg.Done()
for _, v := range addresses {
if set[keys[i].compressed] || set[keys[i].uncompressed] {
fmt.Print("Found an address: " + v + "!\n")
fmt.Printf("%v", keys[i])
fmt.Print("\n")
foundAddresses += 1
found += 1
}
}
}(i, ii, keys)
}
}(i)
foundAddresses = 0
}
}
I would not use a global sync.WaitGroup, it is hard to understand what is happening. Instead, just define it wherever you need.
You are calling wg.Wait() inside the loop block. That is basically blocking the loop every iteration waiting for goroutine to complete. What you really want is to spawn all the goroutines and only then wait for their completition.
if ok {
cmd := exec.Command("clear")
cmd.Stdout = os.Stdout
cmd.Run()
var wg sync.WaitGroup //I am about to spawn goroutines, I need to wait for them
for i.Cmp(bignum) < 0 {
wg.Add(1)
go func(i *big.Int) {
defer wg.Done()
go printKeys(i.String(), kpp)
i.Add(i, big.NewInt(1))
pages_queried += 1
infofunc(i)
}(i)
}
wg.Wait() //Now that all goroutines are working, let's wait
}
You cannot avoid the print overlap when you have multiple goroutines. If that's a problem you might think of using the Go's log stdlib, which will add timestamps for you. Then, you should be able to sort them in chronological order.
Anyway, split the code in more goroutines does not ensure a speed up. If the problem you are trying to solve is intrinsically sequential, then more goroutines will just add more contention and pressure on Go scheduler, leading to the opposite result. More details here. Thus, a goroutine for infofunc will not help. But it can be improved by using a logger library instead of plain fmt package.
func infofunc(i *big.Int) {
duration := time.Since(start_time).Seconds()
if duration != 0 {
log.Printf("\033[5;0H")
log.Printf("Started at %s. Found: %d. Elapsed: %s. Queried: %d pages. Current page: %s. Rate: %d/s", start_time.String(), found, elapsed.String(), pages_queried, i.String(), (pages_queried / duration2))
}
}
For printKeys, I would not create so many goroutines, they are not going to help if work they need to perform is CPU bound, which seems to be the case here.
func printKeys(pageNumber string, keysPerPage int) {
keys := generateKeys(pageNumber, keysPerPage)
length := len(keys)
var addressesLen = len(addresses)
var wg sync.WaitGroup //Local WaitGroup
for i := 0; i < length; i++ {
wg.Add(1)
go func(i int) { //This goroutine could be removed, in my opinion.
defer wg.Done()
for ii := 0; ii < addressesLen; ii++ {
for _, v := range addresses {
if set[keys[i].compressed] || set[keys[i].uncompressed] {
log.Printf("Found an address: %v\n", v)
log.Printf("%v", keys[i])
log.Printf("\n")
foundAddresses += 1
found += 1
}
}
}
}(i)
foundAddresses = 0
}
wg.Wait()
}
I would suggest to write a benchmark on these functions and then enable tracing. In this way you should get an idea where your code is spending most of the time.
I have a function named linearize...I'm trying to speed up its execution but surprised to find that it had become slower. Have I missed something or have messed up with fundamentals..
As per my understanding things should improve..
Thanks,
package main
import (
"fmt"
"math"
"sync"
"time"
)
var rgb []float64
func linearizeWithWg(v float64, idx int, wg *sync.WaitGroup) {
defer wg.Done()
if v <= 0.04045 {
rgb[idx] = v / 12.92
} else {
rgb[idx] = math.Pow((v+0.055)/1.055, 2.4)
}
}
func linearizeWithGoR(v float64) float64 {
res := make(chan float64)
go func(input float64) {
if input <= 0.04045 {
res <- input / 12.92
} else {
res <- math.Pow((input+0.055)/1.055, 2.4)
}
}(v)
return <-res
}
func linearizeNomal(v float64) float64 {
if v <= 0.04045 {
return v / 12.92
}
return math.Pow((v+0.055)/1.055, 2.4)
}
func main() {
start := time.Now()
const C = 1.0 / 255.0
//Normal Execution
for i := 0; i < 100000; i++ {
linearizeNomal(float64(i) * C * 0.5)
linearizeNomal(float64(i) * C * 1.5)
linearizeNomal(float64(i) * C * 4.5)
}
elaspsed := time.Since(start)
fmt.Println(elaspsed)
//With GoRoutines.. Slow
start = time.Now()
for i := 0; i < 100000; i++ {
linearizeWithGoR(float64(i) * C * 0.5)
linearizeWithGoR(float64(i) * C * 1.5)
linearizeWithGoR(float64(i) * C * 2.5)
}
elaspsed = time.Since(start)
fmt.Println(elaspsed)
//With WaitGroup. Slow
for i := 0; i < 100000; i++ {
rgb = make([]float64, 3)
var wg sync.WaitGroup
wg.Add(3)
linearizeWithWg(float64(i)*C*0.5, 0, &wg)
linearizeWithWg(float64(i)*C*1.5, 1, &wg)
linearizeWithWg(float64(i)*C*4.5, 2, &wg)
wg.Wait()
}
elaspsed = time.Since(start)
fmt.Println(elaspsed)
}
The overhead of all the concurrency related functions(channel creation, channel sending, goroutine creation) is way bigger than the two instructions that you execute in each of your goroutine.
In addition, your goroutine version is basically serial because you spawn a goroutine and immediately wait for the result of its channel. The waitgroup version is similar.
Try again with a small number of goroutines each executing a chunk of your loop. #Joker_vD also has a good point to make sure that GOMAXPROCS is bigger than one.
Your problem is that you don't actually do anything concurrently.
in the workgroup example, you need to call go linearizeWithWg(...)
In the goroutine example, you start a goroutine, but then wait for it to end in the function. To run it concurrently, you would need a buffered response channel, and having another goroutine getting the responses
I'm trying to convert a string returned from flag.Arg(n) to an int. What is the idiomatic way to do this in Go?
For example strconv.Atoi.
Code:
package main
import (
"fmt"
"strconv"
)
func main() {
s := "123"
// string to int
i, err := strconv.Atoi(s)
if err != nil {
// ... handle error
panic(err)
}
fmt.Println(s, i)
}
Converting Simple strings
The easiest way is to use the strconv.Atoi() function.
Note that there are many other ways. For example fmt.Sscan() and strconv.ParseInt() which give greater flexibility as you can specify the base and bitsize for example. Also as noted in the documentation of strconv.Atoi():
Atoi is equivalent to ParseInt(s, 10, 0), converted to type int.
Here's an example using the mentioned functions (try it on the Go Playground):
flag.Parse()
s := flag.Arg(0)
if i, err := strconv.Atoi(s); err == nil {
fmt.Printf("i=%d, type: %T\n", i, i)
}
if i, err := strconv.ParseInt(s, 10, 64); err == nil {
fmt.Printf("i=%d, type: %T\n", i, i)
}
var i int
if _, err := fmt.Sscan(s, &i); err == nil {
fmt.Printf("i=%d, type: %T\n", i, i)
}
Output (if called with argument "123"):
i=123, type: int
i=123, type: int64
i=123, type: int
Parsing Custom strings
There is also a handy fmt.Sscanf() which gives even greater flexibility as with the format string you can specify the number format (like width, base etc.) along with additional extra characters in the input string.
This is great for parsing custom strings holding a number. For example if your input is provided in a form of "id:00123" where you have a prefix "id:" and the number is fixed 5 digits, padded with zeros if shorter, this is very easily parsable like this:
s := "id:00123"
var i int
if _, err := fmt.Sscanf(s, "id:%5d", &i); err == nil {
fmt.Println(i) // Outputs 123
}
Here are three ways to parse strings into integers, from fastest runtime to slowest:
strconv.ParseInt(...) fastest
strconv.Atoi(...) still very fast
fmt.Sscanf(...) not terribly fast but most flexible
Here's a benchmark that shows usage and example timing for each function:
package main
import "fmt"
import "strconv"
import "testing"
var num = 123456
var numstr = "123456"
func BenchmarkStrconvParseInt(b *testing.B) {
num64 := int64(num)
for i := 0; i < b.N; i++ {
x, err := strconv.ParseInt(numstr, 10, 64)
if x != num64 || err != nil {
b.Error(err)
}
}
}
func BenchmarkAtoi(b *testing.B) {
for i := 0; i < b.N; i++ {
x, err := strconv.Atoi(numstr)
if x != num || err != nil {
b.Error(err)
}
}
}
func BenchmarkFmtSscan(b *testing.B) {
for i := 0; i < b.N; i++ {
var x int
n, err := fmt.Sscanf(numstr, "%d", &x)
if n != 1 || x != num || err != nil {
b.Error(err)
}
}
}
You can run it by saving as atoi_test.go and running go test -bench=. atoi_test.go.
goos: darwin
goarch: amd64
BenchmarkStrconvParseInt-8 100000000 17.1 ns/op
BenchmarkAtoi-8 100000000 19.4 ns/op
BenchmarkFmtSscan-8 2000000 693 ns/op
PASS
ok command-line-arguments 5.797s
Try this
import ("strconv")
value := "123"
number,err := strconv.ParseUint(value, 10, 32)
finalIntNum := int(number) //Convert uint64 To int
If you control the input data, you can use the mini version
package main
import (
"testing"
"strconv"
)
func Atoi (s string) int {
var (
n uint64
i int
v byte
)
for ; i < len(s); i++ {
d := s[i]
if '0' <= d && d <= '9' {
v = d - '0'
} else if 'a' <= d && d <= 'z' {
v = d - 'a' + 10
} else if 'A' <= d && d <= 'Z' {
v = d - 'A' + 10
} else {
n = 0; break
}
n *= uint64(10)
n += uint64(v)
}
return int(n)
}
func BenchmarkAtoi(b *testing.B) {
for i := 0; i < b.N; i++ {
in := Atoi("9999")
_ = in
}
}
func BenchmarkStrconvAtoi(b *testing.B) {
for i := 0; i < b.N; i++ {
in, _ := strconv.Atoi("9999")
_ = in
}
}
the fastest option (write your check if necessary). Result :
Path>go test -bench=. atoi_test.go
goos: windows
goarch: amd64
BenchmarkAtoi-2 100000000 14.6 ns/op
BenchmarkStrconvAtoi-2 30000000 51.2 ns/op
PASS
ok path 3.293s
In Go, a string is a primitive type, which means it is read-only, and every manipulation of it will create a new string.
So if I want to concatenate strings many times without knowing the length of the resulting string, what's the best way to do it?
The naive way would be:
var s string
for i := 0; i < 1000; i++ {
s += getShortStringFromSomewhere()
}
return s
but that does not seem very efficient.
New Way:
From Go 1.10 there is a strings.Builder type, please take a look at this answer for more detail.
Old Way:
Use the bytes package. It has a Buffer type which implements io.Writer.
package main
import (
"bytes"
"fmt"
)
func main() {
var buffer bytes.Buffer
for i := 0; i < 1000; i++ {
buffer.WriteString("a")
}
fmt.Println(buffer.String())
}
This does it in O(n) time.
In Go 1.10+ there is strings.Builder, here.
A Builder is used to efficiently build a string using Write methods. It minimizes memory copying. The zero value is ready to use.
Example
It's almost the same with bytes.Buffer.
package main
import (
"strings"
"fmt"
)
func main() {
// ZERO-VALUE:
//
// It's ready to use from the get-go.
// You don't need to initialize it.
var sb strings.Builder
for i := 0; i < 1000; i++ {
sb.WriteString("a")
}
fmt.Println(sb.String())
}
Click to see this on the playground.
Supported Interfaces
StringBuilder's methods are being implemented with the existing interfaces in mind. So that you can switch to the new Builder type easily in your code.
Grow(int) -> bytes.Buffer#Grow
Len() int -> bytes.Buffer#Len
Reset() -> bytes.Buffer#Reset
String() string -> fmt.Stringer
Write([]byte) (int, error) -> io.Writer
WriteByte(byte) error -> io.ByteWriter
WriteRune(rune) (int, error) -> bufio.Writer#WriteRune - bytes.Buffer#WriteRune
WriteString(string) (int, error) -> io.stringWriter
Differences from bytes.Buffer
It can only grow or reset.
It has a copyCheck mechanism built-in that prevents accidentially copying it:
func (b *Builder) copyCheck() { ... }
In bytes.Buffer, one can access the underlying bytes like this: (*Buffer).Bytes().
strings.Builder prevents this problem.
Sometimes, this is not a problem though and desired instead.
For example: For the peeking behavior when the bytes are passed to an io.Reader etc.
bytes.Buffer.Reset() rewinds and reuses the underlying buffer whereas the strings.Builder.Reset() does not, it detaches the buffer.
Note
Do not copy a StringBuilder value as it caches the underlying data.
If you want to share a StringBuilder value, use a pointer to it.
Check out its source code for more details, here.
If you know the total length of the string that you're going to preallocate then the most efficient way to concatenate strings may be using the builtin function copy. If you don't know the total length before hand, do not use copy, and read the other answers instead.
In my tests, that approach is ~3x faster than using bytes.Buffer and much much faster (~12,000x) than using the operator +. Also, it uses less memory.
I've created a test case to prove this and here are the results:
BenchmarkConcat 1000000 64497 ns/op 502018 B/op 0 allocs/op
BenchmarkBuffer 100000000 15.5 ns/op 2 B/op 0 allocs/op
BenchmarkCopy 500000000 5.39 ns/op 0 B/op 0 allocs/op
Below is code for testing:
package main
import (
"bytes"
"strings"
"testing"
)
func BenchmarkConcat(b *testing.B) {
var str string
for n := 0; n < b.N; n++ {
str += "x"
}
b.StopTimer()
if s := strings.Repeat("x", b.N); str != s {
b.Errorf("unexpected result; got=%s, want=%s", str, s)
}
}
func BenchmarkBuffer(b *testing.B) {
var buffer bytes.Buffer
for n := 0; n < b.N; n++ {
buffer.WriteString("x")
}
b.StopTimer()
if s := strings.Repeat("x", b.N); buffer.String() != s {
b.Errorf("unexpected result; got=%s, want=%s", buffer.String(), s)
}
}
func BenchmarkCopy(b *testing.B) {
bs := make([]byte, b.N)
bl := 0
b.ResetTimer()
for n := 0; n < b.N; n++ {
bl += copy(bs[bl:], "x")
}
b.StopTimer()
if s := strings.Repeat("x", b.N); string(bs) != s {
b.Errorf("unexpected result; got=%s, want=%s", string(bs), s)
}
}
// Go 1.10
func BenchmarkStringBuilder(b *testing.B) {
var strBuilder strings.Builder
b.ResetTimer()
for n := 0; n < b.N; n++ {
strBuilder.WriteString("x")
}
b.StopTimer()
if s := strings.Repeat("x", b.N); strBuilder.String() != s {
b.Errorf("unexpected result; got=%s, want=%s", strBuilder.String(), s)
}
}
If you have a string slice that you want to efficiently convert to a string then you can use this approach. Otherwise, take a look at the other answers.
There is a library function in the strings package called Join:
http://golang.org/pkg/strings/#Join
A look at the code of Join shows a similar approach to Append function Kinopiko wrote: https://golang.org/src/strings/strings.go#L420
Usage:
import (
"fmt";
"strings";
)
func main() {
s := []string{"this", "is", "a", "joined", "string\n"};
fmt.Printf(strings.Join(s, " "));
}
$ ./test.bin
this is a joined string
I just benchmarked the top answer posted above in my own code (a recursive tree walk) and the simple concat operator is actually faster than the BufferString.
func (r *record) String() string {
buffer := bytes.NewBufferString("");
fmt.Fprint(buffer,"(",r.name,"[")
for i := 0; i < len(r.subs); i++ {
fmt.Fprint(buffer,"\t",r.subs[i])
}
fmt.Fprint(buffer,"]",r.size,")\n")
return buffer.String()
}
This took 0.81 seconds, whereas the following code:
func (r *record) String() string {
s := "(\"" + r.name + "\" ["
for i := 0; i < len(r.subs); i++ {
s += r.subs[i].String()
}
s += "] " + strconv.FormatInt(r.size,10) + ")\n"
return s
}
only took 0.61 seconds. This is probably due to the overhead of creating the new BufferString.
Update: I also benchmarked the join function and it ran in 0.54 seconds.
func (r *record) String() string {
var parts []string
parts = append(parts, "(\"", r.name, "\" [" )
for i := 0; i < len(r.subs); i++ {
parts = append(parts, r.subs[i].String())
}
parts = append(parts, strconv.FormatInt(r.size,10), ")\n")
return strings.Join(parts,"")
}
package main
import (
"fmt"
)
func main() {
var str1 = "string1"
var str2 = "string2"
out := fmt.Sprintf("%s %s ",str1, str2)
fmt.Println(out)
}
This is the fastest solution that does not require
you to know or calculate the overall buffer size first:
var data []byte
for i := 0; i < 1000; i++ {
data = append(data, getShortStringFromSomewhere()...)
}
return string(data)
By my benchmark, it's 20% slower than the copy solution (8.1ns per
append rather than 6.72ns) but still 55% faster than using bytes.Buffer.
You could create a big slice of bytes and copy the bytes of the short strings into it using string slices. There is a function given in "Effective Go":
func Append(slice, data[]byte) []byte {
l := len(slice);
if l + len(data) > cap(slice) { // reallocate
// Allocate double what's needed, for future growth.
newSlice := make([]byte, (l+len(data))*2);
// Copy data (could use bytes.Copy()).
for i, c := range slice {
newSlice[i] = c
}
slice = newSlice;
}
slice = slice[0:l+len(data)];
for i, c := range data {
slice[l+i] = c
}
return slice;
}
Then when the operations are finished, use string ( ) on the big slice of bytes to convert it into a string again.
Note added in 2018
From Go 1.10 there is a strings.Builder type, please take a look at this answer for more detail.
Pre-201x answer
The benchmark code of #cd1 and other answers are wrong. b.N is not supposed to be set in benchmark function. It's set by the go test tool dynamically to determine if the execution time of the test is stable.
A benchmark function should run the same test b.N times and the test inside the loop should be the same for each iteration. So I fix it by adding an inner loop. I also add benchmarks for some other solutions:
package main
import (
"bytes"
"strings"
"testing"
)
const (
sss = "xfoasneobfasieongasbg"
cnt = 10000
)
var (
bbb = []byte(sss)
expected = strings.Repeat(sss, cnt)
)
func BenchmarkCopyPreAllocate(b *testing.B) {
var result string
for n := 0; n < b.N; n++ {
bs := make([]byte, cnt*len(sss))
bl := 0
for i := 0; i < cnt; i++ {
bl += copy(bs[bl:], sss)
}
result = string(bs)
}
b.StopTimer()
if result != expected {
b.Errorf("unexpected result; got=%s, want=%s", string(result), expected)
}
}
func BenchmarkAppendPreAllocate(b *testing.B) {
var result string
for n := 0; n < b.N; n++ {
data := make([]byte, 0, cnt*len(sss))
for i := 0; i < cnt; i++ {
data = append(data, sss...)
}
result = string(data)
}
b.StopTimer()
if result != expected {
b.Errorf("unexpected result; got=%s, want=%s", string(result), expected)
}
}
func BenchmarkBufferPreAllocate(b *testing.B) {
var result string
for n := 0; n < b.N; n++ {
buf := bytes.NewBuffer(make([]byte, 0, cnt*len(sss)))
for i := 0; i < cnt; i++ {
buf.WriteString(sss)
}
result = buf.String()
}
b.StopTimer()
if result != expected {
b.Errorf("unexpected result; got=%s, want=%s", string(result), expected)
}
}
func BenchmarkCopy(b *testing.B) {
var result string
for n := 0; n < b.N; n++ {
data := make([]byte, 0, 64) // same size as bootstrap array of bytes.Buffer
for i := 0; i < cnt; i++ {
off := len(data)
if off+len(sss) > cap(data) {
temp := make([]byte, 2*cap(data)+len(sss))
copy(temp, data)
data = temp
}
data = data[0 : off+len(sss)]
copy(data[off:], sss)
}
result = string(data)
}
b.StopTimer()
if result != expected {
b.Errorf("unexpected result; got=%s, want=%s", string(result), expected)
}
}
func BenchmarkAppend(b *testing.B) {
var result string
for n := 0; n < b.N; n++ {
data := make([]byte, 0, 64)
for i := 0; i < cnt; i++ {
data = append(data, sss...)
}
result = string(data)
}
b.StopTimer()
if result != expected {
b.Errorf("unexpected result; got=%s, want=%s", string(result), expected)
}
}
func BenchmarkBufferWrite(b *testing.B) {
var result string
for n := 0; n < b.N; n++ {
var buf bytes.Buffer
for i := 0; i < cnt; i++ {
buf.Write(bbb)
}
result = buf.String()
}
b.StopTimer()
if result != expected {
b.Errorf("unexpected result; got=%s, want=%s", string(result), expected)
}
}
func BenchmarkBufferWriteString(b *testing.B) {
var result string
for n := 0; n < b.N; n++ {
var buf bytes.Buffer
for i := 0; i < cnt; i++ {
buf.WriteString(sss)
}
result = buf.String()
}
b.StopTimer()
if result != expected {
b.Errorf("unexpected result; got=%s, want=%s", string(result), expected)
}
}
func BenchmarkConcat(b *testing.B) {
var result string
for n := 0; n < b.N; n++ {
var str string
for i := 0; i < cnt; i++ {
str += sss
}
result = str
}
b.StopTimer()
if result != expected {
b.Errorf("unexpected result; got=%s, want=%s", string(result), expected)
}
}
Environment is OS X 10.11.6, 2.2 GHz Intel Core i7
Test results:
BenchmarkCopyPreAllocate-8 20000 84208 ns/op 425984 B/op 2 allocs/op
BenchmarkAppendPreAllocate-8 10000 102859 ns/op 425984 B/op 2 allocs/op
BenchmarkBufferPreAllocate-8 10000 166407 ns/op 426096 B/op 3 allocs/op
BenchmarkCopy-8 10000 160923 ns/op 933152 B/op 13 allocs/op
BenchmarkAppend-8 10000 175508 ns/op 1332096 B/op 24 allocs/op
BenchmarkBufferWrite-8 10000 239886 ns/op 933266 B/op 14 allocs/op
BenchmarkBufferWriteString-8 10000 236432 ns/op 933266 B/op 14 allocs/op
BenchmarkConcat-8 10 105603419 ns/op 1086685168 B/op 10000 allocs/op
Conclusion:
CopyPreAllocate is the fastest way; AppendPreAllocate is pretty close to No.1, but it's easier to write the code.
Concat has really bad performance both for speed and memory usage. Don't use it.
Buffer#Write and Buffer#WriteString are basically the same in speed, contrary to what #Dani-Br said in the comment. Considering string is indeed []byte in Go, it makes sense.
bytes.Buffer basically use the same solution as Copy with extra book keeping and other stuff.
Copy and Append use a bootstrap size of 64, the same as bytes.Buffer
Append use more memory and allocs, I think it's related to the grow algorithm it use. It's not growing memory as fast as bytes.Buffer
Suggestion:
For simple task such as what OP wants, I would use Append or AppendPreAllocate. It's fast enough and easy to use.
If need to read and write the buffer at the same time, use bytes.Buffer of course. That's what it's designed for.
My original suggestion was
s12 := fmt.Sprint(s1,s2)
But above answer using bytes.Buffer - WriteString() is the most efficient way.
My initial suggestion uses reflection and a type switch. See (p *pp) doPrint and (p *pp) printArg
There is no universal Stringer() interface for basic types, as I had naively thought.
At least though, Sprint() internally uses a bytes.Buffer. Thus
`s12 := fmt.Sprint(s1,s2,s3,s4,...,s1000)`
is acceptable in terms of memory allocations.
=> Sprint() concatenation can be used for quick debug output.
=> Otherwise use bytes.Buffer ... WriteString
Expanding on cd1's answer:
You might use append() instead of copy().
append() makes ever bigger advance provisions, costing a little more memory, but saving time.
I added two more benchmarks at the top of yours.
Run locally with
go test -bench=. -benchtime=100ms
On my thinkpad T400s it yields:
BenchmarkAppendEmpty 50000000 5.0 ns/op
BenchmarkAppendPrealloc 50000000 3.5 ns/op
BenchmarkCopy 20000000 10.2 ns/op
This is actual version of benchmark provided by #cd1 (Go 1.8, linux x86_64) with the fixes of bugs mentioned by #icza and #PickBoy.
Bytes.Buffer is only 7 times faster than direct string concatenation via + operator.
package performance_test
import (
"bytes"
"fmt"
"testing"
)
const (
concatSteps = 100
)
func BenchmarkConcat(b *testing.B) {
for n := 0; n < b.N; n++ {
var str string
for i := 0; i < concatSteps; i++ {
str += "x"
}
}
}
func BenchmarkBuffer(b *testing.B) {
for n := 0; n < b.N; n++ {
var buffer bytes.Buffer
for i := 0; i < concatSteps; i++ {
buffer.WriteString("x")
}
}
}
Timings:
BenchmarkConcat-4 300000 6869 ns/op
BenchmarkBuffer-4 1000000 1186 ns/op
goutils.JoinBetween
func JoinBetween(in []string, separator string, startIndex, endIndex int) string {
if in == nil {
return ""
}
noOfItems := endIndex - startIndex
if noOfItems <= 0 {
return EMPTY
}
var builder strings.Builder
for i := startIndex; i < endIndex; i++ {
if i > startIndex {
builder.WriteString(separator)
}
builder.WriteString(in[i])
}
return builder.String()
}
I do it using the following :-
package main
import (
"fmt"
"strings"
)
func main (){
concatenation:= strings.Join([]string{"a","b","c"},"") //where second parameter is a separator.
fmt.Println(concatenation) //abc
}
package main
import (
"fmt"
)
func main() {
var str1 = "string1"
var str2 = "string2"
result := make([]byte, 0)
result = append(result, []byte(str1)...)
result = append(result, []byte(str2)...)
result = append(result, []byte(str1)...)
result = append(result, []byte(str2)...)
fmt.Println(string(result))
}
Simple and easy to digest solution. Details in the comments.
Copy overwrites the elements of slice. We are slicing single-single element and overwriting it.
package main
import (
"fmt"
)
var N int = 100000
func main() {
slice1 := make([]rune, N, N)
//Efficient with fast performance, Need pre-allocated memory
//We can add a check if we reached the limit then increase capacity
//using append, but would be fined for data copying to new array. Also append happens after the length of current slice.
for i := 0; i < N; i++ {
copy(slice1[i:i+1], []rune{'N'})
}
fmt.Println(slice1)
//Simple but fast solution, Every time the slice capacity is reached we get a fine of effort that goes
//in copying data to new array
slice2 := []rune{}
for i := 0; i <= N; i++ {
slice2 = append(slice2, 'N')
}
fmt.Println(slice2)
}
benchmark result with memory allocation statistics. check benchmark code at github.
use strings.Builder to optimize performance.
go test -bench . -benchmem
goos: darwin
goarch: amd64
pkg: github.com/hechen0/goexp/exps
BenchmarkConcat-8 1000000 60213 ns/op 503992 B/op 1 allocs/op
BenchmarkBuffer-8 100000000 11.3 ns/op 2 B/op 0 allocs/op
BenchmarkCopy-8 300000000 4.76 ns/op 0 B/op 0 allocs/op
BenchmarkStringBuilder-8 1000000000 4.14 ns/op 6 B/op 0 allocs/op
PASS
ok github.com/hechen0/goexp/exps 70.071s
s := fmt.Sprintf("%s%s", []byte(s1), []byte(s2))
strings.Join() from the "strings" package
If you have a type mismatch(like if you are trying to join an int and a string), you do RANDOMTYPE (thing you want to change)
EX:
package main
import (
"fmt"
"strings"
)
var intEX = 0
var stringEX = "hello all you "
var stringEX2 = "people in here"
func main() {
s := []string{stringEX, stringEX2}
fmt.Println(strings.Join(s, ""))
}
Output :
hello all you people in here