I've got this small code snippet to test 2 ways of converting byte slice to string object, one function to allocate a new string object, another uses unsafe pointer arithmetic to construct string*, which doesn't allocate new memory:
package main
import (
"fmt"
"reflect"
"unsafe"
)
func byteToString(b []byte) string {
return string(b)
}
func byteToStringNoAlloc(b []byte) string {
if len(b) == 0 {
return ""
}
sh := reflect.StringHeader{uintptr(unsafe.Pointer(&b[0])), len(b)}
return *(*string)(unsafe.Pointer(&sh))
}
func main() {
b := []byte("hello")
fmt.Printf("1st element of slice: %v\n", &b[0])
str := byteToString(b)
sh := (*reflect.StringHeader)(unsafe.Pointer(&str))
fmt.Printf("New alloc: %v\n", sh)
toStr := byteToStringNoAlloc(b)
shNoAlloc := (*reflect.StringHeader)(unsafe.Pointer(&toStr))
fmt.Printf("No alloc: %v\n", shNoAlloc) // why different from &b[0]
}
I run this program under go 1.13:
1st element of slice: 0xc000076068
New alloc: &{824634204304 5}
No alloc: &{824634204264 5}
I exptect that the "1st element of slice" should print out the same address like "No alloc", but acturally they're very different. Where did I get wrong?
First of all, type conversions are calling a internal functions, for this case it's slicebytetostring.
https://golang.org/src/runtime/string.go?h=slicebytetostring#L75
It does copy of slice's content into new allocated memory.
In the second case you're creating a new header of the slice and cast it into string header the new unofficial holder of slice's content.
The problem of this is that garbage collector doesn't handle such kind of cases and resulting string header will be marked as a single structure which has no relations with the actual slice which holds the actual content, so, your resulting string would be valid only while the actual content holders are alive (don't count this string header itself).
So once garbage collector sweep the actual content, your string will still point to the same address but already freed memory, and you'll get the panic error or undefined behavior if you touch it.
By the way, there's no need to use reflect package and its headers because direct cast already creates new header as a result:
*(*string)(unsafe.Pointer(&byte_slice))
This question already has answers here:
Why does Go handle closures differently in goroutines?
(2 answers)
Closed 10 months ago.
I'm learning Go and trying to understand its concurrency features.
I have the following program.
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
for i := 0; i < 5; i++ {
wg.Add(1)
x := i
go func() {
defer wg.Done()
fmt.Println(x)
}()
}
wg.Wait()
fmt.Println("Done")
}
When executed I got:
4
0
1
3
2
It's just what I want. However, if I make slight modification to it:
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
for i := 0; i < 5; i++ {
wg.Add(1)
go func() {
defer wg.Done()
fmt.Println(i)
}()
}
wg.Wait()
fmt.Println("Done")
}
What I got will be:
5
5
5
5
5
I don't quite understand the difference. Can anyone help to explain what happened here and how Go runtime execute this code?
You have new variable on each run of x := i,
This code shows difference well, by printing the address of x inside goroutine:
The Go Playground:
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
for i := 0; i < 5; i++ {
wg.Add(1)
x := i
go func() {
defer wg.Done()
fmt.Println(&x)
}()
}
wg.Wait()
fmt.Println("Done")
}
output:
0xc0420301e0
0xc042030200
0xc0420301e8
0xc0420301f0
0xc0420301f8
Done
And build your second example with go build -race and run it:
You will see: WARNING: DATA RACE
And this will be fine The Go Playground:
//go build -race
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
for i := 0; i < 5; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
fmt.Println(i)
}(i)
}
wg.Wait()
fmt.Println("Done")
}
output:
0
4
1
2
3
Done
The general rule is, don't share data between goroutines. In the first example, you essentially give each goroutine their own copy of x, and they print it out in whatever order they get to the print statement. In the second example, they all reference the same loop variable, and it is incremented to 5 by the time any of them print it. I don't believe the output there is guaranteed, it just happens that the loop creating goroutines finished faster than the goroutines themselves got to the printing part.
It's a bit hard to explain in plain english, but I'll try my best.
You see, every time you spawn a new goroutine, there is an initialization time, no matter how minuscule it might be, it's always there. So, in your second case, the entire loop has finished incrementing the variable 5 times before any of the goroutines even started. And when the goroutines finish initialization, all they see is the final variable value which is 5.
In your first case though, the x variable keeps a copy of the i variable so that when the goroutines start, x get's passed to them. Remember, it is i that is being incremented here, not x. x is fixed. So, when the goroutines start, they get a fixed value.
This question already has an answer here:
Convert byte slice to io.Reader
(1 answer)
Closed 2 years ago.
I have just started with go and was wondering, if it is possible to convert an []byte slice to an io.Reader. The Otherway around is possible as shown in ioutil.ReadAll.
If not is it possible to use code.google.com/p/go.net/html.Tokenizer somehow with a byte slice?
Yes: bytes.NewBuffer
io.Reader Example:
http://play.golang.org/p/P0VbE8UFpC
package main
import (
"bytes"
"encoding/base64"
"io"
"os"
)
func main() {
// A Buffer can turn a string or a []byte into an io.Reader.
buf := bytes.NewBuffer([]byte("R29waGVycyBydWxlIQ=="))
dec := base64.NewDecoder(base64.StdEncoding, buf)
io.Copy(os.Stdout, dec)
}
You can use the NewReader in the bytes package:
in := bytes.NewReader(b []byte)
https://golang.org/pkg/bytes/#NewReader
When taking a substring of a string in Go, no new memory is allocated. Instead, the underlying representation of the substring contains a Data pointer that is an offset of the original string's Data pointer.
This means that if I have a large string and wish to keep track of a small substring, the garbage collector will be unable to free any of the large string until I release all references to the shorter substring.
Slices have a similar problem, but you can get around it by making a copy of the subslice using copy(). I am unaware of any similar copy operation for strings. What is the idiomatic and fastest way to make a "copy" of a substring?
For example,
package main
import (
"fmt"
"unsafe"
)
type String struct {
str *byte
len int
}
func main() {
str := "abc"
substr := string([]byte(str[1:]))
fmt.Println(str, substr)
fmt.Println(*(*String)(unsafe.Pointer(&str)), *(*String)(unsafe.Pointer(&substr)))
}
Output:
abc bc
{0x4c0640 3} {0xc21000c940 2}
I know this is an old question, but there are a couple ways you can do this without creating two copies of the data you want.
First is to create the []byte of the substring, then simply coerce it to a string using unsafe.Pointer. This works because the header for a []byte is the same as that for a string, except that the []byte has an extra Cap field at the end, so it just gets truncated.
package main
import (
"fmt"
"unsafe"
)
func main() {
str := "foobar"
byt := []byte(str[3:])
sub := *(*string)(unsafe.Pointer(&byt))
fmt.Println(str, sub)
}
The second way is to use reflect.StringHeader and reflect.SliceHeader to do a more explicit header transfer.
package main
import (
"fmt"
"unsafe"
"reflect"
)
func main() {
str := "foobar"
byt := []byte(str[3:])
bytPtr := (*reflect.SliceHeader)(unsafe.Pointer(&byt)).Data
strHdr := reflect.StringHeader{Data: bytPtr, Len: len(byt)}
sub := *(*string)(unsafe.Pointer(&strHdr))
fmt.Println(str, sub)
}
Is there a simple way to format a string in Go without printing the string?
I can do:
bar := "bar"
fmt.Printf("foo: %s", bar)
But I want the formatted string returned rather than printed so I can manipulate it further.
I could also do something like:
s := "foo: " + bar
But this becomes difficult to read when the format string is complex, and cumbersome when one or many of the parts aren't strings and have to be converted first, like
i := 25
s := "foo: " + strconv.Itoa(i)
Is there a simpler way to do this?
Sprintf is what you are looking for.
Example
fmt.Sprintf("foo: %s", bar)
You can also see it in use in the Errors example as part of "A Tour of Go."
return fmt.Sprintf("at %v, %s", e.When, e.What)
1. Simple strings
For "simple" strings (typically what fits into a line) the simplest solution is using fmt.Sprintf() and friends (fmt.Sprint(), fmt.Sprintln()). These are analogous to the functions without the starter S letter, but these Sxxx() variants return the result as a string instead of printing them to the standard output.
For example:
s := fmt.Sprintf("Hi, my name is %s and I'm %d years old.", "Bob", 23)
The variable s will be initialized with the value:
Hi, my name is Bob and I'm 23 years old.
Tip: If you just want to concatenate values of different types, you may not automatically need to use Sprintf() (which requires a format string) as Sprint() does exactly this. See this example:
i := 23
s := fmt.Sprint("[age:", i, "]") // s will be "[age:23]"
For concatenating only strings, you may also use strings.Join() where you can specify a custom separator string (to be placed between the strings to join).
Try these on the Go Playground.
2. Complex strings (documents)
If the string you're trying to create is more complex (e.g. a multi-line email message), fmt.Sprintf() becomes less readable and less efficient (especially if you have to do this many times).
For this the standard library provides the packages text/template and html/template. These packages implement data-driven templates for generating textual output. html/template is for generating HTML output safe against code injection. It provides the same interface as package text/template and should be used instead of text/template whenever the output is HTML.
Using the template packages basically requires you to provide a static template in the form of a string value (which may be originating from a file in which case you only provide the file name) which may contain static text, and actions which are processed and executed when the engine processes the template and generates the output.
You may provide parameters which are included/substituted in the static template and which may control the output generation process. Typical form of such parameters are structs and map values which may be nested.
Example:
For example let's say you want to generate email messages that look like this:
Hi [name]!
Your account is ready, your user name is: [user-name]
You have the following roles assigned:
[role#1], [role#2], ... [role#n]
To generate email message bodies like this, you could use the following static template:
const emailTmpl = `Hi {{.Name}}!
Your account is ready, your user name is: {{.UserName}}
You have the following roles assigned:
{{range $i, $r := .Roles}}{{if $i}}, {{end}}{{.}}{{end}}
`
And provide data like this for executing it:
data := map[string]interface{}{
"Name": "Bob",
"UserName": "bob92",
"Roles": []string{"dbteam", "uiteam", "tester"},
}
Normally output of templates are written to an io.Writer, so if you want the result as a string, create and write to a bytes.Buffer (which implements io.Writer). Executing the template and getting the result as string:
t := template.Must(template.New("email").Parse(emailTmpl))
buf := &bytes.Buffer{}
if err := t.Execute(buf, data); err != nil {
panic(err)
}
s := buf.String()
This will result in the expected output:
Hi Bob!
Your account is ready, your user name is: bob92
You have the following roles assigned:
dbteam, uiteam, tester
Try it on the Go Playground.
Also note that since Go 1.10, a newer, faster, more specialized alternative is available to bytes.Buffer which is: strings.Builder. Usage is very similar:
builder := &strings.Builder{}
if err := t.Execute(builder, data); err != nil {
panic(err)
}
s := builder.String()
Try this one on the Go Playground.
Note: you may also display the result of a template execution if you provide os.Stdout as the target (which also implements io.Writer):
t := template.Must(template.New("email").Parse(emailTmpl))
if err := t.Execute(os.Stdout, data); err != nil {
panic(err)
}
This will write the result directly to os.Stdout. Try this on the Go Playground.
try using Sprintf(); it will not print the output but save it for future purpose.
check this out.
package main
import "fmt"
func main() {
address := "NYC"
fmt.Sprintf("I live in %v", address)
}
when you run this code, it will not output anything. But once you assigned the Sprintf() to a separate variable, it can be used for future purposes.
package main
import "fmt"
func main() {
address := "NYC"
fmt.Sprintf("I live in %v", address)
var city = fmt.Sprintf("lives in %v", address)
fmt.Println("Michael",city)
}
I've created go project for string formatting from template (it allow to format strings in C# or Python style) and by performance tests it is fater than fmt.Sprintf, you could find it here https://github.com/Wissance/stringFormatter
it works in following manner:
func TestStrFormat(t *testing.T) {
strFormatResult, err := Format("Hello i am {0}, my age is {1} and i am waiting for {2}, because i am {0}!",
"Michael Ushakov (Evillord666)", "34", "\"Great Success\"")
assert.Nil(t, err)
assert.Equal(t, "Hello i am Michael Ushakov (Evillord666), my age is 34 and i am waiting for \"Great Success\", because i am Michael Ushakov (Evillord666)!", strFormatResult)
strFormatResult, err = Format("We are wondering if these values would be replaced : {5}, {4}, {0}", "one", "two", "three")
assert.Nil(t, err)
assert.Equal(t, "We are wondering if these values would be replaced : {5}, {4}, one", strFormatResult)
strFormatResult, err = Format("No args ... : {0}, {1}, {2}")
assert.Nil(t, err)
assert.Equal(t, "No args ... : {0}, {1}, {2}", strFormatResult)
}
func TestStrFormatComplex(t *testing.T) {
strFormatResult, err := FormatComplex("Hello {user} what are you doing here {app} ?", map[string]string{"user":"vpupkin", "app":"mn_console"})
assert.Nil(t, err)
assert.Equal(t, "Hello vpupkin what are you doing here mn_console ?", strFormatResult)
}
In your case, you need to use Sprintf() for format string.
func Sprintf(format string, a ...interface{}) string
Sprintf formats according to a format specifier and returns the resulting string.
s := fmt.Sprintf("Good Morning, This is %s and I'm living here from last %d years ", "John", 20)
Your output will be :
Good Morning, This is John and I'm living here from last 20 years.
fmt.SprintF function returns a string and you can format the string the very same way you would have with fmt.PrintF
I came to this page specifically looking for a way to format an error string. So if someone needs help with the same, you want to use the fmt.Errorf() function.
The method signature is func Errorf(format string, a ...interface{}) error.
It returns the formatted string as a value that satisfies the error interface.
You can look up more details in the documentation - https://golang.org/pkg/fmt/#Errorf.
Instead of using template.New, you can just use the new builtin with
template.Template:
package main
import (
"strings"
"text/template"
)
func format(s string, v interface{}) string {
t, b := new(template.Template), new(strings.Builder)
template.Must(t.Parse(s)).Execute(b, v)
return b.String()
}
func main() {
bar := "bar"
println(format("foo: {{.}}", bar))
i := 25
println(format("foo: {{.}}", i))
}