strings.Builder memory usage - string

Question

Using this file (data file):
package main
import (
"io/ioutil"
"time"
)
func main() {
ioutil.ReadFile("100mb.file")
time.Sleep(time.Duration(time.Minute))
}
Showed memory usage for me of 107 MB. With this similar file:
package main
import (
"bytes"
"os"
"time"
)
func read(path_s string) (bytes.Buffer, error) {
buf_o := bytes.Buffer{}
open_o, e := os.Open(path_s)
if e != nil {
return buf_o, e
}
buf_o.ReadFrom(open_o)
open_o.Close()
return buf_o, nil
}
func main() {
read("100mb.file")
time.Sleep(time.Duration(time.Minute))
}
Memory usage went to 273 MB. Finally this similar file:
package main
import (
"io"
"os"
"strings"
"time"
)
func read(path_s string) (strings.Builder, error) {
str_o := strings.Builder{}
open_o, e := os.Open(path_s)
if e != nil {
return str_o, e
}
io.Copy(&str_o, open_o)
open_o.Close()
return str_o, nil
}
func main() {
read("100mb.file")
time.Sleep(time.Duration(time.Minute))
}
Memory usage went to 432 MB. I tried to be careful and close files where
possible. Why is the memory usage so high for the second example, and especially
the final example? Can I change something so that they are closer to the first
example?

Answer 1

ioutil.ReadFile("100mb.file") gets the size of the file, allocates a []byte that size and slurps the bytes up into that slice.
buf_o.ReadFrom(open_o) allocates an initial []byte of some size and reads into that slice. If there's more data in the reader than space in the slice, then the function allocates a larger slice, copies existing data to that slice and reads more. This repeats until EOF.
The function ioutil.ReadFile uses bytes Buffer.ReadFrom internally. Take a look at the ioutil.ReadFile implementation to see how to improve direct use of the bytes.Buffer. A synopsis of the logic is this:
var buf bytes.Buffer
// Open file
f, err := os.Open(path)
if err != nil {
return &buf, err
}
defer f.Close()
// Get size.
fi, err := f.Stat()
if err != nil {
return &buf, err
}
// Grow to size of file plus extra slop to ensure no realloc.
buf.Grow(int(fi.Size()) + bytes.MinRead)
_, err := buf.ReadFrom(f)
return &buf, err
The strings.Builder example reallocates the internal buffer several times as in the bytes.Buffer example. In addition, io.Copy allocates a buffer. You can improve the strings.Builder example by growing the builder to the size of the file before reading.
Here's the code for strings.Builder:
var buf strings.Builder
// Open file
f, err := os.Open(path)
if err != nil {
return &buf, err
}
defer f.Close()
// Get size.
fi, err := f.Stat()
if err != nil {
return &buf, err
}
buf.Grow(int(fi.Size()))
_, err = io.Copy(&buf, f)
return &buf, err
io.Copy or some other code using an extra buffer is required because strings.Builder does not have a ReadFrom method. The strings.Builder type does not have a ReadFrom method because that method can leak a reference to the backing array of the internal slice of bytes.

Answer 2

Using suggestion from Muffin Top, I took my second example and added this
directly before the call to ReadFrom:
stat_o, e := open_o.Stat()
if e != nil {
return buf_o, e
}
buf_o.Grow(bytes.MinRead + int(stat_o.Size()))
and the memory went down to 107 MB, basically the same as the first example.