i am trying to create a simple program to read lines from a text file and print them out to the console in golang. I spent lots of time going over my code and I simply can't understand why only the last line is being printed out to the screen. can anyone tell me where I am going wrong here? Everything here should compile and run.
package main
import (
"bufio"
"fmt"
"os"
)
func Readln(r *bufio.Reader) (string, error) {
var (
isPrefix bool = true
err error = nil
line, ln []byte
)
for isPrefix && err == nil {
line, isPrefix, err = r.ReadLine()
ln = append(ln, line...)
}
return string(ln), err
}
func main() {
f, err := os.Open("tickers.txt")
if err != nil {
fmt.Printf("error opening file: %v\n", err)
os.Exit(1)
}
r := bufio.NewReader(f)
s, e := Readln(r)
for e == nil {
fmt.Println(s)
s, e = Readln(r)
}
}
I therefore suspect that the problem is in your tickers.txt file line endings. The docs for ReadLine() also indicate that for most situations a Scanner is more suitable.
The following SO question has some useful information for alternative implementations: reading file line by line in go
I then used the example in the above question to re-implement your main function as follows:
f, err := os.Open("tickers.txt")
if err != nil {
fmt.Printf("error opening file: %v\n", err)
os.Exit(1)
}
scanner := bufio.NewScanner(f)
for scanner.Scan() {
fmt.Println(scanner.Text())
}
if err := scanner.Err(); err != nil {
fmt.Println(err)
}
Related
I got a binary that works like the below:
> ./my_bin raw.avi output_file.avi
output_file.avi is what I want, some verbose information will print in the terminal when the job is succeeded, like:
Copyright 2022 Company Inc... Success.
I want to run this command inside my code and redirect the output_file.avi into some byte array so that I don't have to read it from disk and delete it. My approach looks like the below Golang snippet:
func wrongOne(stdin []byte) ([]byte, error) {
inBuf := bytes.NewBuffer(stdin)
outBuf := bytes.NewBuffer(nil)
cmd := exec.Command("./my_bin", "/dev/stdin", "/dev/stdout")
cmd.Stdin = inBuf
cmd.Stdout = outBuf
err := cmd.Run()
if err != nil {
return nil, err
}
return outBuf.Bytes(), nil // wrong
}
However, the return byte array is longer than the below approach, which leads to failure on the MD5 check.
func correctOne(stdin []byte) ([]byte, error) {
inBuf := bytes.NewBuffer(stdin)
cmd := exec.Command("./my_bin", "/dev/stdin", "output_file")
cmd.Stdin = inBuf
err := cmd.Run()
if err != nil {
return nil, err
}
return os.ReadFile("output_file")
}
the wrongOne function can be modified to following code to be correct:
func modifiedWrongOne(stdin []byte) ([]byte, error) {
inBuf := bytes.NewBuffer(stdin)
outBuf := bytes.NewBuffer(nil)
cmd := exec.Command("./my_bin", "/dev/stdin", "/dev/stdout")
cmd.Stdin = inBuf
cmd.Stdout = outBuf
err := cmd.Run()
if err != nil {
return nil, err
}
correct, _ := correctOne(stdin)
return outBuf.Bytes()[:len(correct)], nil // diff
}
I presume that the output verbose information is included in the /dev/stdout so that the wrongOne function doesn't works. i.e.,
the output of wrongOne = the output of correctOne + []byte{"Copyright 2022 Company Inc... Success."}
Is there any solution that I can get the output_file.avi in the pipe without save it as file and read it from disk? Thanks!
The command writes the copyright notice to stdout. To avoid commingling the copyright notice with the output file, use a file other than /dev/stdout as the output file.
The function below uses Cmd.ExtraFiles to connect a pipe to fd 3 in the child process. The function copies data from the pipe to a byte buffer and returns those bytes to the caller.
func otherOne(stdin []byte) ([]byte, error) {
r, w, err := os.Pipe()
if err != nil {
return nil, err
}
defer r.Close()
defer w.Close()
cmd := exec.Command("./my_bin", "/dev/stdin", "/dev/fd/3")
cmd.Stdin = bytes.NewReader(stdin)
cmd.ExtraFiles = []*os.File{w} // The first file is fd 3.
if err := cmd.Start(); err != nil {
return nil, err
}
w.Close()
var outbuf bytes.Buffer
if _, err := io.Copy(&outbuf, r); err != nil {
return nil, err
}
if err := cmd.Wait(); err != nil {
return nil, err
}
return outbuf.Bytes(), nil
}
After months, I figure out another way to solve this problem. The basic idea is similar with Cerise, i.e., using /dev/fd/3 to redirect the output file. After that, we redirect /dev/fd/3 to /dev/stdout, verbose log to /dev/stderr
by 3>&1, 1>&2, respectively. An additional gen.sh is added. Here's the solution:
#gen.sh
./mybin /dev/stdin /dev/fd/3 3>&1 1>&2
// gen.go
func gen(stdin []byte) ([]byte, error) {
inBuf := bytes.NewBuffer(stdin)
outBuf := bytes.NewBuffer(nil)
cmd := exec.Command("./gen.sh")
cmd.Stdin = inBuf
cmd.Stdout = outBuf
cmd.Stderr = os.Stderr
err := cmd.Run()
if err != nil {
return nil, err
}
return outBuf.Bytes(), nil
}
On Linux I use the following code to enable and disable console echo:
unix.IoctlGetTermios(int(os.Stdin.Fd()), unix.TCGETS)
unix.IoctlSetTermios(unix.Stdout, unix.TCSETS, term)
But it won't compile on Mac, what should I use on Mac?
You could use TIOCGETA and TIOCSETA instead on Mac OS
// IoctlSetTermios performs an ioctl on fd with a *Termios.
//
// The req value will usually be TCSETA or TIOCSETA.
func IoctlSetTermios(fd int, req uint, value *Termios) error {
import (
"bufio"
"fmt"
"log"
"os"
"golang.org/x/sys/unix"
)
func main() {
STDIN := int(os.Stdin.Fd())
tio, err := unix.IoctlGetTermios(STDIN, unix.TIOCGETA)
if err != nil {
log.Fatal(err)
}
fmt.Println("Input sth:")
tio.Lflag &^= unix.ECHO
err = unix.IoctlSetTermios(STDIN, unix.TIOCSETA, tio)
if err != nil {
log.Fatal(err)
}
reader := bufio.NewReader(os.Stdin)
line, _ := reader.ReadString('\n')
tio.Lflag |= unix.ECHO
err = unix.IoctlSetTermios(STDIN, unix.TIOCSETA, tio)
if err != nil {
log.Fatal(err)
}
fmt.Println(line)
}
Instead of using unix.TCGETS / unix.TIOCGETA or unix.TCSETS / unix.TIOCSETA you can define constants in different files and then use build constraints to build those files based on the operating system.
You can check how github.com/moby/term does it here:
https://github.com/moby/term/blob/c43b287e0e0f2460e4ba913936af2f6bdcbe9ed5/termios_bsd.go
https://github.com/moby/term/blob/c43b287e0e0f2460e4ba913936af2f6bdcbe9ed5/termios_nonbsd.go
With this method your code is decoupled from the operating system and you can always use the same constant.
In Go, I am trying to create a function that reads and processes the next line of input:
// Read a string of hex from stdin and parse to an array of bytes
func ReadHex() []byte {
r := bufio.NewReader(os.Stdin)
t, _ := r.ReadString('\n')
data, _ := hex.DecodeString(strings.TrimSpace(t))
return data
}
Unfortunately, this only works the first time it is called. It captures the first line but is unable to capture subsequent lines piped via standard input.
I suspect, if the same persistent bufio.Reader() object was used on each subsequent call, it would work but I haven't been able to achieve this without passing it manually on each function call.
Yes, try this:
package main
import (
"bufio"
"encoding/hex"
"fmt"
"log"
"os"
"strings"
)
func ReadFunc() func() []byte {
r := bufio.NewReader(os.Stdin)
return func() []byte {
t, err := r.ReadString('\n')
if err != nil {
log.Fatal(err)
}
data, err := hex.DecodeString(strings.TrimSpace(t))
if err != nil {
log.Fatal(err)
}
return data
}
}
func main() {
r, w, err := os.Pipe()
if err != nil {
log.Fatal(err)
}
os.Stdin = r
w.Write([]byte(`ffff
cafebabe
ff
`))
w.Close()
ReadHex := ReadFunc()
fmt.Println(ReadHex())
fmt.Println(ReadHex())
fmt.Println(ReadHex())
}
Output:
[255 255]
[202 254 186 190]
[255]
Using a struct, try this:
package main
import (
"bufio"
"encoding/hex"
"fmt"
"io"
"log"
"os"
"strings"
)
// InputReader struct
type InputReader struct {
bufio.Reader
}
// New creates an InputReader
func New(rd io.Reader) *InputReader {
return &InputReader{Reader: *bufio.NewReader(rd)}
}
// ReadHex returns a string of hex from stdin and parse to an array of bytes
func (r *InputReader) ReadHex() []byte {
t, err := r.ReadString('\n')
if err != nil {
log.Fatal(err)
}
data, err := hex.DecodeString(strings.TrimSpace(t))
if err != nil {
log.Fatal(err)
}
return data
}
func main() {
r, w, err := os.Pipe()
if err != nil {
log.Fatal(err)
}
os.Stdin = r
w.Write([]byte(`ffff
cafebabe
ff
`))
w.Close()
rdr := New(os.Stdin)
fmt.Println(rdr.ReadHex())
fmt.Println(rdr.ReadHex())
fmt.Println(rdr.ReadHex())
}
Using this file (data file):
package main
import (
"io/ioutil"
"time"
)
func main() {
ioutil.ReadFile("100mb.file")
time.Sleep(time.Duration(time.Minute))
}
Showed memory usage for me of 107 MB. With this similar file:
package main
import (
"bytes"
"os"
"time"
)
func read(path_s string) (bytes.Buffer, error) {
buf_o := bytes.Buffer{}
open_o, e := os.Open(path_s)
if e != nil {
return buf_o, e
}
buf_o.ReadFrom(open_o)
open_o.Close()
return buf_o, nil
}
func main() {
read("100mb.file")
time.Sleep(time.Duration(time.Minute))
}
Memory usage went to 273 MB. Finally this similar file:
package main
import (
"io"
"os"
"strings"
"time"
)
func read(path_s string) (strings.Builder, error) {
str_o := strings.Builder{}
open_o, e := os.Open(path_s)
if e != nil {
return str_o, e
}
io.Copy(&str_o, open_o)
open_o.Close()
return str_o, nil
}
func main() {
read("100mb.file")
time.Sleep(time.Duration(time.Minute))
}
Memory usage went to 432 MB. I tried to be careful and close files where
possible. Why is the memory usage so high for the second example, and especially
the final example? Can I change something so that they are closer to the first
example?
ioutil.ReadFile("100mb.file") gets the size of the file, allocates a []byte that size and slurps the bytes up into that slice.
buf_o.ReadFrom(open_o) allocates an initial []byte of some size and reads into that slice. If there's more data in the reader than space in the slice, then the function allocates a larger slice, copies existing data to that slice and reads more. This repeats until EOF.
The function ioutil.ReadFile uses bytes Buffer.ReadFrom internally. Take a look at the ioutil.ReadFile implementation to see how to improve direct use of the bytes.Buffer. A synopsis of the logic is this:
var buf bytes.Buffer
// Open file
f, err := os.Open(path)
if err != nil {
return &buf, err
}
defer f.Close()
// Get size.
fi, err := f.Stat()
if err != nil {
return &buf, err
}
// Grow to size of file plus extra slop to ensure no realloc.
buf.Grow(int(fi.Size()) + bytes.MinRead)
_, err := buf.ReadFrom(f)
return &buf, err
The strings.Builder example reallocates the internal buffer several times as in the bytes.Buffer example. In addition, io.Copy allocates a buffer. You can improve the strings.Builder example by growing the builder to the size of the file before reading.
Here's the code for strings.Builder:
var buf strings.Builder
// Open file
f, err := os.Open(path)
if err != nil {
return &buf, err
}
defer f.Close()
// Get size.
fi, err := f.Stat()
if err != nil {
return &buf, err
}
buf.Grow(int(fi.Size()))
_, err = io.Copy(&buf, f)
return &buf, err
io.Copy or some other code using an extra buffer is required because strings.Builder does not have a ReadFrom method. The strings.Builder type does not have a ReadFrom method because that method can leak a reference to the backing array of the internal slice of bytes.
Using suggestion from Muffin Top, I took my second example and added this
directly before the call to ReadFrom:
stat_o, e := open_o.Stat()
if e != nil {
return buf_o, e
}
buf_o.Grow(bytes.MinRead + int(stat_o.Size()))
and the memory went down to 107 MB, basically the same as the first example.
I have the following Go code which will eventually fill the disk and fail with ENOSPC (just a proof of concept). How can I determine from the err returned by os.Write that it indeed failed because of ENOSPC (so I need a way to grab errno after the write operation) ?
package main
import (
"log"
"os"
)
func main() {
fd, _ := os.Create("dump.txt")
defer fd.Close()
for {
buf := make([]byte, 1024)
_, err := fd.Write(buf)
if err != nil {
log.Fatalf("%T %v", err, err)
}
}
}
EDIT: Updated the program as #FUZxxl suggested:
package main
import (
"log"
"os"
"syscall"
)
func main() {
fd, _ := os.Create("dump.txt")
defer fd.Close()
for {
buf := make([]byte, 1024)
_, err := fd.Write(buf)
if err != nil {
log.Printf("%T %v\n", err, err)
errno, ok := err.(syscall.Errno)
if ok {
log.Println("type assert ok")
if errno == syscall.ENOSPC {
log.Println("got ENOSPC")
}
} else {
log.Println("type assert not ok")
}
break
}
}
}
However, I'm not getting the expected result. Here is the output:
2015/02/15 10:13:27 *os.PathError write dump.txt: no space left on device
2015/02/15 10:13:27 type assert not ok
File operations generally return an *os.PathError; cast err to os.PathError and use the Err field to examine the underlying cause, like this:
patherr, ok := err.(*os.PathError)
if ok && patherr.Err == syscall.ENOSPC {
log.Println("Out of disk space!")
}