Unable to process strings in files with different encoding [Not accepting answers] - string

Question

I am trying to process the strings present in a particular file, The file is written in English. The problem arises when the encoding of the file differs from "UTF-8". But the file with encoding as "UTF-16-le" does not behave as expected. My main goal is manipulate the strings from the read file. For example the strings.TrimSpace() only works with the UTF-8 file,
I am aware that golang only supports UTF-8 by default, Any alternate approach would be helpful.
Personal Question
Also I would like to point out, many new programming languages, do process the strings irrespective of the encoding, And why does Go only support UTF-8. If at least there would be an alternative way to pass the encoding format to the reader, that might still help.
What I tried
I tried using utf-8 and utf-16 std packages
Code
(main.go)
sample code to show the difference.
package main
import (
"fmt"
"io/ioutil"
"net/http"
"strings"
)
func processFile(src string) {
data, _ := ioutil.ReadFile(src)
fmt.Println("--- original source ---")
fmt.Println(string(data))
fmt.Println(http.DetectContentType(data))
fmt.Println("\n--- modified source ---")
for _, val := range strings.Split(string(data), "\n") {
fmt.Println(strings.TrimSpace(val))
}
}
func main() {
processFile("./utf-16-english.txt")
processFile("./utf-8-english.txt")
}
File-1
(utf-8-english.txt)
Hello
This is
Sample
Document
File-2
(utf-16-english.txt)
Hello
This is
Sample
Document
EDIT
Seems that the only way to process strings in a better way is to convert them to UTF-8. Kindly refer the marked answer.
As per comments I have written the result from the program to respective files. And the special symbols are not present, but the process with strings, works fine with UTF-8

Answer 1

You have to decode the utf-16 encoded file. The decoding will convert the input to utf-8, after which you can use the string libraries to process the input.
You can use something like this:
import "unicode/utf16"
func processFile(src string, decode func(in[]byte) string) {
data, _ := ioutil.ReadFile(src)
fmt.Println("--- original source ---")
fmt.Println(decode(data))
fmt.Println("\n--- modified source ---")
for _, val := range strings.Split(decode(data), "\n") {
fmt.Println(strings.TrimSpace(val))
}
}
func main() {
processFile("./utf-16-english.txt",func(in []byte) string {
return string(utf16.Decode(in)) })
processFile("./utf-8-english.txt",func(in []byte) string {
return string(in)})
}