What's the best way (fastest performance) to convert from []int8 to string?
For []byte we could do string(byteslice), but for []int8 it gives an error:
cannot convert ba (type []int8) to type string
I got the ba from SliceScan() method of *sqlx.Rows that produces []int8 instead of string
Is this solution the fastest?
func B2S(bs []int8) string {
ba := []byte{}
for _, b := range bs {
ba = append(ba, byte(b))
}
return string(ba)
}
EDIT my bad, it's uint8 instead of int8.. so I can do string(ba) directly.
Note beforehand: The asker first stated that input slice is []int8 so that is what the answer is for. Later he realized the input is []uint8 which can be directly converted to string because byte is an alias for uint8 (and []byte => string conversion is supported by the language spec).
You can't convert slices of different types, you have to do it manually.
Question is what type of slice should we convert to? We have 2 candidates: []byte and []rune. Strings are stored as UTF-8 encoded byte sequences internally ([]byte), and a string can also be converted to a slice of runes. The language supports converting both of these types ([]byte and []rune) to string.
A rune is a unicode codepoint. And if we try to convert an int8 to a rune in a one-to-one fashion, it will fail (meaning wrong output) if the input contains characters which are encoded to multiple bytes (using UTF-8) because in this case multiple int8 values should end up in one rune.
Let's start from the string "世界" whose bytes are:
fmt.Println([]byte("世界"))
// Output: [228 184 150 231 149 140]
And its runes:
fmt.Println([]rune("世界"))
// [19990 30028]
It's only 2 runes and 6 bytes. So obviously 1-to-1 int8->rune mapping won't work, we have to go with 1-1 int8->byte mapping.
byte is alias for uint8 having range 0..255, to convert it to []int8 (having range -128..127) we have to use -256+bytevalue if the byte value is > 127 so the "世界" string in []int8 looks like this:
[-28 -72 -106 -25 -107 -116]
The backward conversion what we want is: bytevalue = 256 + int8value if the int8 is negative but we can't do this as int8 (range -128..127) and neither as byte (range 0..255) so we also have to convert it to int first (and back to byte at the end). This could look something like this:
if v < 0 {
b[i] = byte(256 + int(v))
} else {
b[i] = byte(v)
}
But actually since signed integers are represented using 2's complement, we get the same result if we simply use a byte(v) conversion (which in case of negative numbers this is equivalent to 256 + v).
Note: Since we know the length of the slice, it is much faster to allocate a slice with this length and just set its elements using indexing [] and not calling the built-in append function.
So here is the final conversion:
func B2S(bs []int8) string {
b := make([]byte, len(bs))
for i, v := range bs {
b[i] = byte(v)
}
return string(b)
}
Try it on the Go Playground.
Not entirely sure it is the fastest, but I haven't found anything better.
Change ba := []byte{} for ba := make([]byte,0, len(bs) so at the end you have:
func B2S(bs []int8) string {
ba := make([]byte,0, len(bs))
for _, b := range bs {
ba = append(ba, byte(b))
}
return string(ba)
}
This way the append function will never try to insert more data that it can fit in the slice's underlying array and you will avoid unnecessary copying to a bigger array.
What is sure from "Convert between slices of different types" is that you have to build the right slice from your original int8[].
I ended up using rune (int32 alias) (playground), assuming that the uint8 were all simple ascii character. That is obviously an over-simplification and icza's answer has more on that.
Plus the SliceScan() method ended up returning uint8[] anyway.
package main
import (
"fmt"
)
func main() {
s := []int8{'a', 'b', 'c'}
b := make([]rune, len(s))
for i, v := range s {
b[i] = rune(v)
}
fmt.Println(string(b))
}
But I didn't benchmark it against using a []byte.
Use unsafe package.
func B2S(bs []int8) string {
return strings.TrimRight(string(*(*[]byte)unsafe.Pointer(&bs)), "\x00")
}
Send again ^^
Related
I have a string like this:
package main
import "fmt"
func main() {
some := "p1k4"
for i, j := range some {
fmt.Println()
}
}
I want take each two consecutive characters in the string and print them. the output should like p1, 1k, k4, 4p.
I have tried it and still having trouble finding the answer, how should I write the code in go and get the output I want?
Go stores strings in memory as their UTF-8 encoded byte sequence. This maps ASCII charactes one-to-one in bytes, but characters outside of that range map to multiple bytes.
So I would advise to use the for range loop over a string, which ranges over the runes (characters) of the string, properly decoding multi-byte runes. This has the advantage that it does not require allocation (unlike converting the string to []rune). You may also print the pairs using fmt.Printf("%c%c", char1, char2), which also will not require allocation (unlike converting runes back to string and concatenating them).
To learn more about strings, characters and runes in Go, read blog post: Strings, bytes, runes and characters in Go
Since the loop only returns the "current" rune in the iteration (but not the previous or the next rune), use another variable to store the previous (and first) runes so you have access to them when printing.
Let's write a function that prints the pairs as you want:
func printPairs(s string) {
var first, prev rune
for i, r := range s {
if i == 0 {
first, prev = r, r
continue
}
fmt.Printf("%c%c, ", prev, r)
prev = r
}
// Print last pair: prev is the last rune
fmt.Printf("%c%c\n", prev, first)
}
Testing it with your input and with another string that has multi-byte runes:
printPairs("p1k4")
printPairs("Go-世界")
Output will be (try it on the Go Playground):
p1, 1k, k4, 4p
Go, o-, -世, 世界, 界G
package main
import (
"fmt"
)
func main() {
str := "12345"
for i := 0; i < len(str); i++ {
fmt.Println(string(str[i]) + string(str[(i+1)%len(str)]))
}
}
This is a simple for loop over your string with the first character appended at the back:
package main
import "fmt"
func main() {
some := "p1k4"
ns := some + string(some[0])
for i := 0; i < len(ns)-1; i++ {
fmt.Println(ns[i:i+2])
}
}
How do I get the index of a substring in a string skipping starting with a certain position/with a certain offset, e.g.:
package main
import (
"fmt"
"strings"
)
func main() {
string := "something.value=something=end"
index1 := strings.Index(string, "value=")
fmt.Println(index1) // prints 10
// index2 = ... How do I get the position of the second =, 25?
}
Similar offset in PHP int strpos ( string $haystack , mixed $needle [, int $offset = 0 ] )
The strings package does not provide you such function, but in practice it's rarely needed. Often the strings.Split() function is used to easily split strings into tokens / parts.
But if you do need it: you can simply slice a string, which is efficient (no copy is made, the result shares memory with the original string value).
So effectively the function you're looking for would look like this:
func Index(s, substr string, offset int) int {
if len(s) < offset {
return -1
}
if idx := strings.Index(s[offset:], substr); idx >= 0 {
return offset + idx
}
return -1
}
Example using it:
s := "something.value=something=end"
index1 := strings.Index(s, "value=")
fmt.Println(index1) // prints 10
index2 := Index(s, "=", index1+len("value="))
fmt.Println(index2) // prints 25
Output (try it on the Go Playground):
10
25
Note that when slicing a string, and the offset you have to pass to our Index() function is the byte index, not the rune (character) index. They are equal as long as you have characters with less than 128 codepoints, but beyond that the byte index will be greater than the rune index because those codepoints map to multiple bytes in UTF-8 encoding (which is how Go stores strings in memory). strings.Index() returns you the byte index, and len(s) also returns you the byte-length, so the example will work with all strings properly.
Your original task using strings.Split() could look like this:
s := "something.value=something=end"
parts := strings.Split(s, "=")
fmt.Println(parts)
Which outputs (try it on the Go Playground):
[something.value something end]
The value you want to "parse" out is in parts[1].
taking a slice of string with utf-8 characters may produce corrupted strings as you need to convert it to runes
[]rune(videoHtml)[0:index]
I have the following code, it is supposed to cast a rune into a string and print it. However, I am getting undefined characters when it is printed. I am unable to figure out where the bug is:
package main
import (
"fmt"
"strconv"
"strings"
"text/scanner"
)
func main() {
var b scanner.Scanner
const a = `a`
b.Init(strings.NewReader(a))
c := b.Scan()
fmt.Println(strconv.QuoteRune(c))
}
That's because you used Scanner.Scan() to read a rune but it does something else. Scanner.Scan() can be used to read tokens or runes of special tokens controlled by the Scanner.Mode bitmask, and it returns special constants form the text/scanner package, not the read rune itself.
To read a single rune use Scanner.Next() instead:
c := b.Next()
fmt.Println(c, string(c), strconv.QuoteRune(c))
Output:
97 a 'a'
If you just want to convert a single rune to string, use a simple type conversion. rune is alias for int32, and converting integer numbers to string:
Converting a signed or unsigned integer value to a string type yields a string containing the UTF-8 representation of the integer.
So:
r := rune('a')
fmt.Println(r, string(r))
Outputs:
97 a
Also to loop over the runes of a string value, you can simply use the for ... range construct:
for i, r := range "abc" {
fmt.Printf("%d - %c (%v)\n", i, r, r)
}
Output:
0 - a (97)
1 - b (98)
2 - c (99)
Or you can simply convert a string value to []rune:
fmt.Println([]rune("abc")) // Output: [97 98 99]
There is also utf8.DecodeRuneInString().
Try the examples on the Go Playground.
Note:
Your original code (using Scanner.Scan()) works like this:
You called Scanner.Init() which sets the Mode (b.Mode) to scanner.GoTokens.
Calling Scanner.Scan() on the input (from "a") returns scanner.Ident because "a" is a valid Go identifier:
c := b.Scan()
if c == scanner.Ident {
fmt.Println("Identifier:", b.TokenText())
}
// Output: "Identifier: a"
I know I'm a bit late to the party but here's a []rune to string function:
func runesToString(runes []rune) (outString string) {
// don't need index so _
for _, v := range runes {
outString += string(v)
}
return
}
yes, there is a named return but I think it's ok in this case as it reduces the number of lines and the function is only short
This simple code works in converting a rune to a string
s := fmt.Sprintf("%c", rune)
Since I came to this question searching for rune and string and char, thought this may help newbies like me
// str := "aഐbc"
// testString(str)
func testString(oneString string){
//string to byte slice - No sweat -just type cast it
// As string IS A byte slice
var twoByteArr []byte = []byte(oneString)
// string to rune Slices - No sweat
// string IS A slice of runes
var threeRuneSlice []rune = []rune(oneString)
// Hmm! String seems to have a dual personality it is both a slice of bytes and
// a slice of runes - yeah - read on
// A rune slice can be convered to string -
// No sweat - as string == rune slice
var thrirdString string = string(threeRuneSlice)
// There is a catch here and that is in printing "characters", using for loop and range
fmt.Println("Chars in oneString")
for i,r := range oneString {
fmt.Printf(" %d %v %c ",i,r,r) //you may not get index 0,1,2,3 here
// since the range runs specially over strings https://blog.golang.org/strings
}
fmt.Println("\nChars in threeRuneSlice")
for i,r := range threeRuneSlice {
fmt.Printf(" %d %v %c ",i,r,r) // i = 0,1,2,4 , perfect!!
// as runes are made up of 4 bytes (rune is int32 and byte in unint8
// and a set of bytes is used to represent a rune which is used to
// represent UTF characters == the REAL CHARECTER
}
fmt.Println("\nValues in oneString ")
for j := 0; j < len(oneString); j++ {
fmt.Printf(" %d %v ",j,oneString[j]) // No you cannot get charecters if you iterate through string in this way
// as you are going over bytes here - not runes
}
fmt.Println("\nValues in twoByteArr")
for j := 0; j < len(twoByteArr); j++ {
fmt.Printf(" %d=%v ",j,twoByteArr[j]) // == same as above
}
fmt.Printf("\none - %s, two %s, three %s\n",oneString,twoByteArr,thrirdString)
}
And some more pointless demo https://play.golang.org/p/tagRBVG8k7V
adapted from https://groups.google.com/g/golang-nuts/c/84GCvDBhpbg/m/Tt6089MPFQAJ
to show that the 'characters' are encoded with one to up to 4 bytes depending on the unicode code point
Provide simple examples to understand how to do it quickly.
// rune => string
fmt.Printf("%c\n", 65) // A
fmt.Println(string(rune(0x1F60A))) // 😊
fmt.Println(string([]rune{0x1F468, 0x200D, 0x1F9B0})) // 👨🦰
// string => rune
fmt.Println(strconv.FormatUint(uint64([]rune("😊")[0]), 16)) // 1f60a
fmt.Printf("%U\n", '😊') // U+1F60A
fmt.Printf("%U %U %U\n", '👨', '', '🦰') // U+1F468 U+200D U+1F9B0
go playground
This is just in case someone else is learning Golang and is wondering how to convert from a string to a string representation in binary.
Long story short, I have been looking at the standard library without being able to find the right call. So I started with something similar to the following:
func RuneToBinary(r rune) string {
var buf bytes.Buffer
b := []int64{128, 64, 32, 16, 8, 4, 2, 1}
v := int64(r)
for i := 0; i < len(b); i++ {
t := v-b[i]
if t >= 0 {
fmt.Fprintf(&buf, "1")
v = t
} else {
fmt.Fprintf(&buf, "0")
}
}
return buf.String()
}
This is all well and dandy, but after a couple of days looking around I found that I should have been using the fmt package instead and just format the rune with %b%:
var r rune
fmt.Printf("input: %b ", r)
Is there a better way to do this?
Thanks
Standard library support
fmt.Printf("%b", r) - this solution is already very compact and easy to write and understand. If you need the result as a string, you can use the analog Sprintf() function:
s := fmt.Sprintf("%b", r)
You can also use the strconv.FormatInt() function which takes a number of type int64 (so you first have to convert your rune) and a base where you can pass 2 to get the result in binary representation:
s := strconv.FormatInt(int64(r), 2)
Note that in Go rune is just an alias for int32, the 2 types are one and the same (just you may refer to it by 2 names).
Doing it manually ("Simple but Naive"):
If you'd want to do it "manually", there is a much simpler solution than your original. You can test the lowest bit with r & 0x01 == 0 and shift all bits with r >>= 1. Just "loop" over all bits and append either "1" or "0" depending on the bit:
Note this is just for demonstration, it is nowhere near optimal regarding performance (generates "redundant" strings):
func RuneToBin(r rune) (s string) {
if r == 0 {
return "0"
}
for digits := []string{"0", "1"}; r > 0; r >>= 1 {
s = digits[r&1] + s
}
return
}
Note: negative numbers are not handled by the function. If you also want to handle negative numbers, you can first check it and proceed with the positive value of it and start the return value with a minus '-' sign. This also applies the other manual solution below.
Manual Performance-wise solution:
For a fast solution we shouldn't append strings. Since strings in Go are just byte slices encoded using UTF-8, appending a digit is just appending the byte value of the rune '0' or '1' which is just one byte (not multi). So we can allocate a big enough buffer/array (rune is 32 bits so max 32 binary digits), and fill it backwards so we won't even have to reverse it at the end. And return the used part of the array converted to string at the end. Note that I don't even call the built-in append function to append the binary digits, I just set the respective element of the array in which I build the result:
func RuneToBinFast(r rune) string {
if r == 0 {
return "0"
}
b, i := [32]byte{}, 31
for ; r > 0; r, i = r>>1, i-1 {
if r&1 == 0 {
b[i] = '0'
} else {
b[i] = '1'
}
}
return string(b[i+1:])
}
How can I get the number of characters of a string in Go?
For example, if I have a string "hello" the method should return 5. I saw that len(str) returns the number of bytes and not the number of characters so len("£") returns 2 instead of 1 because £ is encoded with two bytes in UTF-8.
You can try RuneCountInString from the utf8 package.
returns the number of runes in p
that, as illustrated in this script: the length of "World" might be 6 (when written in Chinese: "世界"), but the rune count of "世界" is 2:
package main
import "fmt"
import "unicode/utf8"
func main() {
fmt.Println("Hello, 世界", len("世界"), utf8.RuneCountInString("世界"))
}
Phrozen adds in the comments:
Actually you can do len() over runes by just type casting.
len([]rune("世界")) will print 2. At least in Go 1.3.
And with CL 108985 (May 2018, for Go 1.11), len([]rune(string)) is now optimized. (Fixes issue 24923)
The compiler detects len([]rune(string)) pattern automatically, and replaces it with for r := range s call.
Adds a new runtime function to count runes in a string.
Modifies the compiler to detect the pattern len([]rune(string))
and replaces it with the new rune counting runtime function.
RuneCount/lenruneslice/ASCII 27.8ns ± 2% 14.5ns ± 3% -47.70%
RuneCount/lenruneslice/Japanese 126ns ± 2% 60 ns ± 2% -52.03%
RuneCount/lenruneslice/MixedLength 104ns ± 2% 50 ns ± 1% -51.71%
Stefan Steiger points to the blog post "Text normalization in Go"
What is a character?
As was mentioned in the strings blog post, characters can span multiple runes.
For example, an 'e' and '◌́◌́' (acute "\u0301") can combine to form 'é' ("e\u0301" in NFD). Together these two runes are one character.
The definition of a character may vary depending on the application.
For normalization we will define it as:
a sequence of runes that starts with a starter,
a rune that does not modify or combine backwards with any other rune,
followed by possibly empty sequence of non-starters, that is, runes that do (typically accents).
The normalization algorithm processes one character at at time.
Using that package and its Iter type, the actual number of "character" would be:
package main
import "fmt"
import "golang.org/x/text/unicode/norm"
func main() {
var ia norm.Iter
ia.InitString(norm.NFKD, "école")
nc := 0
for !ia.Done() {
nc = nc + 1
ia.Next()
}
fmt.Printf("Number of chars: %d\n", nc)
}
Here, this uses the Unicode Normalization form NFKD "Compatibility Decomposition"
Oliver's answer points to UNICODE TEXT SEGMENTATION as the only way to reliably determining default boundaries between certain significant text elements: user-perceived characters, words, and sentences.
For that, you need an external library like rivo/uniseg, which does Unicode Text Segmentation.
That will actually count "grapheme cluster", where multiple code points may be combined into one user-perceived character.
package uniseg
import (
"fmt"
"github.com/rivo/uniseg"
)
func main() {
gr := uniseg.NewGraphemes("👍🏼!")
for gr.Next() {
fmt.Printf("%x ", gr.Runes())
}
// Output: [1f44d 1f3fc] [21]
}
Two graphemes, even though there are three runes (Unicode code points).
You can see other examples in "How to manipulate strings in GO to reverse them?"
👩🏾🦰 alone is one grapheme, but, from unicode to code points converter, 4 runes:
👩: women (1f469)
dark skin (1f3fe)
ZERO WIDTH JOINER (200d)
🦰red hair (1f9b0)
There is a way to get count of runes without any packages by converting string to []rune as len([]rune(YOUR_STRING)):
package main
import "fmt"
func main() {
russian := "Спутник и погром"
english := "Sputnik & pogrom"
fmt.Println("count of bytes:",
len(russian),
len(english))
fmt.Println("count of runes:",
len([]rune(russian)),
len([]rune(english)))
}
count of bytes 30 16
count of runes 16 16
I should point out that none of the answers provided so far give you the number of characters as you would expect, especially when you're dealing with emojis (but also some languages like Thai, Korean, or Arabic). VonC's suggestions will output the following:
fmt.Println(utf8.RuneCountInString("🏳️🌈🇩🇪")) // Outputs "6".
fmt.Println(len([]rune("🏳️🌈🇩🇪"))) // Outputs "6".
That's because these methods only count Unicode code points. There are many characters which can be composed of multiple code points.
Same for using the Normalization package:
var ia norm.Iter
ia.InitString(norm.NFKD, "🏳️🌈🇩🇪")
nc := 0
for !ia.Done() {
nc = nc + 1
ia.Next()
}
fmt.Println(nc) // Outputs "6".
Normalization is not really the same as counting characters and many characters cannot be normalized into a one-code-point equivalent.
masakielastic's answer comes close but only handles modifiers (the rainbow flag contains a modifier which is thus not counted as its own code point):
fmt.Println(GraphemeCountInString("🏳️🌈🇩🇪")) // Outputs "5".
fmt.Println(GraphemeCountInString2("🏳️🌈🇩🇪")) // Outputs "5".
The correct way to split Unicode strings into (user-perceived) characters, i.e. grapheme clusters, is defined in the Unicode Standard Annex #29. The rules can be found in Section 3.1.1. The github.com/rivo/uniseg package implements these rules so you can determine the correct number of characters in a string:
fmt.Println(uniseg.GraphemeClusterCount("🏳️🌈🇩🇪")) // Outputs "2".
If you need to take grapheme clusters into account, use regexp or unicode module. Counting the number of code points(runes) or bytes also is needed for validaiton since the length of grapheme cluster is unlimited. If you want to eliminate extremely long sequences, check if the sequences conform to stream-safe text format.
package main
import (
"regexp"
"unicode"
"strings"
)
func main() {
str := "\u0308" + "a\u0308" + "o\u0308" + "u\u0308"
str2 := "a" + strings.Repeat("\u0308", 1000)
println(4 == GraphemeCountInString(str))
println(4 == GraphemeCountInString2(str))
println(1 == GraphemeCountInString(str2))
println(1 == GraphemeCountInString2(str2))
println(true == IsStreamSafeString(str))
println(false == IsStreamSafeString(str2))
}
func GraphemeCountInString(str string) int {
re := regexp.MustCompile("\\PM\\pM*|.")
return len(re.FindAllString(str, -1))
}
func GraphemeCountInString2(str string) int {
length := 0
checked := false
index := 0
for _, c := range str {
if !unicode.Is(unicode.M, c) {
length++
if checked == false {
checked = true
}
} else if checked == false {
length++
}
index++
}
return length
}
func IsStreamSafeString(str string) bool {
re := regexp.MustCompile("\\PM\\pM{30,}")
return !re.MatchString(str)
}
There are several ways to get a string length:
package main
import (
"bytes"
"fmt"
"strings"
"unicode/utf8"
)
func main() {
b := "这是个测试"
len1 := len([]rune(b))
len2 := bytes.Count([]byte(b), nil) -1
len3 := strings.Count(b, "") - 1
len4 := utf8.RuneCountInString(b)
fmt.Println(len1)
fmt.Println(len2)
fmt.Println(len3)
fmt.Println(len4)
}
Depends a lot on your definition of what a "character" is. If "rune equals a character " is OK for your task (generally it isn't) then the answer by VonC is perfect for you. Otherwise, it should be probably noted, that there are few situations where the number of runes in a Unicode string is an interesting value. And even in those situations it's better, if possible, to infer the count while "traversing" the string as the runes are processed to avoid doubling the UTF-8 decode effort.
I tried to make to do the normalization a bit faster:
en, _ = glyphSmart(data)
func glyphSmart(text string) (int, int) {
gc := 0
dummy := 0
for ind, _ := range text {
gc++
dummy = ind
}
dummy = 0
return gc, dummy
}