I've got a string with milliseconds since the epoch (it came originally from a java.lang.System.currentTimeMillis() call). What's the right way to convert this string into a human readable timestamp string in Go?
Looking at the time package I see the Parse function, but the layouts all seem to be normal timezone-based times. Once into a Time object, I can use Format(Time.Stamp) to get the output I want, but I'm not clear on how to get the string into a Time object.
The format string does not support milliseconds since the epoch, so you need to parse manually. For example:
func msToTime(ms string) (time.Time, error) {
msInt, err := strconv.ParseInt(ms, 10, 64)
if err != nil {
return time.Time{}, err
}
return time.Unix(0, msInt*int64(time.Millisecond)), nil
}
Check out http://play.golang.org/p/M1dWGLT8XE to play with the example live.
2021 update: you can use UnixMilli to parse from the integer. This would improve the accepted answer to:
func msToTime(ms string) (time.Time, error) {
msInt, err := strconv.ParseInt(ms, 10, 64)
if err != nil {
return time.Time{}, err
}
return time.UnixMilli(msInt), nil
}
The accepted answer will not work for some date ranges either too far in the future or too far in the past because only a limited date range can be described using 64 bits (plus or minus 300 years from Epoch).
This will work for the entire range describable by milliseconds:
var Epoch = time.Unix(0, 0)
func ParseMillisecondUnixTimestamp(s string) (time.Time, error) {
ts, err := strconv.Atoi(s)
if err != nil {
return time.Time{}, err
}
return Epoch.Add(time.Duration(ts) * time.Millisecond), nil
}
Using time.Unix(ms/1000, 0) would do the trick.
First convert the ms-timestamp to unix-timestamp using the well known relation 1 s = 1000 ms and then use the result as first argument to the time.Unix function
Related
I am creating a program, which is processing and calculating sizes of open-source repositories and libraries, and saving the data to database for further analysis.
I have an input string: github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1.
Parsed to a format: github.com/\!azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1
Then I parse that into a format /home/username/dev/glass/tmp/pkg/mod/github.com/\!azure/go-ansiterm#v0.0.0-20210617225240-d185dfc1b5a1 which is a valid path in my filesystem, where I've downloaded that particular Go Library.
After that, I am passing that path to the gocloc -program (https://github.com/hhatto/gocloc)
And parse the result.
But the issue is, when I am saving that string /home/username/dev/glass/tmp/pkg/mod/github.com/\!azure/go-ansiterm#v0.0.0-20210617225240-d185dfc1b5a1 into a variable, Go actually adds another escape to the string I am saving so it's actually /home/username/dev/glass/tmp/pkg/mod/github.com/\\!azure/go-ansiterm#v0.0.0-20210617225240-d185dfc1b5a1 in memory. (fmt.Println - for example removes that)
Problem is, when I am passing that string as an argument to os/exec, which runs gocloc and that path string, it runs command with two escapes - and that's not a valid path.
Is there any way to work around this? One idea for me is to just a create shell script on what I want to do
This is the function, which parses github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 to a format github.com/\!azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 - and after thats saved into a variable, and the variable has one more escapes, than it should have.
func parseUrlToVendorDownloadFormat(input string) string {
// Split the input string on the first space character
parts := strings.SplitN(input, " ", 2)
if len(parts) != 2 {
return ""
}
// Split the package name on the '/' character
packageNameParts := strings.Split(parts[0], "/")
// Add the '\!' prefix and lowercase each part of the package name
for i, part := range packageNameParts {
if hasUppercase(part) {
packageNameParts[i] = "\\!" + strings.ToLower(part)
}
}
// Join the modified package name parts with '/' characters
packageName := strings.Join(packageNameParts, "/")
return strings.ReplaceAll(packageName+"#"+parts[1], `\\!`, `\!`)
}
After, string is parsed to a format: /home/username/dev/glass/tmp/pkg/mod/github.com/\!azure/go-ansiterm#v0.0.0-20210617225240-d185dfc1b5a1
that is passed to this function:
// Alternative goCloc - command.
func linesOfCode(dir string) (int, error) {
// Run the `gocloc` command in the specified directory and get the output
cmd := exec.Command("gocloc", dir)
output, err := cmd.Output()
if err != nil {
return 0, err
}
lines, err := parseTotalLines(string(output))
if err != nil {
return 0, err
}
return lines, nil
}
Which uses this parse function:
// Parse from the GoCloc response.
func parseTotalLines(input string) (int, error) {
// Split the input string into lines
lines := strings.Split(input, "\n")
// Find the line containing the "TOTAL" row
var totalLine string
for _, line := range lines {
if strings.Contains(line, "TOTAL") {
totalLine = line
break
}
}
// If the "TOTAL" line was not found, return an error
if totalLine == "" {
return 0, fmt.Errorf("could not find TOTAL line in input")
}
// Split the "TOTAL" line into fields
fields := strings.Fields(totalLine)
// If the "TOTAL" line doesn't have enough fields, return an error
if len(fields) < 4 {
return 0, fmt.Errorf("invalid TOTAL line: not enough fields")
}
// Get the fourth field (the code column)
codeStr := fields[3]
// Remove any commas from the code column
codeStr = strings.Replace(codeStr, ",", "", -1)
// Parse the code column as an integer
code, err := strconv.Atoi(codeStr)
if err != nil {
return 0, err
}
return code, nil
}
What I've tried:
Use gocloc as a library, didn't get it to work.
Use single quotes instead of escapes, didn't get it to work, but I think there might be something.
One way to get around this, might be to create separate shell script and pass the dir to that as an argument, and get rid of the escapes there, I don't know ...
If you want to observe all the source code: https://github.com/haapjari/glass and more specificly, it's the files https://github.com/haapjari/glass/blob/main/pkg/plugins/goplg/plugin.go and function enrichWithLibraryData() and utils functions, which are here: https://github.com/haapjari/glass/blob/main/pkg/plugins/goplg/utils.go (the examples above)
Any ideas? How to proceed? Thanks in advance!
I have an input string: github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1.
Parsed to a format: github.com/\!azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1.
Your parser seems to have error. I would expect Azure to become !azure:
github.com/!azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1.
Go Modules Reference
To avoid ambiguity when serving from case-insensitive file systems, the $module and $version elements are case-encoded by replacing every uppercase letter with an exclamation mark followed by the corresponding lower-case letter. This allows modules example.com/M and example.com/m to both be stored on disk, since the former is encoded as example.com/!m.
I have a string mixed with characters and numerals, but i want to increment the last character which happens to be a number, here is what i have, it works, but once i reach 10 rune goes to black since 10 decimal is zero, is there a better way to do this?
package main
import (
"fmt"
)
func main() {
str := "version-1.1.0-8"
rStr := []rune(str)
last := rStr[len(rStr)-1]
rStr[len(rStr)-1] = last + 1
}
So this works for str := "version-1.1.0-8" = version-1.1.0-9
str := version-1.1.0-9 = version-1.1.0-
I understand why it is happening, but I dont know how to fix it
Your intention is to increment the number represented by the last rune, so you should do that: parse out that number, increment it as a number, and "re-encode" it into string.
You can't operate on a single rune, as once the number reaches 10, it can only be represented using 2 runes. Another issue is if the last number is 19, incrementing it needs to alter the previous rune (and not adding a new rune).
Parsing the numbers and re-encoding though is much easier than one might think.
You can take advantage of the fmt package's fmt.Sscanf() and fmt.Sprintf() functions. Parsing and re-encoding is just a single function call.
Let's wrap this functionality into a function:
const format = "version-%d.%d.%d-%d"
func incLast(s string) (string, error) {
var a, b, c, d int
if _, err := fmt.Sscanf(s, format, &a, &b, &c, &d); err != nil {
return "", err
}
d++
return fmt.Sprintf(format, a, b, c, d), nil
}
Testing it:
s := "version-1.1.0-8"
for i := 0; i < 13; i++ {
var err error
if s, err = incLast(s); err != nil {
panic(err)
}
fmt.Println(s)
}
Output (try it on the Go Playground):
version-1.1.0-9
version-1.1.0-10
version-1.1.0-11
version-1.1.0-12
version-1.1.0-13
version-1.1.0-14
version-1.1.0-15
version-1.1.0-16
version-1.1.0-17
version-1.1.0-18
version-1.1.0-19
version-1.1.0-20
version-1.1.0-21
Another option would be to just parse and re-encode the last part, and not the complete version text. This is how it would look like:
func incLast2(s string) (string, error) {
i := strings.LastIndexByte(s, '-')
if i < 0 {
return "", fmt.Errorf("invalid input")
}
d, err := strconv.Atoi(s[i+1:])
if err != nil {
return "", err
}
d++
return s[:i+1] + strconv.Itoa(d), nil
}
Testing and output is the same. Try this one on the Go Playground.
I have a base64 encoded string i'm trying to decrypt with go. The string contains white spaces which should be ignored.
A sample code I'm trying:
s := "eyJ0aHJlZURTU2VydmVyVHJhbnNJRCI6IjEzZmU3MWQ0LWQxMGQtNDIyMC1hMjE2LTIwMDZkMWRkNGNiOCIsImFjc1RyY++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++W5zSUQiOiJkN2M0NWY5OS05NDc4LTQ0YTYtYjFmMi0xMDAwMDAwMDMzNjYiLCJtZXNzYWdlVHlwZSI6IkNSZXEiLCJtZXNzYWdlVmVyc2lvbiI6IjIuMS4wIiwiY2hhbGxlbmdlV2luZG93U2l6ZSI6IjAyIn0%3D"
out, err := base64.URLEncoding.DecodeString(s)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(out))
This code returns:
illegal base64 data at input byte 93
After changing the string padding, and using StdEncoding instead of URLEncoding:
s= strings.Replace(s, "%3D", "=", -1)
out, err := base64.StdEncoding.DecodeString(s)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(out))
The output will be:
{"threeDSServerTransID":"13fe71d4-d10d-4220-a216-2006d1dd4cb8","acsTrc���������������������������������������������������������������������������nsID":"d7c45f99-9478-44a6-b1f2-100000003366","messageType":"CReq","messageVersion":"2.1.0","challengeWindowSize":"02"}
How can I decrypt the string correctly?
What you have is most likely "cut off" from a URL, and it is in URL-encoded form. So to get a Base64 string, you have to first decode it, you may use url.PathUnescape() for this.
Once you have the unescaped string, you may decode it using the base64.StdEncoding encoder. Note that just because it is / was part of a URL, that doesn't make it a base64 string that used the alphabet of the URL-safe version of Base64.
Also the + signs in the middle of it are really just "junk". They shouldn't be there in the first place, so double-check how you get your input, but now that they are there, you have to remove them. For that, you may use strings.Replace().
Final code to decode your invalid input:
s := "eyJ0aHJlZURTU2VydmVyVHJhbnNJRCI6IjEzZmU3MWQ0LWQxMGQtNDIyMC1hMjE2LTIwMDZkMWRkNGNiOCIsImFjc1RyY++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++W5zSUQiOiJkN2M0NWY5OS05NDc4LTQ0YTYtYjFmMi0xMDAwMDAwMDMzNjYiLCJtZXNzYWdlVHlwZSI6IkNSZXEiLCJtZXNzYWdlVmVyc2lvbiI6IjIuMS4wIiwiY2hhbGxlbmdlV2luZG93U2l6ZSI6IjAyIn0%3D"
s = strings.Replace(s, "+", "", -1)
var err error
if s, err = url.PathUnescape(s); err != nil {
panic(err)
}
out, err := base64.StdEncoding.DecodeString(s)
if err != nil {
panic(err)
}
fmt.Println(string(out))
Complete output (try it on the Go Playground):
{"threeDSServerTransID":"13fe71d4-d10d-4220-a216-2006d1dd4cb8",
"acsTransID":"d7c45f99-9478-44a6-b1f2-100000003366","messageType":"CReq",
"messageVersion":"2.1.0","challengeWindowSize":"02"}
Note that the + sign is a valid symbol in the alphabet of the standard Base64, and you can even decode the Base64 without removing the + symbols, but then you get junk data remaining in the JSON keys in the result.
The input string has three problems
First the + signs in the middle of it
Second there is garbage (a url encoded +) at the end
Third the string appears to not be valid Base64
To remove the plus signs in the middle, find the index of the start and finish and make a new string
To remove the garbage at the end, terminate the string earlier ( at index 249 of the fixed string)
There is a further problem with the string at index 148 of the fixed string, which I would guess is due to bad data
But the code fragment below shows how to overcome the first two things
package main
import (
"fmt"
"encoding/base64"
"strings"
)
func main() {
s := "eyJ0aHJlZURTU2VydmVyVHJhbnNJRCI6IjEzZmU3MWQ0LWQxMGQtNDIyMC1hMjE2LTIwMDZkMWRkNGNiOCIsImFjc1RyY++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++W5zSUQiOiJkN2M0NWY5OS05NDc4LTQ0YTYtYjFmMi0xMDAwMDAwMDMzNjYiLCJtZXNzYWdlVHlwZSI6IkNSZXEiLCJtZXNzYWdlVmVyc2lvbiI6IjIuMS4wIiwiY2hhbGxlbmdlV2luZG93U2l6ZSI6IjAyIn0%3D"
a:=strings.Index(s,"+")
b:=strings.LastIndex(s,"+")+1
fixed:=s[0:a] + s[b:249]
out, err := base64.StdEncoding.DecodeString(fixed)
if err != nil {
fmt.Println(err)
fmt.Println(fixed)
}
fmt.Println(a,b)
fmt.Println(String(out))
}
I have an int64 like:
1502712864232
Which is the result of a REST GET to a service. I can convert it to a string easily enough. It is a Unixnano timestamp.
What I really need is to convert it to a string which is related to a timezone like "Europe/London". Such as:-
"14/08/2017, 13:14:24"
Such as produced by this handy utility:
http://www.freeformatter.com/epoch-timestamp-to-date-converter.html
Would very much appreciate any assistance.
==> Update.
Thanks to #evanmcdonnal for such a useful answer. Much appreciated.
Turns out that the data I had was not UnixNano at all (sorry) it was milliseconds from Epoch. Source is a Jenkins timestamp....
So... I wrote the following helper function to get what I need:
// Arg 1 is an int64 representing the millis since Epoch
// Arg 2 is a timezome. Eg: "Europe/London"
// Arg 3 is an int relating to the formatting of the returned string
// Needs the time package. Obviously.
func getFormattedTimeFromEpochMillis(z int64, zone string, style int) string {
var x string
secondsSinceEpoch := z / 1000
unixTime := time.Unix(secondsSinceEpoch, 0)
timeZoneLocation, err := time.LoadLocation(zone)
if err != nil {
fmt.Println("Error loading timezone:", err)
}
timeInZone := unixTime.In(timeZoneLocation)
switch style {
case 1:
timeInZoneStyleOne := timeInZone.Format("Mon Jan 2 15:04:05")
//Mon Aug 14 13:36:02
return timeInZoneStyleOne
case 2:
timeInZoneStyleTwo := timeInZone.Format("02-01-2006 15:04:05")
//14-08-2017 13:36:02
return timeInZoneStyleTwo
case 3:
timeInZoneStyleThree := timeInZone.Format("2006-02-01 15:04:05")
//2017-14-08 13:36:02
return timeInZoneStyleThree
}
return x
}
Rather than converting this to a string, you'll want to convert it to a time.Time and from there to a string. You can use the handy Unix method to get a Time object back for that time stamp.
import "time"
import "fmt"
t := time.Unix(0, 1502712864232)
fmt.Println(t.Format("02/01/2006, 15:04:05"))
Edit: added format to the println - note, testing your unix stamp in go playground, that value is neither nano seconds nor seconds, in both cases the time value produced is way off of what it should be. The code above still demonstrates the basic idea of what you want to do but it seems an additional step is necessary or the sample int64 you gave just does not correspond to the string you provided.
relevant docs:
https://golang.org/pkg/time/#Unix
https://golang.org/pkg/fmt/
Like [a-zA-Z0-9] string:
na1dopW129T0anN28udaZ
or hexadecimal string:
8c6f78ac23b4a7b8c0182d
By long I mean 2K and more characters.
This does about 200MBps on my box. There's obvious room for improvement.
type randomDataMaker struct {
src rand.Source
}
func (r *randomDataMaker) Read(p []byte) (n int, err error) {
for i := range p {
p[i] = byte(r.src.Int63() & 0xff)
}
return len(p), nil
}
You'd just use io.CopyN to produce the string you want. Obviously you could adjust the character set on the way in or whatever.
The nice thing about this model is that it's just an io.Reader so you can use it making anything.
Test is below:
func BenchmarkRandomDataMaker(b *testing.B) {
randomSrc := randomDataMaker{rand.NewSource(1028890720402726901)}
for i := 0; i < b.N; i++ {
b.SetBytes(int64(i))
_, err := io.CopyN(ioutil.Discard, &randomSrc, int64(i))
if err != nil {
b.Fatalf("Error copying at %v: %v", i, err)
}
}
}
On one core of my 2.2GHz i7:
BenchmarkRandomDataMaker 50000 246512 ns/op 202.83 MB/s
EDIT
Since I wrote the benchmark, I figured I'd do the obvious improvement thing (call out to the random less frequently). With 1/8 the calls to rand, it runs about 4x faster, though it's a big uglier:
New version:
func (r *randomDataMaker) Read(p []byte) (n int, err error) {
todo := len(p)
offset := 0
for {
val := int64(r.src.Int63())
for i := 0; i < 8; i++ {
p[offset] = byte(val & 0xff)
todo--
if todo == 0 {
return len(p), nil
}
offset++
val >>= 8
}
}
panic("unreachable")
}
New benchmark:
BenchmarkRandomDataMaker 200000 251148 ns/op 796.34 MB/s
EDIT 2
Took out the masking in the cast to byte since it was redundant. Got a good deal faster:
BenchmarkRandomDataMaker 200000 231843 ns/op 862.64 MB/s
(this is so much easier than real work sigh)
EDIT 3
This came up in irc today, so I released a library. Also, my actual benchmark tool, while useful for relative speed, isn't sufficiently accurate in its reporting.
I created randbo that you can reuse to produce random streams wherever you may need them.
You can use the Go package uniuri to generate random strings (or view the source code to see how they're doing it). You'll want to use:
func NewLen(length int) string
NewLen returns a new random string of the provided length, consisting of standard characters.
Or, to specify the set of characters used:
func NewLenChars(length int, chars []byte) string
This is actually a little biased towards the first 8 characters in the set (since 255 is not a multiple of len(alphanum)), but this will get you most of the way there.
import (
"crypto/rand"
)
func randString(n int) string {
const alphanum = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
var bytes = make([]byte, n)
rand.Read(bytes)
for i, b := range bytes {
bytes[i] = alphanum[b % byte(len(alphanum))]
}
return string(bytes)
}
If you want to generate cryptographically secure random string, I recommend you to take a look at this page. Here is a helper function that reads n random bytes from the source of randomness of your OS and then use these bytes to base64encode it. Note that the string length would be bigger than n because of base64.
package main
import(
"crypto/rand"
"encoding/base64"
"fmt"
)
func GenerateRandomBytes(n int) ([]byte, error) {
b := make([]byte, n)
_, err := rand.Read(b)
if err != nil {
return nil, err
}
return b, nil
}
func GenerateRandomString(s int) (string, error) {
b, err := GenerateRandomBytes(s)
return base64.URLEncoding.EncodeToString(b), err
}
func main() {
token, _ := GenerateRandomString(32)
fmt.Println(token)
}
Here Evan Shaw's answer re-worked without the bias towards the first 8 characters of the string. Note that it uses lots of expensive big.Int operations so probably isn't that quick! The answer is crypto strong though.
It uses rand.Int to make an integer of exactly the right size len(alphanum) ** n, then does what is effectively a base conversion into base len(alphanum).
There is almost certainly a better algorithm for this which would involve keeping a much smaller remainder and adding random bytes to it as necessary. This would get rid of the expensive long integer arithmetic.
import (
"crypto/rand"
"fmt"
"math/big"
)
func randString(n int) string {
const alphanum = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
symbols := big.NewInt(int64(len(alphanum)))
states := big.NewInt(0)
states.Exp(symbols, big.NewInt(int64(n)), nil)
r, err := rand.Int(rand.Reader, states)
if err != nil {
panic(err)
}
var bytes = make([]byte, n)
r2 := big.NewInt(0)
symbol := big.NewInt(0)
for i := range bytes {
r2.DivMod(r, symbols, symbol)
r, r2 = r2, r
bytes[i] = alphanum[symbol.Int64()]
}
return string(bytes)
}