we use the following lib
import "crypto/sha1"
while running golangci-lint we got the following errors :
G505: Blocklisted import crypto/sha1: weak cryptographic primitive (gosec) for "crypto/sha1"
G401: Use of weak cryptographic primitive (gosec)
sha := sha1.New()
Is there is something that I can do without excluding them? not sure that I understand those issues. if it was not related to security it's simple tasks to exclude ...
update
what we are doing is
fdrContent, err := ioutil.ReadFile(filepath.Join(path))
// gets the hashcode of the FDR file
h := sha1.New()
code, err := h.Write(fdrContent)
return code, err
I use h.Write in my own gtarsum project as in here:
h := sha256.New()
for {
buf := make([]byte, 1024*1024)
bytesRead, err := tr.Read(buf)
if err != nil {
if err != io.EOF {
panic(err)
}
}
if bytesRead > 0 {
_, err := h.Write(buf[:bytesRead])
All you have to do, if there is no obvious performance issue, is to switch to sha256.
No more warning.
The issue comes sha1 collision, that I have documented here, from the shattered.io project.
Related
I'm testing a function I have that gets all azure vms under a specific subscription.
It uses azure sdk for go's compute.VirtualMachinesClient to do so.
My problem is with the vm clients ListAllComplete function.
It's not returning an error. The code just doesn't seem to be able to make it passed that line.
Any suggestions on the source of the problem would be appreciated.
This is the code, I've used the fmt package to follow how far it gets:
func GetAllAzureVms() ([]compute.VirtualMachine, error) {
fmt.Printf("In getAllAzureVm\n")
var vmList []compute.VirtualMachine
vmClient, err := GetAzureVmClient()
fmt.Printf("Out of GetAzureVmClient\n")
if err != nil {
return nil, err
}
fmt.Print("No error from getazurevmclient\n")
vmListComplete, err := vmClient.ListAllComplete(context.Background(), "statusOnly=false")
fmt.Print("vmClient.ListAllComplete done")
if err != nil {
fmt.Print("vmClient.ListAllComplete error")
return nil, err
}
fmt.Print("here")
for vmListComplete.NotDone() {
vmList = append(vmList, vmListComplete.Value())
err := vmListComplete.NextWithContext(context.Background())
if err != nil {
return nil, err
}
}
fmt.Print("here2")
return vmList, nil
}
It cant make it passed the line:
vmListComplete, err := vmClient.ListAllComplete(context.Background(), "statusOnly=false")
No error is returned.
I have a similar piece of code, not for vmClient unluckily, but for securityCustomRules.
What I've found useful was to use ListComplete() instead of ListAll() and the print values through JSON marhsalling.
I hope you can find it useful anyway.
import (
"context"
"fmt"
"os"
"testing"
"github.com/gruntwork-io/terratest/modules/azure"
"github.com/gruntwork-io/terratest/modules/terraform"
)
securityCustomRulesList, err := securityCustomRulesClient.ListComplete(context.Background(), tfResourceGroupName, tfVnetName+"-"+fmt.Sprint(vnetIndex)+"-subnet-02-nsg")
// - Scan iterator items [https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/services/network/mgmt/2019-09-01/network#ApplicationGatewayListResultIterator]
if err != nil {
fmt.Fprintf(os.Stderr, ">>>> Error parsing securityCustomRulesClient:: %s", err)
return
}
for securityCustomRulesList.NotDone() {
// securityCustomRulesList.Value() -> securityRule [https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/services/network/mgmt/2019-09-01/network#SecurityRule]
v := securityCustomRulesList.Value()
vJson, _ := v.MarshalJSON()
fmt.Printf(">> Network securityCustomRulesList JSON %s \n", string(vJson))
//fmt.Printf(">> Network securityCustomRulesList %s - %s\n", *v.Name, *v.SecurityRulePropertiesFormat.Description)
securityCustomRulesList.NextWithContext(context.Background())
}
Which gives me an output like:
>> Creating security custom rules list instance 'securityCustomRulesList'..
>> Network securityCustomRulesList JSON {"id":"/subscriptions/8f7d6be2/resourceGroups/unit-tests-tfm-azure-network-resource-group/providers/Microsoft.Network/networkSecurityGroups/unit-tests-tfm-azure-network-vnet-0-subnet-02-nsg/securityRules/test-02","name":"test-02","properties":{"access":"Deny","description":"Deny access","destinationAddressPrefix":"10.0.2.1","destinationAddressPrefixes":[],"destinationPortRange":"*","destinationPortRanges":[],"direction":"Inbound","priority":111,"protocol":"*","sourceAddressPrefix":"10.0.1.0/24","sourceAddressPrefixes":[],"sourcePortRange":"*","sourcePortRanges":[]}}
Useful links:
Here you can find a complete usage example from the official terratest/azure module: https://github.com/gruntwork-io/terratest/blob/dae956eb39e91dfb00f3ba85060a6dbf58c6782b/modules/azure/nsg.go
https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/services/network/mgmt/2019-09-01/network#SecurityRule.MarshalJSON
I met a Garbled string question when do Get request in Go, the code is:
req , err:= http.NewRequest(httpMethod, url,strings.NewReader(""))
req.Header.Add("Accept","application/json")
resp, err := http.DefaultClient.Do(req)
body,err := ioutil.ReadAll(resp.Body)
ret := string(body)
log.Warningf("ret: %+v", ret)
if the ret contains only english, it's correct, if contains Chinese, it has garbled string, how to solve this problem, thanks all!
Go strings can hold any type of characters, but when printing them the chars are interpreted as utf-8.
You can try adding:
req.Header.Add("Accept-Charset","utf-8")
If that does not work, you can try using this package to convert from whatever charset it is to utf-8:
https://godoc.org/golang.org/x/text/encoding
The charset depends on the page you are requesting. If it is html, the charset is sometimes specified like this in the response headers:
Content-Type: text/html; charset=utf-8
So you need to figure out what the charset is.
In my case website did not respond with the charset within a Content-Type and did not replied with requested: req.Header.Add("Accept-Charset", "utf-8")
I opended up file in Visual Studio Code and have been switching encoding to find out which one works best "Reopen With Encoding".
Once i figured out which encoding it was i simply used function:
dec := charmap.Windows1250.NewDecoder()
output, _ := dec.Bytes(body)
from: "golang.org/x/text/encoding/charmap"
Full code example:
package main
import (
"fmt"
"net/http"
"io/ioutil"
"golang.org/x/text/encoding/charmap"
)
func main() {
client := &http.Client{}
req, err := http.NewRequest("GET", "example.com", nil)
if err != nil {
fmt.Println(err)
return
}
resp, err := client.Do(req)
if err != nil {
fmt.Println(err)
return
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
return
}
dec := charmap.Windows1250.NewDecoder()
output, err:= dec.Bytes(body)
if err != nil {
fmt.Println(err)
return
}
// do something with output
}
I am writing a media cross-platform distributed media player for use on my own network.
The current version has three/four parts:
A NAS holding the audio files.
A metadata server holding information about the files.
A HTML/JS client that allows manipulation of the metadata server and queuing media for the:
A player deamon.
My problem lies with part 4. The player has no UI, nor does it need one. It will be controlled via network commands from the client and by listening to the media keys on its current host.
The player daemon needs to work on both Windows and Linux, but I can't seem to figure out a way (any way) to read these keys on either OS. Most of the way I know to read the keyboard will not read these keys at all.
With the help of several commenters, I now have it all figured out.
The Linux version is as follows:
package main
import (
“bytes”
“encoding/binary”
“fmt”
“os”
“os/exec”
“syscall”
)
// parses through the /proc/bus/input/devices file for keyboard devices.
// Copied from `github.com/gearmover/keylogger` with trivial modification.
func dumpDevices() ([]string, error) {
cmd := exec.Command(“/bin/sh”, “-c”, “/bin/grep -E ‘Handlers|EV=’ /proc/bus/input/devices | /bin/grep -B1 ‘EV=120013’ | /bin/grep -Eo ‘event[0-9]+’”)
output, err := cmd.Output()
if err != nil {
return nil, err
}
buf := bytes.NewBuffer(output)
var devices []string
for line, err := buf.ReadString(‘\n’); err == nil; {
devices = append(devices, “/dev/input/”+line[:len(line)-1])
line, err = buf.ReadString(‘\n’)
}
return devices, nil
}
// Using MS names, just because I don’t feel like looking up the Linux versions.
var keys = map[uint16]string{
0xa3: “VK_MEDIA_NEXT_TRACK”,
0xa5: “VK_MEDIA_PREV_TRACK”,
0xa6: “VK_MEDIA_STOP”,
0xa4: “VK_MEDIA_PLAY_PAUSE”,
}
// Most of the code here comes from `github.com/gearmover/keylogger`.
func main() {
// drop privileges when executing other programs
syscall.Setgid(65534)
syscall.Setuid(65534)
// dump our keyboard devices from /proc/bus/input/devices
devices, err := dumpDevices()
if err != nil {
fmt.Println(err)
}
if len(devices) == 0 {
fmt.Println(“No input devices found”)
return
}
// bring back our root privs
syscall.Setgid(0)
syscall.Setuid(0)
// Open the first keyboard device.
input, err := os.OpenFile(devices[0], os.O_RDONLY, 0600)
if err != nil {
fmt.Println(err)
return
}
defer input.Close()
// Log media keys
var buffer = make([]byte, 24)
for {
// read the input events as they come in
n, err := input.Read(buffer)
if err != nil {
return
}
if n != 24 {
fmt.Println(“Weird Input Event Size: “, n)
continue
}
// parse the input event according to the <linux/input.h> header struct
binary.LittleEndian.Uint64(buffer[0:8]) // Time stamp stuff I could care less about
binary.LittleEndian.Uint64(buffer[8:16])
etype := binary.LittleEndian.Uint16(buffer[16:18]) // Event Type. Always 1 for keyboard events
code := binary.LittleEndian.Uint16(buffer[18:20]) // Key scan code
value := int32(binary.LittleEndian.Uint32(buffer[20:24])) // press(1), release(0), or repeat(2)
if etype == 1 && value == 1 && keys[code] != “” {
// In a real application I would send a message here.
fmt.Println(keys[code])
}
}
}
And the Windows version:
package main
import (
“fmt”
“syscall”
“time”
)
var user32 = syscall.NewLazyDLL(“user32.dll”)
var procGAKS = user32.NewProc(“GetAsyncKeyState”)
// Key codes from MSDN
var keys = [4]uint{
0xb0, // VK_MEDIA_NEXT_TRACK
0xb1, // VK_MEDIA_PREV_TRACK
0xb2, // VK_MEDIA_STOP
0xb3, // VK_MEDIA_PLAY_PAUSE
}
var names = [4]string{
“VK_MEDIA_NEXT_TRACK”,
“VK_MEDIA_PREV_TRACK”,
“VK_MEDIA_STOP”,
“VK_MEDIA_PLAY_PAUSE”,
}
func main() {
fmt.Println(“Running…”)
// Since I don’t want to trigger dozens of times for each key I need to track state.
// I could check the bits of GAKS’ return value, but that is not reliable.
down := [4]bool{false, false, false, false}
for {
time.Sleep(1 * time.Millisecond)
for i, key := range keys {
// val is not a simple boolean!
// 0 means “not pressed” (also certain errors)
// If LSB is set the key was just pressed (this may not be reliable)
// If MSB is set the key is currently down.
val, _, _ := procGAKS.Call(uintptr(key))
// Turn a press into a transition and track key state.
goingdown := false
if int(val) != 0 && !down[i] {
goingdown = true
down[i] = true
}
if int(val) == 0 && down[i] {
down[i] = false
}
if goingdown {
// In a real application I would send a message here.
fmt.Println(names[i])
}
}
}
}
The only "issue" is that the Linux version must be run as root. For me this is not a problem. If running as root is a problem I think there is a way that involves X11...
In the process of learning golang, I'm trying to write a web app with multiple image upload functionality.
I'm using Azure Blob Storage to store images, but I am having trouble streaming the images from the multipart request to Blob Storage.
Here's the handler I've written so far:
func (imgc *ImageController) UploadInstanceImageHandler(w http.ResponseWriter, r *http.Request, p httprouter.Params) {
reader, err := r.MultipartReader()
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
for {
part, partErr := reader.NextPart()
// No more parts to process
if partErr == io.EOF {
break
}
// if part.FileName() is empty, skip this iteration.
if part.FileName() == "" {
continue
}
// Check file type
if part.Header["Content-Type"][0] != "image/jpeg" {
fmt.Printf("\nNot image/jpeg!")
break
}
var read uint64
fileName := uuid.NewV4().String() + ".jpg"
buffer := make([]byte, 100000000)
// Get Size
for {
cBytes, err := part.Read(buffer)
if err == io.EOF {
fmt.Printf("\nLast buffer read!")
break
}
read = read + uint64(cBytes)
}
stream := bytes.NewReader(buffer[0:read])
err = imgc.blobClient.CreateBlockBlobFromReader(imgc.imageContainer, fileName, read, stream, nil)
if err != nil {
fmt.Println(err)
break
}
}
w.WriteHeader(http.StatusOK)
}
In the process of my research, I've read through using r.FormFile, ParseMultipartForm, but decided on trying to learn how to use MultiPartReader.
I was able to upload an image to the golang backend and save the file to my machine using MultiPartReader.
At the moment, I'm able to upload files to Azure but they end up being corrupted. The file sizes seem on point but clearly something is not working.
Am I misunderstanding how to create a io.Reader for CreateBlockBlobFromReader?
Any help is much appreciated!
As #Mark said, you can use ioutil.ReadAll to read the content into a byte array, the code like below.
import (
"bytes"
"io/ioutil"
)
partBytes, _ := ioutil.ReadAll(part)
size := uint64(len(partBytes))
blob := bytes.NewReader(partBytes)
err := blobClient.CreateBlockBlobFromReader(container, fileName, size, blob, nil)
According to the godoc for CreateBlockBlobFromReader, as below.
The API rejects requests with size > 64 MiB (but this limit is not checked by the SDK). To write a larger blob, use CreateBlockBlob, PutBlock, and PutBlockList.
So if the size is larger than 64MB, the code shoule be like below.
import "encoding/base64"
const BLOB_LENGTH_LIMITS uint64 = 64 * 1024 * 1024
partBytes, _ := ioutil.ReadAll(part)
size := uint64(len(partBytes))
if size <= BLOB_LENGTH_LIMITS {
blob := bytes.NewReader(partBytes)
err := blobClient.CreateBlockBlobFromReader(container, fileName, size, blob, nil)
} else {
// Create an empty blob
blobClient.CreateBlockBlob(container, fileName)
// Create a block list, and upload each block
length := size / BLOB_LENGTH_LIMITS
if length%limits != 0 {
length = length + 1
}
blocks := make([]Block, length)
for i := uint64(0); i < length; i++ {
start := i * BLOB_LENGTH_LIMITS
end := (i+1) * BLOB_LENGTH_LIMITS
if end > size {
end = size
}
chunk := partBytes[start: end]
blockId := base64.StdEncoding.EncodeToString(chunk)
block := Block{blockId, storage.BlockStatusCommitted}
blocks[i] = block
err = blobClient.PutBlock(container, fileName, blockID, chunk)
if err != nil {
.......
}
}
err = blobClient.PutBlockList(container, fileName, blocks)
if err != nil {
.......
}
}
Hope it helps.
A Reader can return both an io.EOF and a valid final bytes read, it looks like the final bytes (cBytes) is not added to read total bytes. Also, careful: if an error is returned by part.Read(buffer) other than io.EOF, the read loop might not exit. Consider ioutil.ReadAll instead.
CreateBlockBlobFromReader takes a Reader, and part is a Reader, so you may be able to pass the part in directly.
You may also want to consider Azure block size limits might be smaller than the image, see Asure blobs.
Shoutcast servers basically speak http, with one important difference: they respond to GET requests with ICY 200 OK instead of HTTP/1.1 200 OK.
Go won't have a bar of it, and correctly fails with the error malformed HTTP version "ICY".
However I would like to make things work and am wondering what the best approach is. My ideas so far:
use a custom http.Transport.Proxy to change ICY to HTTP/1.1 in-flight
an out of process proxy that does the same thing
overload http.ParseHTTPVersion (but golang doesn't have function overloading)
duplicate the entire http package, just to modify ParseHTTPVersion
Number 1. seems the most attractive attractive, but I have no idea how to respect the http "scope" and actually modify all responses on a given http version. Is this the kind of thing http.Transport.Proxy can handle?
Can anyone give me any pointers?
I got this working by creating a custom Dial function that returns a wrapped connection. My wrapper intercepts the first read on the connection and replaces ICY with HTTP/1.1. Not super robust, but proves the concept:
package main
import (
"fmt"
"net"
"net/http"
)
type IcyConnWrapper struct {
net.Conn
haveReadAny bool
}
func (i *IcyConnWrapper) Read(b []byte) (int, error) {
if i.haveReadAny {
return i.Conn.Read(b)
}
i.haveReadAny = true
//bounds checking ommitted. There are a few ways this can go wrong.
//always check array sizes and returned n.
n, err := i.Conn.Read(b[:3])
if err != nil {
return n, err
}
if string(b[:3]) == "ICY" {
//write Correct http response into buffer
copy(b, []byte("HTTP/1.1"))
return 8, nil
}
return n, nil
}
func main() {
tr := &http.Transport{
Dial: func(network, a string) (net.Conn, error) {
realConn, err := net.Dial(network, a)
if err != nil {
return nil, err
}
return &IcyConnWrapper{Conn: realConn}, nil
},
}
client := &http.Client{Transport: tr}
http.DefaultClient = client
resp, err := http.Get("http://178.33.230.189:8100") //random url I found on the internet
fmt.Println(err)
fmt.Println(resp.StatusCode)
}