How to fetch blob from azure - azure

I am trying to download some data from Azure Blob Storage in Go via the official azure-sdk-for-go.
To setup my development environment I have successfully logged in via az login. I have verified that the blob can be accessed via CLI:
az storage blob download --container-name [container-name] --name [blob-name] --account-name [storage-account-name] -f out.txt
This works as expected. To fetch the file unsing go I am using the following snippet (as a reproducer):
func getBlob(account, container, object string) ([]byte, error) {
blobPath := fmt.Sprintf("https://%s.blob.core.windows.net/%s/%s", uri.Host, container, object)
ctx := context.Background()
credential, err := azidentity.NewDefaultAzureCredential(nil)
if err != nil {
return []byte{}, err
}
blobClient, err := azblob.NewBlockBlobClient(blobPath, credential, nil)
if err != nil {
return []byte{}, err
}
get, err := blobClient.Download(ctx, nil)
if err != nil {
return []byte{}, err
}
downloadedData := &bytes.Buffer{}
reader := get.Body(&azblob.RetryReaderOptions{})
_, err = downloadedData.ReadFrom(reader)
if err != nil {
return []byte{}, err
}
err = reader.Close()
if err != nil {
return []byte{}, err
}
data = downloadedData.Bytes()
return data, nil
}
Being logged in via az login I would expect azidentity.NewDefaultAzureCredential(nil) to use this session/cerdentials (see https://learn.microsoft.com/en-us/azure/developer/go/azure-sdk-authentication?tabs=bash#-option-3-sign-in-with-azure-cli), however that appears not no work as expected. The error I get is the following:
===== RESPONSE ERROR (ErrorCode=AuthorizationPermissionMismatch) =====
Description=This request is not authorized to perform this operation using this permission.
RequestId:b078ec61-xxxx-xxxx-xxxx-604682000000
Time:2022-05-05T10:24:18.8093649Z, Details: (none)
exit status 255
What am I missing?
(I am coming from a AWS background so chances are I am making assumptions on how things should work based on that experience.)

Apparently interacting with blobs does not work with credentials provided by azidentity.NewDefaultAzureCredential(). Azure requires either SAS Tokens or Shared Keys in order to work with blobs. Here's an example function that can be used to get a client for specific blob:
func getBlobClient(account, container, object string) (*azblob.BlockBlobClient, error) {
accountKey, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_KEY")
if !ok {
return nil, errors.New("AZURE_STORAGE_ACCOUNT_KEY could not be found")
}
credential, err := azblob.NewSharedKeyCredential(account, accountKey)
if err != nil {
return nil, err
}
accountPath := fmt.Sprintf("https://%s.blob.core.windows.net/", account)
serviceClient, err := azblob.NewServiceClientWithSharedKey(accountPath, credential, nil)
if err != nil {
return nil, err
}
containerClient, err := serviceClient.NewContainerClient(container)
if err != nil {
return nil, err
}
blobClient, err := containerClient.NewBlockBlobClient(object)
if err != nil {
return nil, err
}
return blobClient, nil
}
This uses the AZURE_STORAGE_ACCOUNT_KEY environment variable for the credentials.
The examples that can be found are rather confusing (and possibly wrong), an issue is opened here:
https://github.com/Azure-Samples/storage-blobs-go-quickstart/issues/7

Related

List all Azure container registry images via Golang

I would like to list all images from Azure registry via Golang.
What I founded is this: https://github.com/Azure-Samples/azure-sdk-for-go-samples/tree/main/sdk/resourcemanager/containerregistry but nothing from there seems to help me.
Any ideas please?
LE:
I ended up with this code
package main
import (
"context"
"encoding/json"
"fmt"
"log"
"net/http"
"net/url"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/policy"
"github.com/Azure/azure-sdk-for-go/sdk/azidentity"
"github.com/Azure/azure-sdk-for-go/services/preview/containerregistry/runtime/2019-08-15-preview/containerregistry"
"github.com/Azure/go-autorest/autorest"
)
type ACRTokenProvider struct {
accessToken string
}
func (a *ACRTokenProvider) OAuthToken() string {
return a.accessToken
}
func newACRAuthorizer() (*autorest.BearerAuthorizer, error) {
tenantId := "TENANT_ID"
acrService := "servicename.azurecr.io"
cred, err := azidentity.NewDefaultAzureCredential(nil)
if err != nil {
panic(err)
}
ctx := context.Background()
aadToken, err := cred.GetToken(ctx, policy.TokenRequestOptions{Scopes: []string{"https://management.azure.com/"}})
if err != nil {
panic(err)
}
formData := url.Values{
"grant_type": {"access_token"},
"service": {acrService},
"tenant": {tenantId},
"access_token": {aadToken.Token},
}
jsonResponse, err := http.PostForm(fmt.Sprintf("https://%s/oauth2/exchange", acrService), formData)
if err != nil {
panic(err)
}
var response map[string]interface{}
json.NewDecoder(jsonResponse.Body).Decode(&response)
return autorest.NewBearerAuthorizer(&ACRTokenProvider{accessToken: fmt.Sprint(response["refresh_token"])}), nil
}
func main() {
client := containerregistry.NewRepositoryClient("https://servicename.azurecr.io")
authorizer, err := newACRAuthorizer()
if err != nil {
fmt.Println(err)
}
client.Authorizer = authorizer
// Do what you need to do with client here
attributes, err := client.GetList(context.Background(), "registryName", nil)
if err != nil {
log.Printf("Error while fetching attributes, %v ", err)
}
fmt.Print(attributes)
}
But the response is this
Original Error: autorest/azure: Service returned an error. Status=401 Code="Unknown" Message="Unknown service error" Details=[{"errors":[{"code":"UNAUTHORIZED","detail":[{"Action":"*","Name":"catalog","Type":"registry"}],"message":"authentication required, visit https://aka.ms/acr/authorization for more information."}]}]
What I'm missing?
i'm not sure about the GO SDK, but you can always consume the REST API directly:
GET https://management.azure.com/subscriptions/{subscriptionId}/providers/Microsoft.ContainerRegistry/registries?api-version=2019-05-01
https://learn.microsoft.com/en-us/rest/api/containerregistry/registries/list?tabs=HTTP

Monitoring an existing file with fsnotify

I'm trying to monitor a file using the fsnotify packet in golang.
I saw few examples like this and I would like to know if this is the best way of using fsnotify:
package main
import (
"log"
"github.com/howeyc/fsnotify"
)
func main() {
watcher, err := fsnotify.NewWatcher()
if err != nil {
log.Fatal(err)
}
done := make(chan bool)
// Process events
go func() {
for {
select {
case ev := <-watcher.Event:
log.Println("event:", ev)
case err := <-watcher.Error:
log.Println("error:", err)
}
}
}()
err = watcher.Watch("testDir")
if err != nil {
log.Fatal(err)
}
<-done
var get_info := []string
get_info = read_file(path_to_file)
watcher.Close()
}
Basically I'm passing a path where the file is located and geting the resul in a string variable.
Everytime I change the file I would like to read the file and get the result.
I'm not sure if I'm using fsnotify correctly base on that example. Also, I'm not sure where to put the file path in the fsnotify to monitor that file.
You're leveraging fsnotify pretty much correctly, the only change would likely be that you want to utilize the channel to grab events and then use the event to extract the file name that changed. This would allow you to monitor multiple files and also in your example I don't believe you ever pass a value into done for it to properly finish waiting on the channel and read the file contents.
I'm adding a simple sample below that gets rid of the go routine and simply listens for changes on the main thread.
func main() {
watcher, err := fsnotify.NewWatcher()
if err != nil {
panic(err)
}
err = watcher.Add("file.txt")
if err != nil {
panic(err)
}
for {
select {
case ev := <-watcher.Events:
log.Println("event:", ev)
if ev.Op&fsnotify.Write == fsnotify.Write {
contents, err := ioutil.ReadFile(ev.Name)
if err != nil {
// handle error
}
log.Println("modified file:", string(contents))
}
case err := <-watcher.Errors:
log.Println("error:", err)
}
}
}

Hyperledger fabric: Querying implicit data collection

I am seeing this when querying data from implicit private data collection.
Please see code snippet below.
When I query individual key (using QueryBidPrivate/GetPrivateData), I get corresponding data.
But if I query the complete collection (using GetPrivateDataByRange(collection, "", "")), I get nothing from the Iterator.
peer chaincode query -C mychannel -n govtcontract -c '{"function":"QueryBidPrivate","Args":["100", "1035"]}'
{"bidamt":100,"biddate":"2022-05-04","contractid":"1035","salt":"4567ab4567","vendorid":"100"}
peer chaincode query -C mychannel -n govtcontract -c '{"function":"ListAllBids","Args":[]}'
No output
Is there anything I am missing here ?
// ListAllBids returns all Bids details from private state
func (s *SmartContract) ListAllBids(ctx contractapi.TransactionContextInterface) ([]VendorBid, error) {
// Get client org id and verify it matches peer org id.
// In this scenario, client is only authorized to read/write private data from its own peer.
clientOrgID, err := getClientOrgID(ctx, true)
if err != nil {
return nil, fmt.Errorf("failed to get verified OrgID: %s", err.Error())
}
collection := "_implicit_org_" + clientOrgID
BidIterator, err := ctx.GetStub().GetPrivateDataByRange(collection, "", "")
if err != nil {
logger.Infof("ListAllBids error: %s", err.Error())
return nil, fmt.Errorf("failed to read bid list : %s", err.Error())
}
if BidIterator == nil {
logger.Infof("ListAllBids : null iterator ")
return nil, fmt.Errorf("bid private details does not exist ")
}
defer BidIterator.Close()
logger.Infof("ListAllBids in govtcontract: no error")
var allbids []VendorBid
myMSPID, err := ctx.GetClientIdentity().GetMSPID()
logger.Infof("myMSPID: %s", myMSPID)
for BidIterator.HasNext() {
logger.Infof("Iterator has element: ")
entrybid, err := BidIterator.Next()
if err != nil {
return nil, err
}
var bidvar VendorBid
err = json.Unmarshal(entrybid.Value, &bidvar)
if err != nil {
return nil, err
}
allbids = append(allbids, bidvar)
logger.Infof("Iterator element: %s", entrybid.Value)
}
return allbids, nil
}
=========================================
// QueryBidPrivate returns the Bid details from owner's private data collection
func (s *SmartContract) QueryBidPrivate(ctx contractapi.TransactionContextInterface, vendorId string, contractId string) (string, error) {
// Get client org id and verify it matches peer org id.
// In this scenario, client is only authorized to read/write private data from its own peer.
clientOrgID, err := getClientOrgID(ctx, true)
if err != nil {
return "", fmt.Errorf("failed to get verified OrgID: %s", err.Error())
}
collection := "_implicit_org_" + clientOrgID
bidconkey, err := ctx.GetStub().CreateCompositeKey(vendorId, []string{contractId})
bidDetails, err := ctx.GetStub().GetPrivateData(collection, bidconkey)
if err != nil {
return "", fmt.Errorf("failed to read bid private properties from client org's collection: %s", err.Error())
}
if bidDetails == nil {
return "", fmt.Errorf("bid private details does not exist in client org's collection: %s", contractId)
}
return string(bidDetails), nil
}
GetPrivateDataByPartialCompositeKey() is the function for querying a range of private data that was stored using a composite key. GetPrivateDataByRange() won't retrieve the data stored with a composite key. I think in the above code snippet you have to replace the function call GetPrivateDataByRange(collection, "", "") with GetPrivateDataByPartialCompositeKey(collection, vendorId, []string{})
Sample usage can be found here.
I faced the same error in the smart contract. The issue here is because of storing data on the composite key.
Instead of below code :
for BidIterator.HasNext() {
logger.Infof("Iterator has element: ")
entrybid, err := BidIterator.Next()
if err != nil {
return nil, err
}
var bidvar VendorBid
err = json.Unmarshal(entrybid.Value, &bidvar)
if err != nil {
return nil, err
}
allbids = append(allbids, bidvar)
logger.Infof("Iterator element: %s", entrybid.Value)
}
Use the below function
func constructQueryResponseFromIterator(resultsIterator shim.StateQueryIteratorInterface)
(*bytes.Buffer, error)
{
// buffer is a JSON array containing QueryResults
var buffer bytes.Buffer
buffer.WriteString("[")
bArrayMemberAlreadyWritten := false
for resultsIterator.HasNext() {
queryResponse, err := resultsIterator.Next()
if err != nil {
return nil, err
}
// Add a comma before array members, suppress it for the first array member
if bArrayMemberAlreadyWritten == true {
buffer.WriteString(",")
}
buffer.WriteString("{")
//buffer.WriteString("{\"Key\":")
//buffer.WriteString("\"")
//buffer.WriteString(queryResponse.Key)
//buffer.WriteString("\"")
buffer.WriteString(", \"Record\":")
// Record is a JSON object, so we write as-is
buffer.WriteString(string(queryResponse.Value))
buffer.WriteString("}")
bArrayMemberAlreadyWritten = true
}
buffer.WriteString("]")
return &buffer, nil
}

Hyperledger Fabric Chaincode throws MVCC_READ_CONFLICT

I'm getting an error when I invoke a chaincode function. I've created two adaptations for the function. One uses a regular key, the other a composite key. I thought that using a composite key would solve any MVCC_READ_CONFLICT's since I'm no longer updating the same key.
However I get the error on both functions. Note that both function are contained in the same chaincode. I don't know if that can cause conflicts.
Here's the function with a regular key:
func (*AddTokenCallFunction) Start(stub shim.ChaincodeStubInterface, args []string) pb.Response {
if len(args) != 2 {
s := fmt.Sprintf(ERROR_INCORRECT_AMOUNT_OF_ARGUMENTS, "add-tokens", 2, len(args))
return shim.Error(s)
}
account := args[0]
tokens := args[1]
currentTokensBytes, err := stub.GetState(account)
if err != nil {
s := fmt.Sprintf(ERROR_SYSTEM, err.Error())
return shim.Error(s)
}
currentAmountOfTokens := binary.LittleEndian.Uint64(currentTokensBytes)
tokensToAdd, err := strconv.ParseUint(tokens, 10, 64)
if err != nil {
s := fmt.Sprintf(ERROR_SYSTEM, err.Error())
return shim.Error(s)
}
currentAmountOfTokens += tokensToAdd
tokenBytes, err := UintToBytes(currentAmountOfTokens)
if err != nil {
s := fmt.Sprintf(ERROR_SYSTEM, err.Error())
return shim.Error(s)
}
err = stub.PutState(account, tokenBytes)
if err != nil {
s := fmt.Sprintf(ERROR_SYSTEM, err.Error())
return shim.Error(s)
}
return shim.Success(nil)
}
Here's the same function but with a composite-key:
func (*AddTokenCompositeCallFunction) Start(stub shim.ChaincodeStubInterface, args []string) pb.Response {
if len(args) != 2 {
s := fmt.Sprintf(ERROR_INCORRECT_AMOUNT_OF_ARGUMENTS, "add-composite-tokens", 2, len(args))
return shim.Error(s)
}
account := args[0]
tokens := args[1]
// Retrieve info needed for the update procedure
txid := stub.GetTxID()
compositeIndexaccount := "account~tokens~txID"
// Create the composite key that will allow us to query for all deltas on a particular variable
compositeKey, compositeErr := stub.CreateCompositeKey(compositeIndexaccount, []string{account, tokens, txid})
if compositeErr != nil {
return shim.Error(fmt.Sprintf("Could not create a composite key for %s: %s", account, compositeErr.Error()))
}
// Save the composite key index
compositePutErr := stub.PutState(compositeKey, []byte{0x00})
if compositePutErr != nil {
return shim.Error(fmt.Sprintf("Could not put operation for %s in the ledger: %s", account, compositePutErr.Error()))
}
return shim.Success([]byte(fmt.Sprintf("Successfully added %s to %s", tokens, account)))
}
Could someone explain why I'm still getting a MVCC_READ_CONFLICT on the later implementation? What am I doing wrong? I'm benchmarking and sending the same accountID several times. Though I was under the impression that this would not matter when using a composite-key.
Thanks in advance.
I resolved this issue by removing my own implementation and replacing it with the one from the high-throughput sample [ https://github.com/hyperledger/fabric-samples/blob/release/high-throughput/chaincode/high-throughput.go ].
My guess is that I'm doing something in my implementation that Golang does not agree with. Since the implementations are not that different.

How to feed a password to SSH?

I need to use password authenticated scp to download a file from a server. How do I do so using Go? Tried the following code, but it doesn't pass in the password.
package main
import (
"os/exec"
"time"
)
func main() {
password := "password"
cmd := exec.Command("scp", "admin#192.168.1.150:file", "file")
in, err := cmd.StdinPipe()
if err != nil {
panic(err)
}
defer in.Close()
out, err := cmd.StdoutPipe()
if err != nil {
panic(err)
}
defer out.Close()
if err = cmd.Run(); err != nil {
panic(err)
}
go func() {
time.Sleep(10 * time.Second)
_, err = in.Write([]byte(password + "\n"))
if err != nil {
panic(err)
}
}()
}
Edit: I ended up using the gexpect (github.com/ThomasRooney/gexpect) library.
package main
import (
"github.com/ThomasRooney/gexpect"
"log"
)
func main() {
child, err := gexpect.Spawn("scp admin#192.168.1.150:file file")
if err != nil {
log.Fatalln(err)
}
child.Expect("password:")
child.SendLine("password")
child.Interact()
child.Close()
}
The answer to this self-answered question might help:
Golang write input and get output from terminal process
at least, he mentions in the answer that he "was able to get ssh access working with a password", which is not mentioned explicitly in the question - that's why you probably didn't find it while searching the site?

Resources