We have a solution which takes Incremental Snapshots of all the disks of a virtual machine. Once the snapshot is created, the solution queries it's page ranges to get the changed data.
The issue which we are facing currently is that the page ranges are returned as empty even when it's a first snapshot for the disk and the disk has data. This happens intermittently for OS as well as Data disks. Strangely, if a virtual machine has multiple disks, the page ranges return appropriate information for few and empty for others.
We are using Virtual Machine Disk Incremental Snapshot Operation as below. The solution makes use of GO SDK of Azure for making these operations. Below is the sample code for the operations.
// Prepare azure snapshot api client
snapClient, err := armcompute.NewSnapshotsClient(subscriptionID, azureCred, nil)
// Configure snapshot parameters
snapParams := armcompute.Snapshot{
Location: to.Ptr(location), // Snapshot location
Name: &snapshotName,
Properties: &armcompute.SnapshotProperties{
CreationData: &armcompute.CreationData{
CreateOption: to.Ptr(armcompute.DiskCreateOptionCopy),
SourceResourceID: to.Ptr(getAzureIDFromName(constant.AzureDiskURI, subscriptionID, resourceGroupName, diskID, "")),
}, // Disk ID for which the snapshot needs to be created
Incremental: to.Ptr(true),
},
}
// Create Disk Snapshot (Incremental)
snapPoller, err := snapClient.BeginCreateOrUpdate(ctx, resourceGroupName, snapshotName, snapParams, nil)
if err != nil {
return nil, err
}
Once the snapshot is created successfully, we prepare the changed areas for the snapshot using PageRanges feature as below.
// Grant Snapshot Access
resp, err := snapClient.BeginGrantAccess(ctx, resourceGroupName, snapshotId), armcompute.GrantAccessData{
Access: &armcompute.PossibleAccessLevelValues()[1], // 0 is none, 1 is read and 2 is write
DurationInSeconds: to.Ptr(duration), // 1 hr
}, nil)
if err != nil {
return "", err
}
grantResp, err := resp.PollUntilDone(ctx, nil)
if err != nil {
return "", err
}
currentSnapshotSAS = grantResp.AccessSAS
// Create Page Blob Client
pageBlobClient, err := azblob.NewPageBlobClientWithNoCredential(currentSnapshotSAS, nil)
pageOption := &azblob.PageBlobGetPageRangesDiffOptions{}
pager := pageBlobClient.GetPageRangesDiff(pageOption)
if err != nil {
return nil, err
}
// Gather Page Ranges for all the changed data
var pageRange []*azblob.PageRange
if pager.NextPage(ctx) {
resp := pager.PageResponse()
pageRange = append(pageRange, resp.PageRange...)
}
// Loop through page ranges and collect all changed data indexes
var changedAreaForCurrentIter int64
changedAreasString := ""
for _, page := range pageRanges {
length := (*page.End + 1) - *page.Start
changedAreaForCurrentIter = changedAreaForCurrentIter + length
changedAreasString = changedAreasString + strconv.FormatInt(*page.Start, 10) + ":" + strconv.FormatInt(length, 10) + ","
}
zap.S().Debug("Change areas : [" changedAreasString "]")
It is this when the Change areas is coming in as empty. We have checked Disk properties for the ones which are successful and which failed and they are all the same. There is no Lock configured on the disk.
Following are the SDK versions which we are using.
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v3 v3.0.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.1.0
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.4.1
Can someone please provide pointers explaining what are the factors which could make this problem appear intermittently.
Trying to create a GO function that produces the same result as the Ubuntu Linux "cksum" operation, for example:
$ echo 123 > /tmp/foo
$ cksum /tmp/foo
2330645186 4 /tmp/foo
Could someone please provide a GO function that produces the first substring of the above result ("2330645186")? Thank you.
(Update)
It turns out cksum doesn't implement a cyclic redundancy check based on the CRC32 process (quite). To test CRC32 (the same as you'd find listed for a CRC32 checksum) you can use CRC calculation # http://zorc.breitbandkatze.de/ - go's hash/crc32.ChecksumIEEE implementation matches this
To implement the cksum crc process (also known as POSIX cksum) I instead generated a golang version of the c algorithm found on a cksum man page (which uses a lookup table)
package main
import (
"bufio"
"fmt"
"io"
"os"
)
var tbl = [256]uint32{0x00000000, 0x04C11DB7, 0x09823B6E, 0x0D4326D9,
0x130476DC, 0x17C56B6B, 0x1A864DB2, 0x1E475005,
0x2608EDB8, 0x22C9F00F, 0x2F8AD6D6, 0x2B4BCB61,
0x350C9B64, 0x31CD86D3, 0x3C8EA00A, 0x384FBDBD,
0x4C11DB70, 0x48D0C6C7, 0x4593E01E, 0x4152FDA9,
0x5F15ADAC, 0x5BD4B01B, 0x569796C2, 0x52568B75,
0x6A1936C8, 0x6ED82B7F, 0x639B0DA6, 0x675A1011,
0x791D4014, 0x7DDC5DA3, 0x709F7B7A, 0x745E66CD,
0x9823B6E0, 0x9CE2AB57, 0x91A18D8E, 0x95609039,
0x8B27C03C, 0x8FE6DD8B, 0x82A5FB52, 0x8664E6E5,
0xBE2B5B58, 0xBAEA46EF, 0xB7A96036, 0xB3687D81,
0xAD2F2D84, 0xA9EE3033, 0xA4AD16EA, 0xA06C0B5D,
0xD4326D90, 0xD0F37027, 0xDDB056FE, 0xD9714B49,
0xC7361B4C, 0xC3F706FB, 0xCEB42022, 0xCA753D95,
0xF23A8028, 0xF6FB9D9F, 0xFBB8BB46, 0xFF79A6F1,
0xE13EF6F4, 0xE5FFEB43, 0xE8BCCD9A, 0xEC7DD02D,
0x34867077, 0x30476DC0, 0x3D044B19, 0x39C556AE,
0x278206AB, 0x23431B1C, 0x2E003DC5, 0x2AC12072,
0x128E9DCF, 0x164F8078, 0x1B0CA6A1, 0x1FCDBB16,
0x018AEB13, 0x054BF6A4, 0x0808D07D, 0x0CC9CDCA,
0x7897AB07, 0x7C56B6B0, 0x71159069, 0x75D48DDE,
0x6B93DDDB, 0x6F52C06C, 0x6211E6B5, 0x66D0FB02,
0x5E9F46BF, 0x5A5E5B08, 0x571D7DD1, 0x53DC6066,
0x4D9B3063, 0x495A2DD4, 0x44190B0D, 0x40D816BA,
0xACA5C697, 0xA864DB20, 0xA527FDF9, 0xA1E6E04E,
0xBFA1B04B, 0xBB60ADFC, 0xB6238B25, 0xB2E29692,
0x8AAD2B2F, 0x8E6C3698, 0x832F1041, 0x87EE0DF6,
0x99A95DF3, 0x9D684044, 0x902B669D, 0x94EA7B2A,
0xE0B41DE7, 0xE4750050, 0xE9362689, 0xEDF73B3E,
0xF3B06B3B, 0xF771768C, 0xFA325055, 0xFEF34DE2,
0xC6BCF05F, 0xC27DEDE8, 0xCF3ECB31, 0xCBFFD686,
0xD5B88683, 0xD1799B34, 0xDC3ABDED, 0xD8FBA05A,
0x690CE0EE, 0x6DCDFD59, 0x608EDB80, 0x644FC637,
0x7A089632, 0x7EC98B85, 0x738AAD5C, 0x774BB0EB,
0x4F040D56, 0x4BC510E1, 0x46863638, 0x42472B8F,
0x5C007B8A, 0x58C1663D, 0x558240E4, 0x51435D53,
0x251D3B9E, 0x21DC2629, 0x2C9F00F0, 0x285E1D47,
0x36194D42, 0x32D850F5, 0x3F9B762C, 0x3B5A6B9B,
0x0315D626, 0x07D4CB91, 0x0A97ED48, 0x0E56F0FF,
0x1011A0FA, 0x14D0BD4D, 0x19939B94, 0x1D528623,
0xF12F560E, 0xF5EE4BB9, 0xF8AD6D60, 0xFC6C70D7,
0xE22B20D2, 0xE6EA3D65, 0xEBA91BBC, 0xEF68060B,
0xD727BBB6, 0xD3E6A601, 0xDEA580D8, 0xDA649D6F,
0xC423CD6A, 0xC0E2D0DD, 0xCDA1F604, 0xC960EBB3,
0xBD3E8D7E, 0xB9FF90C9, 0xB4BCB610, 0xB07DABA7,
0xAE3AFBA2, 0xAAFBE615, 0xA7B8C0CC, 0xA379DD7B,
0x9B3660C6, 0x9FF77D71, 0x92B45BA8, 0x9675461F,
0x8832161A, 0x8CF30BAD, 0x81B02D74, 0x857130C3,
0x5D8A9099, 0x594B8D2E, 0x5408ABF7, 0x50C9B640,
0x4E8EE645, 0x4A4FFBF2, 0x470CDD2B, 0x43CDC09C,
0x7B827D21, 0x7F436096, 0x7200464F, 0x76C15BF8,
0x68860BFD, 0x6C47164A, 0x61043093, 0x65C52D24,
0x119B4BE9, 0x155A565E, 0x18197087, 0x1CD86D30,
0x029F3D35, 0x065E2082, 0x0B1D065B, 0x0FDC1BEC,
0x3793A651, 0x3352BBE6, 0x3E119D3F, 0x3AD08088,
0x2497D08D, 0x2056CD3A, 0x2D15EBE3, 0x29D4F654,
0xC5A92679, 0xC1683BCE, 0xCC2B1D17, 0xC8EA00A0,
0xD6AD50A5, 0xD26C4D12, 0xDF2F6BCB, 0xDBEE767C,
0xE3A1CBC1, 0xE760D676, 0xEA23F0AF, 0xEEE2ED18,
0xF0A5BD1D, 0xF464A0AA, 0xF9278673, 0xFDE69BC4,
0x89B8FD09, 0x8D79E0BE, 0x803AC667, 0x84FBDBD0,
0x9ABC8BD5, 0x9E7D9662, 0x933EB0BB, 0x97FFAD0C,
0xAFB010B1, 0xAB710D06, 0xA6322BDF, 0xA2F33668,
0xBCB4666D, 0xB8757BDA, 0xB5365D03, 0xB1F740B4}
type crc struct {
p, r uint32
Size int
final bool
}
func NewCrc() *crc {
return &crc{0, 0, 0, false}
}
func (pr *crc) Add(b byte) {
if pr.final {
return
}
pr.r = (pr.r << 8) ^ tbl[byte(pr.r>>24)^b]
pr.Size++
}
func (pr *crc) Crc() uint32 {
if pr.final {
return pr.r
}
for m := pr.Size; m > 0; {
b := byte(m & 0377)
m = m >> 8
pr.r = (pr.r << 8) ^ tbl[byte(pr.r>>24)^b]
}
pr.final = true //Prevent further modification
pr.r = ^pr.r
return pr.r
}
func cksum(filename string) (uint32, int, error) {
f, err := os.Open(filename)
if err != nil {
return 0, 0, err
}
defer f.Close()
in := bufio.NewReader(f)
pr := NewCrc()
for done := false; !done; {
switch b, err := in.ReadByte(); err {
case io.EOF:
done = true
case nil:
pr.Add(b)
default:
return 0, 0, err
}
}
return pr.Crc(), pr.Size, nil
}
func main() {
var filename = "foo"
crc, size, err := cksum(filename)
if err != nil {
fmt.Println("Error: ", err)
return
}
fmt.Printf("%d %d %s\n", crc, size, filename)
}
Obviously in this case the filename is hardcoded (to foo) but you could change that with flags. The content of foo is 123\n (**note: in windows you'll need to convert line endings to not get a 5 byte file) Results:
linux: $ cksum foo
2330645186 4 foo
linux: $ go run cksum.go
2330645186 4 foo
windows: > go run cksum.go **
2330645186 4 foo
Actually, I found a more simplified answer to my original question:
Using:
https://pkg.go.dev/github.com/cxmcc/unixsums#section-readme
Here is the snippet that provides the posix checksum equivalent value of a file in Go:
data, err := ioutil.ReadFile("/tmp/test.loop")
if err != nil {
log.Fatal(err)
}
fmt.Printf("cksum: %d\n", cksum.Cksum(data))
I'm trying to use go on a raspberry pi to query bluetooth low energy devices. It's functional, I can connect to the device I want and iterate through the services and characteristics of the connected device. Now I'm just trying to streamline things and just read/write the values I'm interested in. It isn't working.
Code:
func onPeriphConnected(p gatt.Peripheral, err error) {
fmt.Println("Connected")
defer p.Device().CancelConnection(p)
if err := p.SetMTU(500); err != nil {
fmt.Printf("Failed to set MTU, err: %s\n", err)
}
batteryServiceId := gatt.MustParseUUID("180f")
// Direct read attempt (not working)
batterySerivce := gatt.NewService(batteryServiceId)
batteryLevelUUID := gatt.MustParseUUID("2a19")
batteryChar := gatt.NewCharacteristic(batteryLevelUUID,batterySerivce,gatt.Property(0x12),0,0)
e, err := p.ReadCharacteristic(batteryChar)
if err != nil {
fmt.Printf("Failed to read battery level, err: %s\n", err)
} else {
fmt.Println(e)
}
// iterate services read (working)
ss, err := p.DiscoverServices(nil)
if err != nil {
fmt.Printf("Failed to discover services, err: %s\n", err)
return
}
for _, s := range ss {
if(s.UUID().Equal(batteryServiceId)) {
fmt.Println("Found the battery service")
// Discovery characteristics
cs, err := p.DiscoverCharacteristics(nil, s)
if err != nil {
fmt.Printf("Failed to discover characteristics, err: %s\n", err)
continue
}
for _, c := range cs {
msg := " Characteristic " + c.UUID().String()
if len(c.Name()) > 0 {
msg += " (" + c.Name() + ")"
}
msg += "\n properties " + c.Properties().String()
fmt.Println(msg)
if (c.Properties() & gatt.CharRead) != 0 {
b, err := p.ReadCharacteristic(c)
if err != nil {
fmt.Printf("Failed to read characteristic, err: %s\n", err)
continue
}
fmt.Printf(" value %x\n", b)
}
}
}
}
}
Results:
Connected
[10 0 0 1]
Found the battery service
Characteristic 2a19 (Battery Level)
properties read notify
value 53
You can see where I expect to get a hex value of 53 I'm instead getting an array of [10 0 0 1]. I'm pretty new to go so I'm probably missing something here or just assembling my read incorrectly. Any pointers are much appreciated. Thanks!
A link to the appropriate documentation would be advisable. I don't know if this link is correct as there seem to be multiple different versions of package gatt.
Edit: see also https://godoc.org/github.com/paypal/gatt/examples/service#NewBatteryService at https://github.com/paypal/gatt/blob/master/examples/service/battery.go, which appears to show the right way of creating a battery service directly. Original answer below.
Having had a quick scan through said documentation, two things leap out at me:
The battery level ID is 2a19. The battery service ID is 180f. You use:
batteryServiceId := gatt.MustParseUUID("180f")
batterySerivce := gatt.NewService(batteryServiceId)
batteryChar := gatt.NewCharacteristic(batteryLevelUUID, batterySerivce,
gatt.Property(0x12), 0, 0)
e, err := p.ReadCharacteristic(batteryChar)
(I kept variable name spellings, but added a bit of white space to fit better on the StackOverflow display.) You never call NewDescriptor nor either of AddDescriptor or SetDescriptors on batteryChar. Are such calls required? I don't know; the documentation doesn't say. But the call that works uses DiscoverServices followed by DiscoverCharacteristics, which perhaps does create these (documented but undescribed) Descriptors. They look like they interpose themselves in value-oriented operations, so they might be critical.
Looking further at the code, after or instead of creating a characteristic directly, I think you do have to at least link the characteristic back into the service. The right call might be AddCharacteristic or SetCharacteristics. See this chunk of code in DiscoverCharacteristics.
(Minor.) gatt.Property(0x12) is definitely the wrong way to construct the constant. You probably should use:
gatt.Property.CharRead | gatt.Property.CharNotify
I am writing a function in golang to search for a string in elasticsearch documents which are indexed. I am using elasticsearch golang client elastic. For example consider the object is tweet,
type Tweet struct {
User string
Message string
Retweets int
}
And the search function is
func SearchProject() error{
// Search with a term query
termQuery := elastic.NewTermQuery("user", "olivere")
searchResult, err := client.Search().
Index("twitter"). // search in index "twitter"
Query(&termQuery). // specify the query
Sort("user", true). // sort by "user" field, ascending
From(0).Size(10). // take documents 0-9
Pretty(true). // pretty print request and response JSON
Do() // execute
if err != nil {
// Handle error
panic(err)
return err
}
// searchResult is of type SearchResult and returns hits, suggestions,
// and all kinds of other information from Elasticsearch.
fmt.Printf("Query took %d milliseconds\n", searchResult.TookInMillis)
// Each is a convenience function that iterates over hits in a search result.
// It makes sure you don't need to check for nil values in the response.
// However, it ignores errors in serialization. If you want full control
// over iterating the hits, see below.
var ttyp Tweet
for _, item := range searchResult.Each(reflect.TypeOf(ttyp)) {
t := item.(Tweet)
fmt.Printf("Tweet by %s: %s\n", t.User, t.Message)
}
// TotalHits is another convenience function that works even when something goes wrong.
fmt.Printf("Found a total of %d tweets\n", searchResult.TotalHits())
// Here's how you iterate through results with full control over each step.
if searchResult.Hits != nil {
fmt.Printf("Found a total of %d tweets\n", searchResult.Hits.TotalHits)
// Iterate through results
for _, hit := range searchResult.Hits.Hits {
// hit.Index contains the name of the index
// Deserialize hit.Source into a Tweet (could also be just a map[string]interface{}).
var t Tweet
err := json.Unmarshal(*hit.Source, &t)
if err != nil {
// Deserialization failed
}
// Work with tweet
fmt.Printf("Tweet by %s: %s\n", t.User, t.Message)
}
} else {
// No hits
fmt.Print("Found no tweets\n")
}
return nil
}
This search is printing tweets by the user 'olivere'. But if I give 'olive' then search is not working. How do I search for a string which is part of User/Message/Retweets?
And the Indexing function looks like this,
func IndexProject(p *objects.ElasticProject) error {
// Index a tweet (using JSON serialization)
tweet1 := `{"user" : "olivere", "message" : "It's a Raggy Waltz"}`
put1, err := client.Index().
Index("twitter").
Type("tweet").
Id("1").
BodyJson(tweet1).
Do()
if err != nil {
// Handle error
panic(err)
return err
}
fmt.Printf("Indexed tweet %s to index %s, type %s\n", put1.Id, put1.Index, put1.Type)
return nil
}
Output:
Indexed tweet 1 to index twitter, type tweet
Got document 1 in version 1 from index twitter, type tweet
Query took 4 milliseconds
Tweet by olivere: It's a Raggy Waltz
Found a total of 1 tweets
Found a total of 1 tweets
Tweet by olivere: It's a Raggy Waltz
Version
Go 1.4.2
Elasticsearch-1.4.4
Elasticsearch Go Library
github.com/olivere/elastic
Could anyone help me on this.? Thank you
How you search and find data depends on your analyser - from your code it's likely that the standard analyser is being used (i.e. you haven't specified an alternative in your mapping).
The Standard Analyser will only index complete words. So to match "olive" against "olivere" you could either:
Change the search process
e.g. switch from a term query to a Prefix query or use a Query String query with a wildcard.
Change the index process
If you want to find strings within larger strings then look at using nGrams or Edge nGrams in your analyser.
multiQuery := elastic.NewMultiMatchQuery(
term,
"name", "address", "location", "email", "phone_number", "place", "postcode",
).Type("phrase_prefix")