Here's my setup: I have a SW_Dev stream which is loaded in SW_Dev_RWS workspace. This is where the daily work is done. Every once in a while, I make a snapshot of SW_Dev, and create a workspace from that snapshot, SW_Release_v1_RWS. Then I change the flow target of SW_Release_v1_RWS to SW_Release stream deliver all changes there and also make a snapshot. The goal is to have snapshots for all releases in SW_Release stream. I also keep SW_Release_v1_RWS loaded for a while, since I often need to produce release binaries with small tweaks for testing.
The problem I have appears when I have unfinished work in SW_Dev which I have to store in a suspended changeset. For some reason, that suspended changeset also appears (as suspended) in SW_Release_v1_RWS. Of course that means there are no actual modifications to the release workspace, but it doesn't look tidy. What's worse, if I try to discard a suspended changeset from SW_Release_v1_RWS, it also disappears from SW_Dev_RWS.
Is there a way to prevent suspended changesets created in SW_Dev_RWS from appearing in SW_Release_v1_RWS? Perhaps there is a different approach I should adopt for saving release snapshots in SW_Release stream?
Related
I'm using node-cache to cache data from a CLI application that observes changes in files and caches them to avoid new data processing.
the problem is that I noticed that this cache is destroyed on each command, since each time the tool is called in the terminal a new instance is generated and the old one is destroyed. probably, the data is also destroyed.
I need to keep, for a specific TTL, two things in cache/memory, even if the process ends:
the processed data
the specific instance of fs.watcher, watching and executing caching operations
the question is: how do i do it? I've been searching for days on the internet and trying alternatives and I can't find a solution.
I need to keep ... things in cache/memory, even if the process ends
That's, pretty much by definition, not possible. When a process terminates, all its resources are freed up for use by something else (barring a memory-leak bug in the OS itself).
It sounds like you need to refactor your app into a service that can run in the background and separate front-end that can communicate with it.
can anyone explain what Batch in Azure YAML Schema Trigger does?
The only explanation on MSFT website is
batch changes if true; start a new build for every push if false (default)
and this isn't really clear to me
Batch changes or Batch trigger actually means batching your CI runs.
If you have many team members uploading changes often, you may want to reduce the number of runs you start. If you set batch to true, when a pipeline is running, the system waits until the run is completed, then starts another run with all changes that have not yet been built.
To clarify this example, let us say that a push A to master caused the above pipeline to run. While that pipeline is running, additional pushes B and C occur into the repository. These updates do not start new independent runs immediately. But after the first run is completed, all pushes until that point of time are batched together and a new run is started.
My interpretation of MS documentation is that the batch boolean is meant to address concerns with encountering frequent pushes to the same trigger branch or set of branches (and possibly tags) and works such that if the build pipeline is already running, any additional changes pushed to the listed branches will be batched together and queued behind the current run. This does mean that those subsequent pushes will be part of the same subsequent pipeline run which is a little strange, but given that's how Microsoft intended it to work it should be fine.
Basically, for repos that have a high potential for demanding pipeline runs and multiple overlapping pushes occurring, batching is great.
For reference, here is the documentation link: https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/azure-repos-git?view=azure-devops&tabs=yaml#batching-ci-runs
My use case scenario of Azure cloud block blob is like following:
1. There is a blob called phone-numbers.txt in azure storage container
2. Machine A downloads the blob, makes some changes, then upload it to the container with the same blob name.
3. Machine B may do the same thing as machine A, but with different changes.
Machine A and B could download the blob at the exactly same time, but machine A may be faster completing and uploading the change to container.
The issue happens when machine B uploads its change without knowing that machine A has made a change to the blob. Of course, I can have each machine do Append only, but if the changes made by both machines are same, I need to keep only one instead of appending the same change twice.
So I was wondering if Azure store has any existing mechanism to prevent this overwrite happening. If Azure does not have it, I was thinking about using the metadata in the blob and having a version field in the metadata:
When a machine updates the blob, it increments the version by +1. Before the machine uploads the blob, it downloads the same blob and check whether its version has changed from its first download. I know this is not perfect and the blob download seems redundant.
Any other better approaches to prevent the overwriting from happening?
IMHO blob leasing is not the right solution to this problem. Let me explain.
Let's assume both process A and B downloads the blob to work on it and A is able to acquire the lease on the blob. Now consider these scenarios:
B has finished the job before A: Even if B has finished the job before A, B can't save the blob as A has exclusive lock on the blob. B will need to wait for A to release the lock before B can save the changes.
A has finished the job before B: In this scenario the lock will be released hence B will be able to save the changes but it will have no idea that A has made the changes thus A's changes will be overwritten with B's changes.
To solve this issue, you will need to use something called Optimistic Concurrency which is supported by default in Azure Storage. This is accomplished by making use of of ETag property of a blob which gets updated anytime a blob is changed.
So considering the same example as above, both A and B downloaded the blob and start working on editing it.
B has finished the job before A: In this case, B will be able to save the changes. As soon as B saves the changes, ETag of the blob will be changed and now when A try to save the changes an error (419 - Precondition Failed) will be thrown by storage service which will tell A that the blob has changed. A will need to download the blob again and make the changes and save again.
A has finished the job before B: Same thing will happen as above but now B will get the same error and B will need to re-download the blob and make the changes and save again.
You can learn more about concurrency in Blob Storage here: https://learn.microsoft.com/en-us/azure/storage/common/storage-concurrency#managing-concurrency-in-blob-storage
Take a look at the Microsoft docs for Leasing Blobs. This effectively allows you to request a lease/lock on the blob before you update, which will prevent other processes from updating (or deleting) that blob for the duration of the lease. So each process can do something like the following to prevent concurrent updates to the same blob:
Attempt to acquire lease for blob - This will fail if a lease has already been acquired by another process. You can specify a timeout for this attempt so that it does not fail immediately and gives the other process a chance to finish updating the blob. Once this lease has been acquired, no updates to the blob can be made by other processes until it is released.
Download blob - Since an exclusive lease has been acquired by this process during step (1), the downloaded blob is guaranteed to contain the most recent changes.
Make changes to contents
Overwrite old blob by uploading new contents - Since an exclusive lease has been acquired by this process during step (1), the blob being overwritten is guaranteed to just be the same blob that was downloaded in step (2) because the lease protects against updates from other processes for its duration.
Release lease - Frees up the blob to be updated by other processes.
you will have to compare metadata of the 2 files.
https://myaccount.blob.core.windows.net/mycontainer/myblob?comp=metadata
Alternatively consider a multi-writer scenario with append blobs where each machine can append a block to an existing blob.
I know there are upper limits on file size when it comes to media, but i'm testing with small images.
If i change a page, a simple content changed, the scheduled task syncs the change and i can see the update on the target server. If i upload an image to the media library, it doesn't sync via the scheduled tasks. I can sync it manually though.
I've read through the documentation, and didn't see anything where media files wouldn't sync, just the exact opposite though.
So what am i missing to get media files to sync?
It's beacuse the scheduled task synchronizes only page changes. You cannot use the default Content synchronization task to synchronize other objects.
But you can use event handlers and some API to synchronize any type of staging task.
https://docs.kentico.com/display/K9/Automatically+synchronizing+staging+and+integration+tasks
I have a continuous Webjob running on my Azure Website. It is responsible for doing some work after retrieving items from a QueueTrigger. I am attempting to increase the rate in which the items are processed off the Queue. As I scale out my App Service Plan, the processing rate increases as expected.
My concern is that it seems wasteful to pay for additional VMs just to run additional instances of my Webjob. I am looking for options/best practices to run multiple instances of the same Webjob on a single server.
I've tried starting multiple JobHosts in individual threads within Main(), but either that doesn't work or I was doing something wrong... the Webjob would fail to run due to what looks like each thread trying to access 'WebJobSdk.marker'. My current solution is to publish my Webjob multiple times, each time modifying 'webJobName' slightly in 'webjob-publish-settings.json' so that the same project is considered a different Webjob at publish time. This works great so far, expect that it creates a lot of additional work each time I need to make any update.
Ultimately, I'm looking for some advice on what the recommended way of accomplishing this would be. Ideally, I would like to get the multiple instances running via code, and only have to publish once when I need to update the code.
Any thoughts out there?
You can use the JobHostConfiguration.QueuesConfiguration.BatchSize and NewBatchThreshold settings to control the concurrency level of your queue processing. The latter NewBatchThreshold setting is new in the current in progress beta1 release. However, by enabling "prerelease" packages in your Nuget package manager, you'll see the new release if you'd like to try it. Raising the NewBatchThreshold setting increases the concurrency level - e.g. setting it to 100 means that once the number of currently running queue functions drops below 100, a new batch of messages will be fetched for concurrent processing.
The marker file bug was fixed in this commit a while back, and again is part of the current in progress v1.1.0 release.