How to check number of jobs remaining in job collection of Azure scheculer - azure

I have implemented Azure scheduler by following below blog
http://fabriccontroller.net/a-complete-overview-to-get-started-with-the-windows-azure-scheduler/
Now I am able to create/update/delete jobs in the scheduler but how can I check whether the jobcollection is full? Basically I want to create another collection whenever my current jobcollection is full.
Sharing my code snippet
public class AzureSchedulerStorage : ISchedulerStorage
{
private CertificateCloudCredentials credentials;
//private CloudServiceManagementClient cloudServiceClient;
private string cloudServiceName;
// private IHalseyLogger logger;
public AzureSchedulerStorage(string cloudServiceName, CertificateCloudCredentials credentials)
{
this.cloudServiceName = cloudServiceName;
this.credentials = credentials;
// this.logger = logger;
}
public SchedulerOperationStatusResponse CreateJobCollection(string jobCollectionName)
{
var schedulerServiceClient = new SchedulerManagementClient(credentials);
var jobCollectionCreateParameters = new JobCollectionCreateParameters()
{
Label = jobCollectionName,
IntrinsicSettings = new JobCollectionIntrinsicSettings()
{
Plan = JobCollectionPlan.Standard,
Quota = new JobCollectionQuota()
{
MaxJobCount = 50,
MaxRecurrence = new JobCollectionMaxRecurrence()
{
Frequency = JobCollectionRecurrenceFrequency.Minute
}
}
}
};
var result = schedulerServiceClient.JobCollections.Create(this.cloudServiceName, jobCollectionName, jobCollectionCreateParameters);
return result;
}
public JobCollectionGetResponse GetJobCollection(string jobCollectionName)
{
var schedulerServiceClient = new SchedulerManagementClient(credentials);
var result = schedulerServiceClient.JobCollections.Get(this.cloudServiceName, jobCollectionName);
return result;
}
public void CreateOrUpdate(string jobcollectionName, string jobId, DateTime startDate)
{
var schedulerClient = new SchedulerClient(this.cloudServiceName, jobcollectionName, this.credentials);
var job = new JobCreateOrUpdateParameters()
{
Action = new JobAction()
{
Type = JobActionType.Https,
Request = new JobHttpRequest()
{
Body = "customer=sandrino&command=sendnewsletter",
Headers = new Dictionary<string, string>()
{
{ "Content-Type", "application/x-www-form-urlencoded" },
{ "x-something", "value123" }
},
Method = "POST",
Uri = new Uri("http://postcatcher.in/catchers/527af9acfe325802000001cb"),
}
},
StartTime = startDate,
};
var result = schedulerClient.Jobs.CreateOrUpdate(jobId, job);
}
}
}

After looking at the Scheduler API, it seems that there is no straightforward approach to get the length of a Job Collection.
Probably you can attempt to create a Job and if there is a quota error, then you can create a new Job Collection and add that job.

Related

How to cancel a running trigger function within azure durable functions?

I am trying the fan-out, fan-in pattern. Here's my code
[FunctionName("af_cancellation")]
public static async Task<string> RunOrchestrator(
[OrchestrationTrigger] DurableOrchestrationContext context, ILogger log)
{
var taskList = new List<Task<string>>();
var tokenSource = new CancellationTokenSource();
taskList.Add(context.CallActivityAsync<string>("af_cancellation_Hello", new { ct = tokenSource.Token, city = "Tokyo" }));
taskList.Add(context.CallActivityAsync<string>("af_cancellation_Hello", new { ct = tokenSource.Token, city = "Seattle" }));
taskList.Add(context.CallActivityAsync<string>("af_cancellation_Hello", new { ct = tokenSource.Token, city = "London" }));
try
{
await Task.WhenAll(taskList);
}
catch (FunctionException)
{
log.LogError("trigger function failed");
tokenSource.Cancel();
}
string output = "";
foreach (var t in taskList)
{
output += t.Result;
}
return output;
}
I would like to cancel all the tasks in taskList if any of them throw an exception. What i am noticing is that await Task.WhenAll finishes all the tasks before moving forward.
Here's the sample trigger function
[FunctionName("af_cancellation_Hello")]
public static string SayHello([ActivityTrigger] DurableActivityContext context, ILogger log)
{
var data = context.GetInput<dynamic>();
var name = (string)data.city;
// unable to de-serialize cancellation token here but we'll ignore that.
var ct = JsonConvert.DeserializeObject<CancellationToken>(data.ct);
if (name != "London")
{
System.Threading.Thread.Sleep(1000 * 30);
}
else
{
System.Threading.Thread.Sleep(1000 * 10);
throw new FunctionException("don't like london");
}
log.LogInformation($"Saying hello to {name}.");
return $"Hello {name}!";
}
How can i achieve this?
According to this I think it's not possible. If you need to undo the job from the other activities that succeeded, you must take a look on the Saga Pattern and launch compensating activities.
more info:
https://microservices.io/patterns/data/saga.html
https://www.enterpriseintegrationpatterns.com/patterns/conversation/CompensatingAction.html

CSOM CreateSPAsyncReadJob stays in Queue state

I am referring to the Migration Asynchronous Read API that allows creating a read job on SharePoint using CSOM. I am able to create the read job successfully but unfortunately, the job stays in a queue state since long.
The function returns the Object that includes UniqueJobID, AzureContainerManifestUri, AzureQueueReportUri and EncryptionKey
By using clientContext.Site.GetMigrationJobStatus method I am able to check the read job status that always returns Queued
Here is the sample code for reference:
using (var clientContext = new ClientContext(siteUrl))
{
clientContext.Credentials = new SharePointOnlineCredentials(userName, password);
var result = clientContext.Site.CreateSPAsyncReadJob($"{siteUrl}/List/MyList", new AsyncReadOptions { });
clientContext.ExecuteQuery();
MigrationJobState state;
do
{
var status = clientContext.Site.GetMigrationJobStatus(result[0].JobId);
clientContext.ExecuteQuery();
state = status.Value;
} while (state == MigrationJobState.Queued);
}
I have also tried to connect to the AzureQueueReportUri queue that contains the message with encrypted content. I am not sure how we can decrypt the content to make it human readable. Here is the sample message:
{
"Label": "Encrypted",
"JobId": "079ece4a-cfd2-4676-a27d-2662beb5bb0a",
"IV": "RYc+ZA2feX1hnAcVWR1R+w==",
"Content": "qbjTBbb2N+DkNumLoCJSAAfwj8etDLgjxp+b2T9k03L9WfRJKlFBIZO457q+CbHA+8DHJS7VbPzVMoW6ybo2GxgteTYVP+yVUOPPvz57VGQJyzg2gss+Bsjn73GTWWUfwC/W+oWnEpt8PawZysCjSNf6A4HKZKewkskCshN/pND8ZpevrGt2qq0dTt0NkTIkuYv5AvIP7DSWjdl7nN/W5x4c2nR0sPFqKYom41a4tIqrruzwCDEEjWLFtuXAQ+UN2TMV9PWabRFe9n/P1RHrAJaNU+JjJiJm+lE1dQChz+7OuQoJsYnbjYTbqEE8CnIB0/E0zTrc3zLc6th8MBsKpZJjd31ovqr/Xez6zCnvMKotSdScFtTgQqHxmVDBMfMgi2mm8cKQpdKwRufP/YhaDQlvFkmj2FQN0KAMNxwFBh/MWCVhz5uCJ50CGhChcn4h"
}
I am also not able to connect the AzureContainerManifestUri blob container. It fails with an error Authentication Error. Signature did not match.
Can anyone please guide me how can I proceed ahead?
The method parameters have been changed. Here is the latest updated documentation: https://learn.microsoft.com/en-us/sharepoint/dev/apis/export-amr-api
Sample Code: https://gist.github.com/techmadness/484e7de0a7c51e5faf952a79f1eacb85
using System;
using System.Linq;
using System.Security;
using System.Threading;
using Microsoft.SharePoint.Client;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
using Microsoft.WindowsAzure.Storage.Queue;
namespace ConsoleApp1
{
internal class Program
{
private static void Main(string[] args)
{
var userName = "admin#tenant.onmicrosoft.com";
var password = GetSecurePassword("password");
var siteUrl = "https://tenant.sharepoint.com/sites/testsite";
var listUrl = $"{siteUrl}/testlist";
var azStorageConnectionStrong = "DefaultEndpointsProtocol=https;AccountName=abcd;AccountKey=xyz";
using (var clientContext = new ClientContext(siteUrl))
{
clientContext.Credentials = new SharePointOnlineCredentials(userName, password);
var azManifestContainer = CreateContainerIfNotExists(azStorageConnectionStrong, "spread-manifest-container");
var azReportQueue = CreateQueueIfNotExists(azStorageConnectionStrong, "spread-report-queue");
var azManifestContainerUrl = GetSASUrl(azManifestContainer);
var azReportQueueUrl = GetSASUrl(azReportQueue);
var output = clientContext.Site.CreateSPAsyncReadJob(
listUrl,
new AsyncReadOptions
{
IncludeDirectDescendantsOnly = true,
IncludeSecurity = true,
},
null,
azManifestContainerUrl,
azReportQueueUrl);
clientContext.ExecuteQuery();
CloudQueueMessage message;
do
{
Thread.Sleep(TimeSpan.FromSeconds(10));
message = azReportQueue.GetMessage();
if (message != null)
{
Console.WriteLine(message.AsString);
azReportQueue.DeleteMessage(message);
}
} while (message != null);
Console.ReadLine();
}
}
private static SecureString GetSecurePassword(string pwd)
{
SecureString securePassword = new SecureString();
foreach (var ch in pwd.ToArray())
{
securePassword.AppendChar(ch);
}
return securePassword;
}
private static CloudBlobContainer CreateContainerIfNotExists(string storageConnectionString, string containerName)
{
var storageAccount = CloudStorageAccount.Parse(storageConnectionString);
var blobClient = storageAccount.CreateCloudBlobClient();
var container = blobClient.GetContainerReference(containerName);
container.CreateIfNotExistsAsync().GetAwaiter().GetResult();
return container;
}
private static CloudQueue CreateQueueIfNotExists(string storageConnectionString, string queueName)
{
var cloudStorageAccount = CloudStorageAccount.Parse(storageConnectionString);
var queueClient = cloudStorageAccount.CreateCloudQueueClient();
var queue = queueClient.GetQueueReference(queueName);
queue.CreateIfNotExistsAsync().GetAwaiter().GetResult();
return queue;
}
public static string GetSASUrl(CloudBlobContainer container)
{
var sharedAccessSignature = container.GetSharedAccessSignature(new SharedAccessBlobPolicy
{
Permissions = SharedAccessBlobPermissions.Read | SharedAccessBlobPermissions.Write,
SharedAccessStartTime = DateTime.UtcNow.AddDays(-1),
SharedAccessExpiryTime = DateTime.UtcNow.AddDays(7),
});
return container.StorageUri.PrimaryUri + sharedAccessSignature;
}
public static string GetSASUrl(CloudQueue queue)
{
var sharedAccessSignature = queue.GetSharedAccessSignature(new SharedAccessQueuePolicy
{
Permissions = SharedAccessQueuePermissions.Add | SharedAccessQueuePermissions.Read,
SharedAccessStartTime = DateTime.UtcNow.AddDays(-1),
SharedAccessExpiryTime = DateTime.UtcNow.AddDays(7)
});
return queue.StorageUri.PrimaryUri + sharedAccessSignature;
}
}
}

Azure Table Storage not saving all object properties

I have a problem with the Azure Table Storage. What I'm trying to achieve is saving the ChangeToken of the SharePoint list in order to use the webhooks properly.
Here is the code:
public class TablesHelper
{
private static readonly string TokenTableName = "TokenTable";
public static async Task<ListChangeToken> GetChangeTokenForListAsync(string listId)
{
var retrieveOperation = TableOperation.Retrieve<ListChangeToken>("Lists", listId, new List<string>() { "ChangeToken" });
var tableReference = await GetTableReferenceAsync(TokenTableName);
var tableResult = await tableReference.ExecuteAsync(retrieveOperation);
if(tableResult.Result != null)
{
return tableResult.Result as ListChangeToken;
}
return null;
}
public static async Task SaveChangeTokenForListAsync(ListChangeToken changeToken)
{
var insertOperation = TableOperation.Insert(changeToken);
var tableReference = await GetTableReferenceAsync(TokenTableName);
var result = await tableReference.ExecuteAsync(insertOperation);
}
private static async Task<CloudTable> GetTableReferenceAsync(string tableName)
{
var storageAccount = CloudStorageAccount.Parse(ConfigurationHelper.CloudStorage);
var tableClient = storageAccount.CreateCloudTableClient();
var reference = tableClient.GetTableReference(tableName);
await reference.CreateIfNotExistsAsync();
return reference;
}
}
The ListChangeToken class:
public class ListChangeToken : TableEntity
{
public ListChangeToken(string listId, string changeToken)
{
this.PartitionKey = "Lists";
this.RowKey = listId;
this.ChangeToken = changeToken;
}
public ListChangeToken() { }
public string ChangeToken { get; set;}
}
As per request, the function calling TablesHelper:
[FunctionName("EventHandler")]
public static async Task Run([QueueTrigger("events", Connection = "CloudStorage")]string myQueueItem, TraceWriter log)
{
var notificationGroup = Newtonsoft.Json.JsonConvert.DeserializeObject<NotificationGroup>(myQueueItem);
var contextHelper = new ContextHelper();
foreach (var notification in notificationGroup.Value)
{
UriBuilder uriBuilder = new UriBuilder();
uriBuilder.Scheme = "https";
uriBuilder.Host = ConfigurationHelper.TenantDomain;
uriBuilder.Path = notification.SiteUrl;
using (var ctx = contextHelper.GetAppOnlyContext(uriBuilder.ToString()))
{
//Read change token
var currentChangeToken = await TablesHelper.GetChangeTokenForListAsync(notification.Resource);
if(currentChangeToken == null)
{
log.Error($"No change token found for list {notification.Resource}. This is a NO GO. Please use the '/api/Setup' function.");
}
var listId = Guid.Parse(notification.Resource);
var changes = await CSOMHelper.GetListItemChangesAsync(ctx, listId, currentChangeToken.ChangeToken);
if(changes.Count > 0)
{
var lastChange = changes[changes.Count - 1];
//Save the last change token
var changeTokenValue = lastChange.ChangeToken.StringValue;
await TablesHelper.SaveChangeTokenForListAsync(new ListChangeToken(
notification.Resource,
changeTokenValue
));
await HandleChanges(ctx, changes);
}
}
}
log.Info($"C# Queue trigger function processed: {myQueueItem}");
}
The problem is that always, when using the "GetChangeTokenForListAsync" the Entity is received properly, but the .ChangeToken property is always null. It is also not visible when browsing with the Azure Storage Explorer. What am I doing wrong here?
The issue is related to the Azure Storage Emulator (V. 5.7.0.0). The same code works perfectly when working with the "live" Azure.

running stored procedures into own model with servicestack ormlite

Is there any examples to be found for running a stored procedure on serviceStack MVC using ormlite? mythz ? seen this block of code:
var results = new List<EnergyCompare>
{dbFactory.Exec(dbCmd =>
{
dbCmd.CommandType = CommandType.StoredProcedure;
dbCmd.Parameters.Add(new SqlParameter("#id", 1));
dbCmd.CommandText = "GetAuthorById";
return dbCmd.ExecuteReader().ConvertTo<EnergyCompare>();
}
)};
but came with the text of never worked on the google groups!
i can also write this:
using(var db = new SwitchWizardDb())
{
var results2 = db.dbCmd.ExecuteProcedure()
}
but not sure how to complete this with parameters, and in the source code I looked at, it said obsolete?
thanks
Looks like ServiceStack.ORMLite has been updated to make this easier:
List<Poco> results = db.SqlList<Poco>("EXEC GetAnalyticsForWeek 1");
List<Poco> results = db.SqlList<Poco>("EXEC GetAnalyticsForWeek #weekNo", new { weekNo = 1 });
List<int> results = db.SqlList<int>("EXEC GetTotalsForWeek 1");
List<int> results = db.SqlList<int>("EXEC GetTotalsForWeek #weekNo", new { weekNo = 1 });
This example is on the front page of the github repo.
Well I figured it was best to roll my own handler so have created this, any thoughts would be most welcome, especially with how I could pass over params in some kind of func or something:
I have a main class to deal with easy access to my connection object:
public class DatabaseNameSp : IDisposable
{
private readonly SqlConnection _spConn = new SqlConnection(DatabaseNameSp .dbConString);
public readonly SqlCommand SpCmd;
public DatabaseNameSp (string procedureName)
{
_spConn.Open();
SpCmd = new SqlCommand
{
Connection = _spConn,
CommandType = CommandType.StoredProcedure,
CommandText = procedureName
};
}
public void Dispose()
{
_spConn.Close();
SpCmd.Dispose();
}
}
usage:
using (var db = new DatabaseNameSp ("procedurenname"))
{
db.SpCmd.Parameters.Add(new SqlParameter("#Id", 1));
var rdr = db.SpCmd.ExecuteReader(CommandBehavior.CloseConnection);
var results = new List<CustomDTO>();
while (rdr.Read())
{
results.Add(new CustomDTO { Name = rdr["name"].ToString(), Id = rdr["id"].ToString() });
}
return new CustomDTOResponse { Results = results };
}
Any thoughts !
thanks
Here is an example of running a stored procedure with ormLite that may help you:
IList<MyDTO> myList = DbFactory.Run(dbCnx =>
{
using (var dbCmd = dbCnx.CreateCommand())
{
dbCmd.CommandType = CommandType.StoredProcedure;
dbCmd.CommandText = "mySchema.myStoredProc";
dbCmd.Parameters.Add(new SqlParameter("#param1", val1));
dbCmd.Parameters.Add(new SqlParameter("#param2", val2));
var r = dbCmd.ExecuteReader();
return r.ConvertToList<MyDTO>();
}
});
To just simply run a stored procedure with no data returned:
public class ComsManager : Dbase
{
private IDbConnection dbConn;
public ComsManager()
{
dbConn = Dbase.GetConnection();
}
public void Housekeeping()
{
using(var dbCmd = dbConn.CreateCommand())
dbConn.Exec(res => { dbCmd.CommandType = CommandType.StoredProcedure; dbCmd.CommandText = "SP_housekeeping"; dbCmd.ExecuteNonQuery(); });
}

How to Mock Subsonic ExecuteReader method?

I have a method that calls stored procedure and returns the data after executing DataReader.
I am trying to test the method using mock. I am not sure how to return value?
Anyone did this? Appreciate your responses.
Here is my code:
// Call the StoredProcedure
public List<string> GetCompletedBatchList(int fileId)
{
List<string> completedBatches = new List<string>();
StoredProcedure sp = new StoredProcedure("GetDistributedBatches", this.dataProvider);
sp.Command.AddParameter("FileID", fileId, DbType.Int32, ParameterDirection.Input);
sp.Command.AddParameter("Result", null, DbType.Int32, ParameterDirection.InputOutput);
using (var rdr = sp.ExecuteReader())
{
while (rdr != null && rdr.Read())
{
if (rdr[0] != null)
{
completedBatches.Add(rdr[0].ToString());
}
}
}
return completedBatches;
}
Here is the Test Method:
[Test]
public void Can_get_completedBatches()
{
var file = new File() { FileID = 1, DepositDate = DateTime.Now };
repo.Add<File>(file);
CompletedBatches completedBatches = new CompletedBatches(provider.Object);
//Here I am not sure how to Return
provider.Setup(**x => x.ExecuteReader(It.IsAny<QueryCommand>())).Returns** =>
{
cmd.OutputValues.Add(0);
});
var completedBatchesList = completedBatches.GetCompletedBatchList(file.FileID);
Assert.AreEqual(0, completedBatchesList.Count());
}
If you want to create a DataReader of a certain shape and size then I suggest you look at DataTable.CreateDataReader DataTable.CreateDataReader. You can then setup the ExecuteReader in your example to return this datareader.
The following link helped me...
How to mock an SqlDataReader using Moq - Update
I used MockDbDataReader method to mock the data
[Test]
public void Can_get_completedBatches_return_single_batch()
{
var date = DateTime.Now;
var file = new File() { FileID = 202, DepositDate = DateTime.Now };
var batch1 = new Batch() { FileID = 202, BatchID = 1767, LockboxNumber = "1", IsLocked = true, LockedBy = "testUser" };
var transaction1 = new Transaction() { BatchID = 1767, TransactionID = 63423, CheckAmount = 100.0 };
var distribution1 = new Distribution() { TransactionID = 63423, InvoiceNumber = "001", Amount = 100.0, DateCreated = date, DateModified = date, TransType = 2 };
repo.Add<File>(file);
repo.Add<Batch>(batch1);
repo.Add<Transaction>(transaction1);
repo.Add<Distribution>(distribution1);
CompletedBatches completedBatches = new CompletedBatches(provider.Object);
provider.Setup(x => x.ExecuteReader(It.IsAny<QueryCommand>())).Returns(MockDbDataReader());
var completedBatchesList = completedBatches.GetCompletedBatchList(202);
Assert.AreEqual(1, completedBatchesList.Count());
}
// You should pass here a list of test items, their data
// will be returned by IDataReader
private DbDataReader MockDbDataReader(List<TestData> ojectsToEmulate)
{
var moq = new Mock<DbDataReader>();
// This var stores current position in 'ojectsToEmulate' list
int count = -1;
moq.Setup(x => x.Read())
// Return 'True' while list still has an item
.Returns(() => count < ojectsToEmulate.Count - 1)
// Go to next position
.Callback(() => count++);
moq.Setup(x => x["BatchID"])
// Again, use lazy initialization via lambda expression
.Returns(() => ojectsToEmulate[count].ValidChar);
return moq.Object;
}

Resources