Problem Statement: We have a requirement to upload log data to Azure Storage from a Xamarin.IOS application. The logs are not created by the user of the application, and there's no constraint on the user to keep the application open for any amount of time after the logs are generated. We want to reliably upload our logs with a couple points in mind:
The user might send the app into the background
The file sizes can be up to 15MB
We don't care when we get them. We're open to scheduling a task for this.
In looking at potential solutions to this problem, the Xamarin documentation states that in iOS7+:
NSURLSession allows us to create tasks to:
Transfer content through network and device interruptions.
Upload and download large files ( Background Transfer Service ).
So it seems like NSURLSession is a good candidate for this sort of work, but I wonder if I am reinventing the wheel. Does the WindowsAzure.Storage client library respect app backgrounding with an upload implementation based on NSURLSession, or if I want to upload the data in the background, is it necessary to upload to an intermediate server I control with a POST method, and then relay data to Azure Storage? There doesn't seem to be any indication from the public Azure documentation that uploads can be done via scheduled task.
I got this working. I've simplified the classes and methods into a single method. Only the necessities are here.
public void UploadFile(File playbackFile)
{
/// Specify your credentials
var sasURL = "?<the sastoken>";
/// Azure blob storage URL
var storageAccount = "https://<yourstorageaccount>.blob.core.windows.net/<your container name>";
/// specify a UNIQUE session name
var configuration =
NSUrlSessionConfiguration.CreateBackgroundSessionConfiguration("A background session name");
/// create the session with a delegate to recieve callbacks and debug
var session = NSUrlSession.FromConfiguration(
configuration,
new YourSessionDelegate(),
new NSOperationQueue());
/// Construct the blob endpoint
var url = $"{storageAccount}/{playbackFile.Name}{sasURL}";
var uploadUrl = NSUrl.FromString(url);
/// Add any headers for Blob PUT. x-ms-blob-type is REQUIRED
var dic = new NSMutableDictionary();
dic.Add(new NSString("x-ms-blob-type"), new NSString("BlockBlob"));
/// Create the request with NSMutableUrlRequest
/// A default NSUrlRequest.FromURL() is immutable with a GET method
var request = new NSMutableUrlRequest(uploadUrl);
request.Headers = dic;
request.HttpMethod = "PUT";
/// Create the task
var uploadTask = session.CreateUploadTask(
request,
NSUrl.FromFilename(playbackFile.FullName));
/// Start the task
uploadTask.Resume();
}
/// Delegate to recieve callbacks. Implementations are omitted for brevity
public class YourSessionDelegate: NSUrlSessionDataDelegate
{
public override void DidBecomeInvalid(NSUrlSession session, NSError error)
{
Console.WriteLine(error.Description);
}
public override void DidSendBodyData(NSUrlSession session, NSUrlSessionTask task, long bytesSent, long totalBytesSent, long totalBytesExpectedToSend)
{
Console.WriteLine(bytesSent);
}
public override void DidReceiveData(NSUrlSession session, NSUrlSessionDataTask dataTask, NSData data)
{
Console.WriteLine(data);
}
public override void DidCompleteWithError(NSUrlSession session, NSUrlSessionTask task, NSError error)
{
var uploadTask = task as NSUrlSessionUploadTask;
Console.WriteLine(error?.Description);
}
public override void DidReceiveResponse(NSUrlSession session, NSUrlSessionDataTask dataTask, NSUrlResponse response, Action<NSUrlSessionResponseDisposition> completionHandler)
{
Console.WriteLine(response);
}
public override void DidFinishEventsForBackgroundSession(NSUrlSession session)
{
using (AppDelegate appDelegate = UIApplication.SharedApplication.Delegate as AppDelegate)
{
var handler = appDelegate.BackgroundSessionCompletionHandler;
if (handler != null)
{
appDelegate.BackgroundSessionCompletionHandler = null;
handler();
}
}
}
}
Helpful documentation:
https://learn.microsoft.com/en-us/rest/api/storageservices/put-blob
https://developer.apple.com/documentation/foundation/nsmutableurlrequest/1408793-setvalue
https://learn.microsoft.com/en-us/dotnet/api/foundation.insurlsessiontaskdelegate?view=xamarin-ios-sdk-12
Hopefully someone finds this useful and spends less time on this than I did. Thanks #SushiHangover for pointing me in the right direction.
Related
In windows azure we have hosted two asp.net webapi project as app service. We need to enable distributed transaction here. We initiate transaction inside one api. Then inside that transaction scope we fetch propagation token of that transaction and send it as a header during another api call. The code is something like bellow.
[HttpGet]
[Route("api/Test/Transaction/Commit")]
public async Task<string> Commit()
{
using (var scope = new TransactionScope(TransactionScopeOption.Required, new TransactionOptions
{
IsolationLevel = IsolationLevel.ReadCommitted
},
TransactionScopeAsyncFlowOption.Enabled))
{
// cross app domain call
using (var client = new HttpClient())
{
using (var request = new HttpRequestMessage(HttpMethod.Get, ConfigurationManager.AppSettings["IdentityServerUri"] + "api/Test/Transaction/NoCommit"))
{
// forward transaction token
request.AddTransactionPropagationToken();
var response = await client.SendAsync(request);
response.EnsureSuccessStatusCode();
}
}
this.Repository.Insert(new Currency { Ccy = "x", IsoCode = "XIS", Name = "XYZ", CurrencyId = 9 });
await this.Repository.SaveChangesAsync();
scope.Complete();
return "value";
}
}
public static class HttpRequestMessageExtension
{
public static void AddTransactionPropagationToken(this HttpRequestMessage request)
{
if (Transaction.Current != null)
{
var token = TransactionInterop.GetTransmitterPropagationToken(Transaction.Current);
request.Headers.Add("TransactionToken", Convert.ToBase64String(token));
}
}
}
Inside the api(...api/Test/Transaction/NoCommit) to which we are making the call inside transaction scope, fetch that marshaled propagation token of the transaction from header and using it create instance of that transaction and instantiate TransactionScope using that transaction. Later we use this transaction scope to complete that transaction. We have introduced a action filter to apply this and added that filter to the action which is responsible for that api call. Code for that api and action filter is something like bellow.
[HttpGet]
[EnlistToDistributedTransactionActionFilter]
[Route("api/Test/Transaction/NoCommit")]
public async Task<string> NoCommit()
{
this.Repository.Insert(new Client
{
Name = "Test",
AllowedOrigin = "*",
Active = true,
ClientGuid = Guid.NewGuid(),
RefreshTokenLifeTime = 0,
ApplicationType = ApplicationTypes.JavaScript,
Secret = "ffff",
Id = "Test"
}
);
await this.Repository.SaveChangesAsync();
return "value";
}
public class EnlistToDistributedTransactionActionFilter : ActionFilterAttribute
{
private const string TransactionId = "TransactionToken";
/// <summary>
/// Retrieve a transaction propagation token, create a transaction scope and promote the current transaction to a distributed transaction.
/// </summary>
/// <param name="actionContext">The action context.</param>
public override void OnActionExecuting(HttpActionContext actionContext)
{
if (actionContext.Request.Headers.Contains(TransactionId))
{
var values = actionContext.Request.Headers.GetValues(TransactionId);
if (values != null && values.Any())
{
byte[] transactionToken = Convert.FromBase64String(values.FirstOrDefault());
var transaction = TransactionInterop.GetTransactionFromTransmitterPropagationToken(transactionToken);
var transactionScope = new TransactionScope(transaction, TransactionScopeAsyncFlowOption.Enabled);
actionContext.Request.Properties.Add(TransactionId, transactionScope);
}
}
}
/// <summary>
/// Rollback or commit transaction.
/// </summary>
/// <param name="actionExecutedContext">The action executed context.</param>
public override void OnActionExecuted(HttpActionExecutedContext actionExecutedContext)
{
if (actionExecutedContext.Request.Properties.Keys.Contains(TransactionId))
{
var transactionScope = actionExecutedContext.Request.Properties[TransactionId] as TransactionScope;
if (transactionScope != null)
{
if (actionExecutedContext.Exception != null)
{
Transaction.Current.Rollback();
}
else
{
transactionScope.Complete();
}
transactionScope.Dispose();
actionExecutedContext.Request.Properties[TransactionId] = null;
}
}
}
}
So if any exception occurs during this call (api/Test/Transaction/Commit) inside that transaction scope (either in firt api or second api) all the database change done by the both api will be rolled back. This is working fine locally. As locally we get support of MSDTC. But in Azure there is no MSDTC support. In azure we get support from Elastic transaction. Because of this when we are trying to fetch propagation token of the transaction from the first server we are getting exception. So when we try to execute bellow code
var transaction = TransactionInterop.GetTransactionFromTransmitterPropagationToken(transactionToken);
We are getting exception with message "Value does not fall within the expected range". This post saying that this method would require promotion to MSDTC by System.Transactions, but for elastic transaction how we will make it work? For elastic transaction we need to marshal transaction into propagation token. How to do this? Looking for the solution.
Elastic Transactions are designed to allow transactions across Azure SQL Database and Azure SQL Managed Instance from a single .net application in Azure.
It is not built for distributing transactions across clients.
"Only client-coordinated transactions from a .NET application are supported"
https://learn.microsoft.com/en-us/azure/azure-sql/database/elastic-transactions-overview
I have a ASP.NET Web API application with supporting Azure Web Job with functions that are triggered by messages added to a storage queue by the API's controllers. Testing the Web API is simple enough using OWIN but how do I test the web jobs?
Do I run a console app in memory in the test runner? Execute the function directly (that wouldn't be a proper integration test though)? It is a continious job so the app doesn't exit. To make matters worse Azure Web Job-functions are void so there's no output to assert.
There is no need to run console app in memory. You can run JobHost in the memory of your integration test.
var host = new JobHost();
You could use host.Call() or host.RunAndBlock(). You would need to point to Azure storage account as webjobs are not supported in localhost.
It depends on what your function is doing, but you could manually add a message to a queue, add a blob or whatever. You could assert by querying the storage where your webjob executed result, etc.
While #boris-lipschitz is correct, when your job is continious (as op says it is), you can't do anything after calling host.RunAndBlock().
However, if you run the host in a separate thread, you can continue with the test as desired. Although, you have to do some kind of polling in the end of the test to know when the job has run.
Example
Function to be tested (A simple copy from one blob to another, triggered by created blob):
public void CopyBlob(
[BlobTrigger("input/{name}")] TextReader input,
[Blob("output/{name}")] out string output)
{
output = input.ReadToEnd();
}
Test function:
[Test]
public void CopyBlobTest()
{
var blobClient = GetBlobClient("UseDevelopmentStorage=true;");
//Start host in separate thread
var thread = new Thread(() =>
{
Thread.CurrentThread.IsBackground = true;
var host = new JobHost();
host.RunAndBlock();
});
thread.Start();
//Trigger job by writing some content to a blob
using (var stream = new MemoryStream())
using (var stringWriter = new StreamWriter(stream))
{
stringWriter.Write("TestContent");
stringWriter.Flush();
stream.Seek(0, SeekOrigin.Begin);
blobClient.UploadStream("input", "blobName", stream);
}
//Check every second for up to 20 seconds, to see if blob have been created in output and assert content if it has
var maxTries = 20;
while (maxTries-- > 0)
{
if (!blobClient.Exists("output", "blobName"))
{
Thread.Sleep(1000);
continue;
}
using (var stream = blobClient.OpenRead("output", "blobName"))
using (var streamReader = new StreamReader(stream))
{
Assert.AreEqual("TestContent", streamReader.ReadToEnd());
}
break;
}
}
I've been able to simulate this really easily by simply doing the following, and it seems to work fine for me:
private JobHost _webJob;
[OneTimeSetUp]
public void StartupFixture()
{
_webJob = Program.GetHost();
_webJob.Start();
}
[OneTimeTearDown]
public void TearDownFixture()
{
_webJob?.Stop();
}
Where the WebJob Code looks like:
public class Program
{
public static void Main()
{
var host = GetHost();
host.RunAndBlock();
}
public static JobHost GetHost()
{
...
}
}
I want to Initialize a global Dictionary from Database in my web Api. Do i need to inject my DBContext in Global.Asax or Owin Startup. Any example would be much appreciated.
Any kind initialization purposes can be made in your custom defined OWIN Startup class class, like this:
using Microsoft.Owin;
using Microsoft.Owin.Security.OAuth;
using Owin;
using System;
[assembly: OwinStartup(typeof(WebAPIRestWithNest.Startup))]
namespace YourNamespace
{
public class Startup
{
public Dictionary<string, string> Table {get; private set;}
public void Configuration(IAppBuilder app)
{
// token generation
app.UseOAuthAuthorizationServer(new OAuthAuthorizationServerOptions
{
AllowInsecureHttp = false,
TokenEndpointPath = new PathString("/token"),
AccessTokenExpireTimeSpan = TimeSpan.FromHours(8),
Provider = new SimpleAuthorizationServerProvider()
});
// token consumption
app.UseOAuthBearerAuthentication(new OAuthBearerAuthenticationOptions());
app.UseWebApi(WebApiConfig.Register());
Table = ... Connect from DB and fill your table logic ...
}
}
}
After that you can use your Startup.Table property from your application.
In general, it is bad practice to access objects using static field in the asp.net applications because this may lead to bugs that are hardly detected and reproduced: especially this is true for non-immutable/not-thread-safe objects like Dictionary.
I assume you want to cache some DB data in memory to avoid excessive SQL queries. It is good idea to use standard asp.net caching for this purpose:
public IDictionary GetDict() {
var dict = HttpRuntime.Cache.Get("uniqueCacheKey") as IDictionary;
if (pvtData==null) {
dict = doLoadDictionaryFromDB(); // your code that loads data from DB
HttpRuntime.Cache.Add(cacheKey, dict,
null, Cache.NoAbsoluteExpiration,
new TimeSpan(0,5,0), // cache at least for 5 minutes after last access
CacheItemPriority.Normal, null);
}
return dict;
}
This approach allows you to select appropriate expiration policy (without the need to reinventing the wheel with static dictionary).
If you still want to use static dictionary, you can populate it on the application start (global.asax):
void Application_Start(object sender, EventArgs e)
{
// your code that initializes dictionary with data from DB
}
I have successfully implemented Azure Redis Cache using the Microsoft RedisOutputCacheProvider from NuGet which works as expected for general pages.
[ChildActionOnly]
[ChildActionOutputCache(CacheProfile.StaticQueryStringComponent)]
public ActionResult Show(int id)
{
// some code
}
However, I can't seem to get it to work for child actions. Prior to using Redis Cache, it was working using the default OutputCacheProvider.
Does anyone have any ideas, or is it simply a limitation?
Thanks in advance
In your Global.asax.cs, set a custom child action output cache that talks to Redis:
protected void Application_Start()
{
// Register Custom Memory Cache for Child Action Method Caching
OutputCacheAttribute.ChildActionCache = new CustomMemoryCache("My Cache");
}
This cache should derive from MemoryCache and implement the following members:
/// <summary>
/// A Custom MemoryCache Class.
/// </summary>
public class CustomMemoryCache : MemoryCache
{
public CustomMemoryCache(string name)
: base(name)
{
}
public override bool Add(string key, object value, DateTimeOffset absoluteExpiration, string regionName = null)
{
// Do your custom caching here, in my example I'll use standard Http Caching
HttpContext.Current.Cache.Add(key, value, null, absoluteExpiration.DateTime,
System.Web.Caching.Cache.NoSlidingExpiration, System.Web.Caching.CacheItemPriority.Normal, null);
return true;
}
public override object Get(string key, string regionName = null)
{
// Do your custom caching here, in my example I'll use standard Http Caching
return HttpContext.Current.Cache.Get(key);
}
}
More info on my blog post
I have an MVC application in which I have a controller that receives data from the user and then uploads a file to Azure blob storage. The application is using Unity IoC to handle dependency injection.
During the workflow I have isolated the following code as demonstrating the problem
public class MvcController : Controller
{
private IDependencyResolver _dependencyResolver;
public MvcController() : this(DependencyResolver.Current)
{
}
public MvcController(IDependencyResolver dependencyResolver)
{
this._dependencyResolver = dependencyResolver;
}
public GetService<T>()
{
T resolved = _dependencyResolver.GetService<T>()
if (resolved == null)
throw new Exception(string.Format("Dependency resolver does not contain service of type {0}", typeof(T).Name));
return resolved;
}
}
public class MyController : MvcController
{
[NoAsyncTimeout]
public async Task<ActionResult> SaveFileAsync(/* A bunch of arguments */)
{
/* A bunch of code */
//This line gets a concrete instance from HttpContext.Current successfully...
IMyObject o = GetService<IMyObject>();
await SaveFileToAzure(/* A bunch of parameters */);
.
.
/* Sometime later */
Method2(/* A bunch of parameters */);
}
private Method2(/* A bunch of parameters */)
{
//This line fails because HttpContext.Current is null
IMyObject o = GetService<IMyObject>();
/* A bunch of other code */
}
private async Task SaveFileToAzure(/* A bunch of parameters */)
{
//Grab a blob container to store the file data...
CloudBlobContainer blobContainer = GetBlobContainer();
ICloudBlob blob = blobContainer.GetBlockBlobReference(somePath);
Stream dataStream = GetData();
System.Threading.CancellationToken cancelToken = GetCancellationToken();
//All calls to DependencyResolver.GetService<T>() after this line of code fail...
response = await blob.UploadStreamAsync(dataStream, cancelToken);
/* A bunch of other code */
}
}
Unity has a registration for my object:
container.RegisterType<IMyObject, MyObject>(new HttpLifetimeManager());
My lifetime manager is defined as follows:
public sealed class HttpRequestLifetimeManager : LifetimeManager
{
public Guid Key { get; private set; }
public HttpRequestLifetimeManager()
{
this.Key = Guid.NewGuid();
}
public override object GetValue()
{
return HttpContext.Current.Items[(object)this.Key];
}
public override void SetValue(object newValue)
{
HttpContext.Current.Items[(object)this.Key] = newValue;
}
public override void RemoveValue()
{
HttpContext.Current.Items.Remove((object)this.Key);
}
}
Nothing complicated.
Stepping into the HttpRequestLifetimeManager on the failing GetService() calls shows that after the UploadStreamAsync() call HttpContext.Current is null...
Has anyone else come across this problem? If so, is this a bug? Is this expected behaviour? Am I doing something out of the ordinary? What should I do to resolve it?
I can hack around it by storing a reference to HttpContext.Current prior to the offending call and restoring it after, but that doesn't seem like the right approach.
Any ideas?
To echo #Joachim - http context may not be available to your async thread. Compare the current thread id where you can see httpcontext is available, to the thread id where you can see that it isn't - i'm assuming you will see they are 2 different threads. If my assumption is correct this may be a sign that your main thread (the one with httpcontext) does not have a "synchronizationcontext". (you can see http://blogs.msdn.com/b/pfxteam/archive/2012/01/20/10259049.aspx for more details of how that works) If so, it may mean that the code immediately after your await statement is actually not running on the same thread as the code prior to the await statement! So from your perspective, one moment you have http context and the next you don't because execution has actually been switched to another thread! You should probably look at implementing / setting a synchronizationcontext on your main thread if that's the case and then control will be returned to your original thread with http context and that should fix your problem, or alternatively you could retrieve your object from http context on the original thread and find a way to pass it as a parameter to the async method/s so that they don't need to access http context to get their state.