Testing a Multipart file upload Azure Function - azure

Question

So I have written a simple Azure Function (AF) that accepts (via Http Post method) an IFormCollection, loops through the file collection, pushes each file into an Azure Blob storage container and returns the url to each file.
The function itself works perfectly when I do a single file or multiple file post through Postman using the 'multipart/form-data' header. However when I try to post a file through an xUnit test, I get the following error:
System.IO.InvalidDataException : Multipart body length limit 16384 exceeded.
I have searched high and low for a solution, tried different things, namely;
Replicating the request object to be as close as possible to Postmans request.
Playing around with the 'boundary' in the header.
Setting 'RequestFormLimits' on the function.
None of these have helped so far.
The details are the project are as follows:
Azure Function v3: targeting .netcoreapp3.1
Startup.cs
public class Startup : FunctionsStartup
{
public IConfiguration Configuration { get; private set; }
public override void Configure(IFunctionsHostBuilder builder)
{
var x = builder;
InitializeConfiguration(builder);
builder.Services.AddSingleton(Configuration.Get<UploadImagesAppSettings>());
builder.Services.AddLogging();
builder.Services.AddSingleton<IBlobService,BlobService>();
}
private void InitializeConfiguration(IFunctionsHostBuilder builder)
{
var executionContextOptions = builder
.Services
.BuildServiceProvider()
.GetService<IOptions<ExecutionContextOptions>>()
.Value;
Configuration = new ConfigurationBuilder()
.SetBasePath(executionContextOptions.AppDirectory)
.AddJsonFile("appsettings.json")
.AddJsonFile("appsettings.Development.json", optional: true)
.AddEnvironmentVariables()
.Build();
}
}
UploadImages.cs
public class UploadImages
{
private readonly IBlobService BlobService;
public UploadImages(IBlobService blobService)
{
BlobService = blobService;
}
[FunctionName("UploadImages")]
[RequestFormLimits(ValueLengthLimit = int.MaxValue,
MultipartBodyLengthLimit = 60000000, ValueCountLimit = 10)]
public async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = "images")] HttpRequest req)
{
List<Uri> returnUris = new List<Uri>();
if (req.ContentLength == 0)
{
string badResponseMessage = $"Request has no content";
return new BadRequestObjectResult(badResponseMessage);
}
if (req.ContentType.Contains("multipart/form-data") && req.Form.Files.Count > 0)
{
foreach (var file in req.Form.Files)
{
if (!file.IsValidImage())
{
string badResponseMessage = $"{file.FileName} is not a valid/accepted Image file";
return new BadRequestObjectResult(badResponseMessage);
}
var uri = await BlobService.CreateBlobAsync(file);
if (uri == null)
{
return new ObjectResult($"Could not blob the file {file.FileName}.");
}
returnUris.Add(uri);
}
}
if (!returnUris.Any())
{
return new NoContentResult();
}
return new OkObjectResult(returnUris);
}
}
Exception Thrown:
The below exception is thrown at the second if statement above, when it tries to process req.Form.Files.Count > 0, i.e.
if (req.ContentType.Contains("multipart/form-data") && req.Form.Files.Count > 0) {}
Message:
System.IO.InvalidDataException : Multipart body length limit 16384 exceeded.
Stack Trace:
MultipartReaderStream.UpdatePosition(Int32 read)
MultipartReaderStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool`1 bytePool, Nullable`1 limit, CancellationToken cancellationToken)
MultipartReader.ReadNextSectionAsync(CancellationToken cancellationToken)
FormFeature.InnerReadFormAsync(CancellationToken cancellationToken)
FormFeature.ReadForm()
DefaultHttpRequest.get_Form()
UploadImages.Run(HttpRequest req) line 42
UploadImagesTests.HttpTrigger_ShouldReturnListOfUploadedUris(String fileNames)
xUnit Test Project: targeting .netcoreapp3.1
Over to the xUnit Test project, basically I am trying to write an integration test. The project references the AF project and has the following classes:
TestHost.cs
public class TestHost
{
public TestHost()
{
var startup = new TestStartup();
var host = new HostBuilder()
.ConfigureWebJobs(startup.Configure)
.ConfigureServices(ReplaceTestOverrides)
.Build();
ServiceProvider = host.Services;
}
public IServiceProvider ServiceProvider { get; }
private void ReplaceTestOverrides(IServiceCollection services)
{
// services.Replace(new ServiceDescriptor(typeof(ServiceToReplace), testImplementation));
}
private class TestStartup : Startup
{
public override void Configure(IFunctionsHostBuilder builder)
{
SetExecutionContextOptions(builder);
base.Configure(builder);
}
private static void SetExecutionContextOptions(IFunctionsHostBuilder builder)
{
builder.Services.Configure<ExecutionContextOptions>(o => o.AppDirectory = Directory.GetCurrentDirectory());
}
}
}
TestCollection.cs
[CollectionDefinition(Name)]
public class TestCollection : ICollectionFixture<TestHost>
{
public const string Name = nameof(TestCollection);
}
HttpRequestFactory.cs: To create Http Post Request
public static class HttpRequestFactory
{
public static DefaultHttpRequest Create(string method, string contentType, Stream body)
{
var request = new DefaultHttpRequest(new DefaultHttpContext());
var contentTypeWithBoundary = new MediaTypeHeaderValue(contentType)
{
Boundary = $"----------------------------{DateTime.Now.Ticks.ToString("x")}"
};
var boundary = MultipartRequestHelper.GetBoundary(
contentTypeWithBoundary, (int)body.Length);
request.Method = method;
request.Headers.Add("Cache-Control", "no-cache");
request.Headers.Add("Content-Type", contentType);
request.ContentType = $"{contentType}; boundary={boundary}";
request.ContentLength = body.Length;
request.Body = body;
return request;
}
private static string GetBoundary(MediaTypeHeaderValue contentType, int lengthLimit)
{
var boundary = HeaderUtilities.RemoveQuotes(contentType.Boundary);
if (string.IsNullOrWhiteSpace(boundary.Value))
{
throw new InvalidDataException("Missing content-type boundary.");
}
if (boundary.Length > lengthLimit)
{
throw new InvalidDataException(
$"Multipart boundary length limit {lengthLimit} exceeded.");
}
return boundary.Value;
}
}
The MultipartRequestHelper.cs class is available here
And Finally the Test class:
[Collection(TestCollection.Name)]
public class UploadImagesTests
{
readonly UploadImages UploadImagesFunction;
public UploadImagesTests(TestHost testHost)
{
UploadImagesFunction = new UploadImages(testHost.ServiceProvider.GetRequiredService<IBlobService>());
}
[Theory]
[InlineData("testfile2.jpg")]
public async void HttpTrigger_ShouldReturnListOfUploadedUris(string fileNames)
{
var formFile = GetFormFile(fileNames);
var fileStream = formFile.OpenReadStream();
var request = HttpRequestFactory.Create("POST", "multipart/form-data", fileStream);
var response = (OkObjectResult)await UploadImagesFunction.Run(request);
//fileStream.Close();
Assert.True(response.StatusCode == StatusCodes.Status200OK);
}
private static IFormFile GetFormFile(string fileName)
{
string fileExtension = fileName.Substring(fileName.IndexOf('.') + 1);
string fileNameandPath = GetFilePathWithName(fileName);
IFormFile formFile;
var stream = File.OpenRead(fileNameandPath);
switch (fileExtension)
{
case "jpg":
formFile = new FormFile(stream, 0, stream.Length,
fileName.Substring(0, fileName.IndexOf('.')),
fileName)
{
Headers = new HeaderDictionary(),
ContentType = "image/jpeg"
};
break;
case "png":
formFile = new FormFile(stream, 0, stream.Length,
fileName.Substring(0, fileName.IndexOf('.')),
fileName)
{
Headers = new HeaderDictionary(),
ContentType = "image/png"
};
break;
case "pdf":
formFile = new FormFile(stream, 0, stream.Length,
fileName.Substring(0, fileName.IndexOf('.')),
fileName)
{
Headers = new HeaderDictionary(),
ContentType = "application/pdf"
};
break;
default:
formFile = null;
break;
}
return formFile;
}
private static string GetFilePathWithName(string filename)
{
var outputFolder = Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location);
return $"{outputFolder.Substring(0, outputFolder.IndexOf("bin"))}testfiles\\{filename}";
}
}
The test seems to be hitting the function and req.ContentLength does have a value. Considering this, could it have something to do with the way the File Streams are being managed? Perhaps not the right way?
Any inputs on this would be greatly appreciated.
Thanks
UPDATE 1
As per this post, I have also tried setting the ValueLengthLimit and MultipartBodyLengthLimit in the Startup of the Azure Function and/or the Test Project as opposed to attributes on the Azure Function. The exception then changed to:
"The inner stream position has changed unexpectedly"
Following this, I then set the fileStream position in the test project to SeekOrigin.Begin. I started getting the same error:
"Multipart body length limit 16384 exceeded."

Answer 1

It took me a 50km bike ride and a good nights sleep but I finally figured this one out :-).
The Azure function (AF) accepts an HttpRequest object as a parameter with the name of 'req' i.e.
public async Task Run(
[HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = "images")] HttpRequest req)
The hierarchy of the files object in the HttpRequest object (along with the parameter names) is as follows:
HttpRequest -> req
FormCollection -> Form
FormFileCollection -> Files
This is what the AF accepts and one would access the files collection by using req.Form.Files
In my test case, instead of posting a FormCollection object, I was trying to post a Stream of a file to the Azure Function.
var formFile = GetFormFile(fileNames);
var fileStream = formFile.OpenReadStream();
var request = HttpRequestFactory.Create("POST", "multipart/form-data", fileStream);
As a result of this, req.Form had a Stream value that it could not interpret and the req.Form.Files was raising an exception.
In order to rectify this, I had to do the following:
Revert all changes made as part of UPDATE 1. This means that I removed the 'RequestFormLimits' settings from the Startup file and left them as attributes on the functions Run method.
Instantiate a FormFileCollection object and add the IFormFile to it
Instantiate a FormCollection object using this FormFileCollection as a parameter.
Add the FormCollection to the request object.
To achieve the above, I had to make the following changes in code.
Change 'Create' method in the HttpRequestFactory
public static DefaultHttpRequest Create(string method, string contentType, FormCollection formCollection)
{
var request = new DefaultHttpRequest(new DefaultHttpContext());
var boundary = $"----------------------------{DateTime.Now.Ticks.ToString("x")}";
request.Method = method;
request.Headers.Add("Cache-Control", "no-cache");
request.Headers.Add("Content-Type", contentType);
request.ContentType = $"{contentType}; boundary={boundary}";
request.Form = formCollection;
return request;
}
Add a private static GetFormFiles() method
I wrote an additional GetFormFiles() method that calls the existing GetFormFile() method, instantiate a FormFileCollection object and add the IFormFile to it. This method in turn returns a FormFileCollection.
private static FormFileCollection GetFormFiles(string fileNames)
{
var formFileCollection = new FormFileCollection();
foreach (var file in fileNames.Split(','))
{
formFileCollection.Add(GetFormFile(file));
}
return formFileCollection;
}
Change the Testmethod
The test method calls the GetFormFiles() to get a FormFileCollection then
instantiates a FormCollection object using this FormFileCollection as a parameter and then passes the FormCollection object as a parameter to the HttpRequest object instead of passing a Stream.
[Theory]
[InlineData("testfile2.jpg")]
public async void HttpTrigger_ShouldReturnListOfUploadedUris(string fileNames)
{
var formFiles = GetFormFiles(fileNames);
var formCollection = new FormCollection(null, formFiles);
var request = HttpRequestFactory.Create("POST", "multipart/form-data", formCollection);
var response = (OkObjectResult) await UploadImagesFunction.Run(request);
Assert.True(response.StatusCode == StatusCodes.Status200OK);
}
So in the end the issue was not really with the 'RequestFormLimits' but rather with the type of data I was submitting in the POST message.
I hope this answer provides a different perspective to someone that comes across the same error message.
Cheers.