Azure Pipelines Blob storage URL's - azure

We use Azure DevOps for building and deploying our applications and services in Azure. Right now we are also switching our on premise deployments to Azure DevOps as well.
Unfortunately we have one issue with our servers which are on the one hand protected by a firewall (outgoing internet traffic is mostly blocked) and on the other hand are part of Azure DevOps deployment groups. As soon as we create a new release and start the deployment to a stage the artifact download process fails since the server is not permitted to download the artifacts from the blob storage where the artifacts are stored.
Warning, https://elgvsblobprodsu6weus10.blob.core.windows.net/dbcXXXXXXX/XXXX
Try 1/5, retryable exception caught. Retrying in 00:00:01. Details:
2019-10-21T19:59:02.7071549Z No LastRequestResponse on exception HttpRequestException: An error occurred while sending the request. System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: Error 12029 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'A connection with the server could not be established'.
2019-10-21T19:59:02.7071940Z at System.Threading.Tasks.RendezvousAwaitable`1.GetResult()
2019-10-21T19:59:02.7072055Z at System.Net.Http.WinHttpHandler.StartRequest(WinHttpRequestState state)
2019-10-21T19:59:02.7072112Z --- End of inner exception stack trace ---
2019-10-21T19:59:02.7074791Z at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
2019-10-21T19:59:02.7074976Z at Microsoft.VisualStudio.Services.Common.TaskCancellationExtensions.EnforceCancellation[TResult](Task`1 task, CancellationToken cancellationToken, Func`1 makeMessage, String file, String member, Int32 line)
2019-10-21T19:59:02.7075133Z at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreHttpClient.<>c__DisplayClass56_0.<<GetRedirectResponseAsync>b__0>d.MoveNext()
2019-10-21T19:59:02.7075213Z --- End of stack trace from previous location where exception was thrown ---
2019-10-21T19:59:02.7075312Z at Microsoft.VisualStudio.Services.Content.Common.AsyncHttpRetryHelper`1.InvokeAsync(CancellationToken cancellationToken)
So is there any list of endpoints which are used for storing Azure Pipeline artifacts that can be added to our firewall policy, do we have to exclude *.blob.core.windows.net in general or is there any different approach?
Unfortunately the Microsoft docs do only provide a list of Urls which are required for establishing a connection with Azure DevOps in general, but not with the artifacts store.

Is there any list of endpoints which are used for storing Azure
Pipeline artifacts that can be added to our firewall policy.
Of course yes. We store these data in the corresponding region's Azure Data center.
Download the Json file in this link, the file list the IP Address range of different regions. But, here is one important thing you need pay attention to. For security, we update these IP address range regularly. (Also, publish it to users regularly)
If you choose this approach to achieve you want, you need to download the JSON file regularly, then make necessary updates in your access rules:
[!IMPORTANT] IP address ranges for Azure services can change, and
updates are published weekly. Download the JSON file regularly, and
make necessary updates in your access rules. If your scenario involves
configuring network security group rules in an Azure virtual network
to access Azure Container Registry, use the AzureContainerRegistry
service tag instead.
See this doc: Allow access by IP address range.
But, if you think this method is inconvenient for you and your team. Another unique method is the one you are using now: allow access to all Azure blob storage accounts using the wildcard *.blob.core.windows.net.

Related

Azure Traffic Manager errors when adding 2nd App Service as endpoint

We have added an app service as an endpoint to Azure Traffic Manager, and everything is working fine. However, when trying to add a second app service it fails with the following error:
Some of the provided Azure Website endpoints are not valid: Traffic manager configuration unexpectedly failed in region 'uksouth' with exception: Microsoft.Web.Hosting.Administration.Client.GeomasterClientException: Call to geomaster failed!, HttpStatusCode=BadRequest, RequestId='cae63ca1-0a3d-4f87-bd8e-9b881186e114', Uri=https://ln1.geomaster.azurewebsites.windows.net:444/subscriptions/fe12301c-5b6f-45f7-a038-ce2d4dbeec94/providers/Microsoft.Web/verifyTrafficManagerConfiguration?api-version=2018-02-01, CorrelationId=06de79f7-a67a-4a0e-ac5f-f6db24d5f908 at Microsoft.Web.Hosting.Administration.Client.InterGeomasterClient.Send[P,R](HttpMethod verb, String path, String queryString, P payload, Boolean throwOnError) at Microsoft.Web.Hosting.Administration.Client.InterGeomasterClient.<>c__DisplayClass22_0`2.<Post>b__0() at Microsoft.Practices.EnterpriseLibrary.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func) at Microsoft.Web.Hosting.Administration.Client.RegionalToRegionalClient.VerifyAndRegisterTrafficManagerConfiguration(String subscriptionName, CsmTrafficManagerConfiguration csmTrafficManagerConfiguration) at Microsoft.Web.Hosting.Administration.GeoScale.Sql.SubscriptionController.ForwardVerifyAndRegisterApiCallToRegionalGeomaster(RESTApiMetricsTracker tracker, String location, String subscriptionName, String trafficManagerDomainName, String[] hostNamesForsitesInRegion, Boolean registerTrafficManagerDomainName, Boolean failIneligibleSites)
We seem to be able to add two different app services without error, even this particular one with a different app service without error. It seems to be this particular combination of app services that fails as if they are somehow incompatible?
Not sure if it's significant but it seems combinations with an old app service (i.e. created a couple of years ago doesn't work with a recently created app service) but adding two app services that have been created recently works OK.
This error could be because of using the free tier of Traffic Management. If you are using the free tier of the old app, then change the tier plan.
Alternatively:
If the two apps are running on standard tier and still the issue occurs, then it must be the location error mentioned in the exception. That is South. Make sure the regions of the apps running are compatible with all the services you want to use. Some of the services may not be enabled in all the availability zones.

azure function cannot access to on-premise network

I have an azure trigger function and I want the service to take the files from folder which is in on-premise then process them. The folder is that we only can access via VPN. The service has a VNET configuration for this but still cannot access the folder.
What should I do?
Please let me know why I am getting this error or do I need to change something in my process? Thanks in advance.
tried to ping the folder address, got this exception
System.Net.NetworkInformation.PingException: An exception occurred during a Ping request. ---> System.ComponentModel.Win32Exception (5): Access is denied. at System.Net.NetworkInformation.Ping.InitialiseIcmpHandle() at System.Net.NetworkInformation.Ping.DoSendPingCore(IPAddress address, Byte[] buffer, Int32 timeout, PingOptions options, Boolean isAsync) at System.Net.NetworkInformation.Ping.Send(IPAddress address, Int32 timeout, Byte[] buffer, PingOptions options) --- End of inner exception stack trace --- at System.Net.NetworkInformation.Ping.Send(IPAddress address, Int32 timeout, Byte[] buffer, PingOptions options) at System.Net.NetworkInformation.Ping.Send(String hostNameOrAddress, Int32 timeout, Byte[] buffer, PingOptions options) at ....
tried to access to the path, got the exception below
System.UnauthorizedAccessException: Access to the path '\\\xxxxxx\test' is denied. at System.IO.Enumeration.FileSystemEnumerator`1.CreateDirectoryHandle(String path, Boolean ignoreNotFound) at System.IO.Enumeration.FileSystemEnumerator`1.Init() at ....
You cannot initiate the connection from Azure Functions to on-prem, network connections can only be initiated by clients connecting to the Private endpoint, Service providers (in this case Azure Functions) do not have any routing configuration to initiate connections into service consumers. Connections can only be establish in a single direction.
If the onprem folder is a OneDrive folder you have the option of using the Microsoft Graph API to send to a webhook which could be your Function App. Details are here.
https://learn.microsoft.com/en-us/graph/webhooks
This is only one option with the limited information I have on your scenario.

Getting AuthorizationFailure (403) response on Azure Blob Storage via REST API with SAS Token

After a few days fighting with this issue I have decided to post it here in case someone can help out by giving me some pointers.
I have an Azure Storage Account with a Blob container with encrypted images. The images have been uploaded via PUT request (from a browser) to the Azure Blob Storage REST API, using SAS signature (generated by an Azure VM) and customer provided keys (x-ms-encryption headers). However, when trying to download the images from the Azure VM with a GET request (using curl), I get the following 403 "AuthorizationFailure" error:
<?xml version="1.0" encoding="utf-8”?>
<Error>
<Code>AuthorizationFailure</Code>
<Message>
This request is not authorized to perform this operation.
RequestId:1b203db6-c01e-0013-1553-6adb9b000000
Time:2020-08-04T11:33:42.9494992Z
</Message>
</Error>
Funny thing though, when I perform exactly the same GET request (exactly the same headers) with curl, Postman or the browser from my own local machine and even from other Azure VMs located in different resource groups and different virtual networks, it works as expected and I can successfully download the encrypted image. This confirms that the SAS signature and the headers passed are correctly constructed.
I have reviewed all the configuration of the VM, Network Security Group, Virtual Network and Storage Account in the Azure Portal and compared it with the other VMs that successfully download the images, and all the settings are exactly the same except for:
In the VM that fails to download the image, I'm authorized as a contributor for that VM's resource group only, while in the VM that works I'm the owner of the account.
In the account where I am the owner, Azure has created a new resource group called NetworkWatcherRG, which seems to have no resources attached to it. In the VM that has the problem I do not have access to that resource group, so I ignore whether it has been created by Azure or not, and whether it has any impact on the problem I'm facing.
I would really appreciate any ideas or suggestions on what the issue might be.

Why do I see a FunctionIndexingException when creating a QueueTrigger WebJob Function?

I created a function like this
public static Task HandleStorageQueueMessageAsync(
[QueueTrigger("%QueueName%", Connection = "%ConnectionStringName%")] string body,
TextWriter logger)
{
if (logger == null)
{
throw new ArgumentNullException(nameof(logger));
}
logger.WriteLine(body);
return Task.CompletedTask;
}
The queue name and the connection string name come from my configuration that has an INameResolver to get the values. The connection string itself I put from my secret store into the app config at app start. If the connection string is a normal storage connection string granting all permissions for the whole account, the method works like expected.
However, in my scenario I am getting an SAS from a partner team that only offers read access to a single queue. I created a storage connection string from that which looks similar like
QueueEndpoint=https://accountname.queue.core.windows.net;SharedAccessSignature=st=2017-09-24T07%3A29%3A00Z&se=2019-09-25T07%3A29%3A00Z&sp=r&sv=2018-03-28&sig=token
(I tried successfully to connect using this connection string in Microsoft Azure Storage Explorer)
The queue name used in the QueueTrigger attribute is also gathered from the SAS
However, now I am getting the following exceptions
$exception {"Error indexing method 'Functions.HandleStorageQueueMessageAsync'"} Microsoft.Azure.WebJobs.Host.Indexers.FunctionIndexingException
InnerException {"No blob endpoint configured."} System.Exception {System.InvalidOperationException}
If you look at the connection string, you can see the exception is right. I did not configure the blob endpoint. However I also don't have access to it and neither do I want to use it. I'm using the storage account only for this QueueTrigger.
I am using Microsoft.Azure.WebJobs v2.2.0. Other dependencies prevent me from upgrading to a v3.x
What is the recommended way for consuming messages from a storage queue when only a SAS URI with read access to a single queue is available? If I am already on the right path, what do I need to do in order to get rid of the exception?
As you have seen, v2 WebJobs SDK requires access to blob endpoint as well. I am afraid it's by design, using connection string without full access like SAS is an improvement tracked but not realized yet.
Here are the permissions required by v2 SDK. It needs to get Blob Service properties(Blob,Service,Read) and Queue Metadata and process messages(Queue,Container&Object,Read&Process).
Queue Trigger is to get messages and delete them after processing, so SAS requires Process permission. It means the SAS string you got is not authorized correctly even if SDK doesn't require blob access.
You could ask partner team to generate SAS Connection String on Azure portal with minimum permissions above. If they can't provide blob access, v3 SDK seems an option to try.
But there are some problems 1. Other dependencies prevent updating as you mentioned 2. v3 SDK is based on .NET Core which means code changes can't be avoided. 3. v3 SDK document and samples are still under construction right now.
I was having a load of issues getting a SAS token to work for a QueueTrigger.
Not having blob included was my problem. Thanks Jerry!
Slightly newer screenshot (I need add also):

Azure Cloud Service (Classic) Roles do not start when I have Swashbuckle and ANY implementation of IOperationFilter

We have an asp.net web api project deployed to an Azure Cloud Service (classic) that has been running fine using Swashbuckle for almost a year. We configure it like so...
GlobalConfiguration.Configuration
.EnableSwagger(c =>
{
c.SingleApiVersion("v1", "PartnerAPI");
c.UseFullTypeNameInSchemaIds();
}).EnableSwaggerUi(c => { });
Recently we needed to tweak the swagger generated output by plugging in an IOperationFilter. However our Azure Cloud Service (Classic) instance will not start if we create a class that implements the IOperationFilter. We don't even try to configure Swagger to use it. Just the fact that there is a class that implements that interface in our solution causes the deploy to fail stating...
2016-12-29T16:10:26.1066042Z ##[error]BadRequest : Your role instances have recycled a number of times during an update or upgrade operation. This indicates that the new version of your service or the configuration settings you provided when configuring the service prevent the role instances from running. Verify your code does not throw unhandled exceptions and that your configuration settings are correct and then start another update or upgrade operation.
Some Notes:
Everything runs fine on my machine, directly and in the azure emulator
Everything runs fine on teammates machine same as above
The following message appears to be related in the event logs on the azure machine when I rdp into it.
File Server Resource Manager was unable to access the following file or volume: 'E:'. This file or volume might be locked by another application right now, or you might need to give Local System access to it.
Same problem in Swashbuckle versions 5 and 5.5
No new nuget packages or references to the project
Only a "using Swashbuckle.Swagger;' that was added to the SwaggerConfig.cs
The Azure Portal reports the following for the "Instance Status Message"...
[12/29T16:43Z]Failed to load role entrypoint. System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information. at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module) at System.Reflection.Assembly.GetTypes() at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.GetRoleEntryPoint(Assembly entryPointAssembly) --- End of inner exception stack trace --- at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.GetRoleEntryPoint(Assembly entryPointAssembly) at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.CreateRoleEntryPoint(RoleType roleTypeEnum) at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.InitializeRoleInternal(RoleType roleTypeEnum)' Last exit time: [2016/12/29, 16:43:59.525]. Last exit code: 0.

Resources