Service Fabric Default Publish Profile other than Local.xml - azure

Our company is developing our new applications using Service Fabric.
A common problem we have, multiple developers use queues, databases, storages that are on remote servers, and each one has different configuration for this, all the settings are stored on ApplicationParameters file per environment, for Local development there is a single Local.5Node.xml. It is very common developers checkin their credentials and overwrite others when we get latest version of these files.
I'm trying to customize the ServiceFabric deployment script 'Deploy-FabricApplication.ps1' to use a custom PublishProfile depending on windows credentials of logged user. I can achieve that updating the deployment file, it works well when we deploy using the publish, but it seems that the default behavior of ServiceFabric when we hit F5(debug) is overwrite the parameters with a specific Local.5Node.xml application parameters.
I explored all service fabric .ps1 files and couldn't find where this is defined. I guess this is defined on .targets file, so I don't know how can I avoid this default behaviour.
Is there any other approach to use custom PublishProfiles on local development machines other than Local.5Node.xml?

I actually just ran into this with setting up some team specific environments. I borrowed information from the following sources:
Web Config Transformation
Replace String In File With MSBUILD
I added multiple parameters files based on what was needed for the different teams. Each one containing their specific resource settings.
I also added a Local.1Node.Template.xml and Local.5Node.Template.xml. I even removed the Local.1Node.xml and Local.5Node.xml from source control and set them up to be ignored while leaving them in the projects so that Visual Studio doesn't think they are truly missing. The contents of the 1Node (5Node is the same except for replacing 1Node with 5Node) are as follows:
<?xml version="1.0" encoding="utf-8"?>
<PublishProfile xmlns="http://schemas.microsoft.com/2015/05/fabrictools">
<ClusterConnectionParameters />
<ApplicationParameterFile Path="..\ApplicationParameters\Local.1Node.$(Configuration).xml" />
</PublishProfile>
I then edited the sfproj file for the Service Fabric project to contain the following MSBuild Task and Target:
<UsingTask TaskName="ReplaceFileText" TaskFactory="CodeTaskFactory" AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.v4.0.dll">
<ParameterGroup>
<InputFilename ParameterType="System.String" Required="true" />
<OutputFilename ParameterType="System.String" Required="true" />
<MatchExpression ParameterType="System.String" Required="true" />
<ReplacementText ParameterType="System.String" Required="true" />
</ParameterGroup>
<Task>
<Reference Include="System.Core" />
<Using Namespace="System" />
<Using Namespace="System.IO" />
<Using Namespace="System.Text.RegularExpressions" />
<Code Type="Fragment" Language="cs">
<![CDATA[
File.WriteAllText(
OutputFilename,
Regex.Replace(File.ReadAllText(InputFilename), MatchExpression, ReplacementText)
);
]]>
</Code>
</Task>
</UsingTask>
<Target Name="UpdateProfile" BeforeTargets="UpdateServiceFabricApplicationManifest">
<ReplaceFileText InputFilename="PublishProfiles\Local.1Node.Template.xml" OutputFilename="PublishProfiles\Local.1Node.xml" MatchExpression="\$\(Configuration\)" ReplacementText="$(Configuration)" />
<ReplaceFileText InputFilename="PublishProfiles\Local.5Node.Template.xml" OutputFilename="PublishProfiles\Local.5Node.xml" MatchExpression="\$\(Configuration\)" ReplacementText="$(Configuration)" />
</Target>
The final step was to setup the different Build Configurations for the teams. I created a FT1-Debug through FT6-Debug based on the Debug configuration in the Service Fabric Service project and the Service Fabric Host project. I left all of my other projects alone.
At this point everyone on the different teams can debug locally with the correct configuration for the cluster they are doing work in just by changing the Build Configuration and pressing F5 to debug.

The VS extension for Service Fabric define a hard coded publish profile when we debug the solution using Visual Studio, it check how many nodes my cluster has and create a link to Local.5Node.xml and Local.1Node.xml depending how many nodes my cluster have.
To accomplish the same results, we end up using custom Application Parameters per developer and each developer update the Publish Profile (Local.5node.xml) to point to their respective Application parameter files.
It is not automated as the required feature, but can solve the main problem.

Related

Asp.Net EF Code First continuous integration with VSTS - Azure Sql Db not deploying

I am having trouble getting migrations to run to create / update my Sql Db hosted on Azure. They run fine locally to my LocalDb but seem to be completely ignored when releasing to Azure.
Details:
- Asp.Net MVC 5.2.3
- Entity Framework 6.1.3
- Visual Studio Team Services (online)
- Azure Web App & Azure SQL Db (under Imagine / Dreamspark subscription)
I created a different application that didn't use CI, but used the Publish feature within Visual Studio and it published fine and worked. I have since removed that web app & db from Azure since the subscription only allows one db.
I'm thinking it may be something I'm missing in setting up the build definition in VSTS, I'm just not sure what.
I have tried:
https://blogs.msdn.microsoft.com/webdev/2014/04/08/ef-code-first-migrations-deployment-to-an-azure-cloud-service/ (this did nothing for me)
https://www.benday.com/2016/11/07/using-tfs-build-to-deploy-entity-framework-database-migrations-with-migrate-exe/ (I received a build error that I didn't have a release definition in my bin folder)
... as well as a bunch of others (I don't have the windows open any longer or I'd link them too).
My current build steps include:
NuGet restore - NuGet Installer
Build solution - Visual Studio Build
Test Assemblies - Visual Studio Test
Publish symbols path - Index Sources & Publish Symbols
Publish Artifact - Publish Build Artifacts
In my local env, EF works when I Enable-Migrations, Add-Migration, Update-Database. I'm guessing none of the above steps do this so I need one or more steps. Ben Day's blog seems like it should work, but for some reason, it's not finding my bin/release file.
The actual application deploys to Azure just fine upon release of the above successful build definition. It's just that the database is completely ignored. Using SSMS, I check the Azure db, and while it exists, it has none of my tables or data.
What am I missing?
Thanks.
**Edit - found the simple item I was missing. I need to add the connection string for the db to the web app in Azure. It now has my initial tables, and stores data using those tables, but doesn't pick up migrations. So something I've done in the last two days got my initial tables out there. Now.. to remember what it was.
**2nd Edit - So, I think the tables made it into the release db before I fixed my connection string in VSTS. One of the things I tried was creating a separate database project. I think when I pushed that at some point, the tables in existence at that time also pushed to release, I just couldn't see them because I hadn't connected the application and db together? At any rate, EF migrations are still not being recognized. I tried Ben Day's suggestion again, but I"m getting the build error:
Unhandled Exception: System.IO.FileNotFoundException: Could not load file or assembly 'EntityFramework, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies. The system cannot find the file specified.
I am using EF 6.1.3
I ran into this as well. I was using Visual Studio to publish my Azure app service. After doing some research (https://learn.microsoft.com/en-us/aspnet/mvc/overview/getting-started/getting-started-with-ef-using-mvc/migrations-and-deployment-with-the-entity-framework-in-an-asp-net-mvc-application#deploy-to-azure), I came to learn that VS Publish transforms your web config when you check the box for "Update Database" in Publish setting, adding a connection string that refers to the database containing your _migrations table, as well as a "contexts" element under "entityFramework". You can see what a previous publish has done with your web.config by looking at "..obj\Release\InsertEFCodeFirstDeploy\transformed\web.config".
So to recreate this same action when using a build and release definition in VSTS, I added 2 XDT "insert" transforms in Web.Release.config in my service project. Mine looks like this:
<?xml version="1.0"?>
<!-- For more information on using Web.config transformation visit http://go.microsoft.com/fwlink/?LinkId=301874 -->
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<connectionStrings>
<add name="MS_TableConnectionString_DatabasePublish" connectionString="Data Source=(localdb)\MSSQLLocalDB;Initial Catalog=AzureStorageEmulatorDb45;Integrated Security=True;Connect Timeout=30;Encrypt=False;TrustServerCertificate=True"
providerName="System.Data.SqlClient" xdt:Transform="Insert" />
</connectionStrings>
<system.web>
<compilation xdt:Transform="RemoveAttributes(debug)" />
</system.web>
<entityFramework>
<contexts xdt:Transform="Insert">
<context type="helpmeshopService.Models.helpmeshopContext, helpmeshopService">
<databaseInitializer type="System.Data.Entity.MigrateDatabaseToLatestVersion`2[[helpmeshopService.Models.helpmeshopContext, helpmeshopService], [helpmeshopService.Migrations.Configuration, helpmeshopService]], EntityFramework, PublicKeyToken=b77a5c561934e089">
<parameters>
<parameter value="MS_TableConnectionString_DatabasePublish" />
</parameters>
</databaseInitializer>
</context>
</contexts>
</entityFramework>
</configuration>
Next, you will need to go to your App Service "Application Settings" in the Azure portal and add a connection string with name, "MS_TableConnectionString_DatabasePublish", and set its value to the same value as for "MS_TableConnectionString" that should already exist in your settings.

How to override web.config values in custom section in Azure Web App?

It is possible in Azure Web App to override web.config AppSettings section easily. E.g. if I have the following web.config:
<appSettings>
<add key="AllowedCORSOrigin" value="http://localhost:26674"/>
</appSettings>
I can override it in the app settings UI in the portal like that:
I have also a custom section in the web.config:
<AdWordsApi>
<add key="OAuth2RefreshToken" value="TOKEN" />
</AdWordsApi>
Is it possible to override it somehow as well? I have tried AdWordsApi.OAuth2RefreshToken and AdWordsApi:OAuth2RefreshToken, but that does not work that easily.
P.S. It's interesting to know if it's possible with other custom sections like e.g if I want another authentication mode on the server.
<system.web>
<authentication mode="None" />
</system.web>
Short answer is that it is not possible.
The mechanism you describes only works with App Settings and Connection Strings. High level, the way it works is:
Your Azure App Settings become environment variables
At runtime, a special module sets those dynamically in the .NET config system. Note that the physical web.config is never modified.
But it would be hard to make such mechanism work on arbitrary config sections, as those could not be dynamically affected without modifying the physical file.
If you are using Visual Studio use web.config transformations to change configuration settings depending on whether you are running locally or deploying to Azure:
How to Transform Web.config
In simple terms you create one more more build configurations (typically Debug & Release). In your Visual Studio solution right-click on your existing web.config file and click "Add Config Transform", this will create a Web.Debug.Config and Web.Release.Config file which you can then customise with specific settings depending on the environment. Link this with your Azure build configuration and you can then have any combination of settings for local and remote deployment.
This is old but leaving this reference to how to use Azure Resource Manager to potentially solve this.
You can transform the values by the listed in VSTS by doing the following steps in App.Release.config:-
Add xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform" in configuration section
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
</configuration>
Add xdt:Transform="Replace" in custom section like below
<AdWordsApi xdt:Transform="Replace">
<add key="OAuth2RefreshToken" value="TOKEN" />
</AdWordsApi>
Create variable token in the release pipeline e.g OAuth2RefreshToken
Then in file config use it as following
<AdWordsApi xdt:Transform="Replace">
<add key="OAuth2RefreshToken" value="#{OAuth2RefreshToken}#" />
</AdWordsApi>
If you are adding any in web.config --> Appsetting, you can overirde it in Azure App Service using variable prefix
Key Name: APPSETTING_AllowedCORSOrigin
Value: http://localhost:26674
https://learn.microsoft.com/en-us/azure/app-service/reference-app-settings?tabs=kudu%2Cdotnet#variable-prefixes

Register custom culture in Windows Azure

I am currently developing an ASP.NET MVC 4 app for the African market and was hoping to register a custom culture using the steps detailed in the following link http://msdn.microsoft.com/en-gb/library/system.globalization.cultureandregioninfobuilder.register(v=vs.110).aspx. Most of my target countries are not in the pre-installed cultures so it sounds like I need to register these cultures. Problem is, my console app for doing the registration will need admin previlidges to complete the culture registration. I am presuming windows azure does not allow developers admin control of the cloud service environment.
Question: What is the best way to register a custom culture in Windows Azure without admin previlidges. Apparently there's a way to do this on Framework 2.0 using the cultureandregioninfobuilder.compile method but this is not a supported method. Is there a better solution? Don't want to have to maintain different project solutions for each culture just so I can support different languages.
Thanks in advance.
You could create a startup task that runs with elevated priviledges, and also run your application on a limited context. Your service configuration file should look like this:
<ServiceDefinition name="MyCloudService" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition" schemaVersion="2012-10.1.8">
<WebRole name="MyWebRole" vmsize="Small">
<Runtime executionContext="limited">
</Runtime>
<Startup>
<Task executionContext="elevated" commandLine="scripts\SetupCulture.cmd" taskType="simple">
<Environment>
<Variable name="CULTURE">
<RoleInstanceValue xpath="/RoleEnvironment/CurrentInstance/ConfigurationSettings/ConfigurationSetting[#name='Culture']/#value" />
</Variable>
</Environment>
</Task>
</Startup>
...
</WebRole>
</ServiceDefinition>
Note with this you can specify the desired culture in the service configuration file. Finally you can use the environment variable "CULTURE" in your .cmd file to use it as parameter to a .NET executable doing the job.

IIS 7 Configuration Paths

I am trying to make my c++ ahadmin application compatible to IIS 7. My app needs to read the websites configuration (via metabase properties in IIS 6).
I read a lot of articles about the configuration paths and I think I have a good idea of how it work - however I am not sure of one thing:
To get to the configuration, I can either commit the MACHINE/WEBROOT/APPHOST/ path or the MACHINE/WEBROOT/APPHOST/Default Web Site.
I understand that the latter refers to the actual web.config of the specific website, and the former refers to the general applicationHost.config file, in which general settings are set.
My app doesn't know however whether a web.config file exists.
My question: if I want to get to this path - Object.ConfiguredObject.Site.Bindings, do I need to commit the APPHOST path or the APPHOST/Default Web Site path?
How do I know that in runtime?
You will always commit your bindings to MACHINE/WEBROOT/APPHOST.
You should go have a look at the schema files in:
%systemroot%\System32\inetsrv\config\schema
They will help you identify where settings should belong.
Update:
Per your comment:
So for example, AccessSSLFlags would
be mapped to
ConfigurationSection.AccessSection.SslFlags
- what section will I commit in that case? How do I know which section I
need to commit?
That all depends. IIS7 supports a mechanism called Feature Delegation. If a feature is delegated then this means a user can configure that feature in their local web.config. Some features are configured under system.webServer, others system.web.
What a user can and can't configure locally in his/her web.config is controlled by entries in two files:
%systemrooot%\system32\inetsrv\config\administration.config
%systemrooot%\system32\inetsrv\config\applicationHost.config
If you go and look at the IIS7 configuration schema in:
%systemroot%\System32\inetsrv\config\schema\IIS_schema.xml
What you'll find is that there are two main types of section:
system.applicationHost/xxxx
system.webServer/xxxx
Anything that is configurable under system.applicationHost is generally not considered a user modifiable configuration item. In fact if you open applicationHost.config you will see:
<sectionGroup name="system.applicationHost">
<section name="applicationPools" allowDefinition="AppHostOnly" overrideModeDefault="Deny" />
<section name="configHistory" allowDefinition="AppHostOnly" overrideModeDefault="Deny" />
<section name="customMetadata" allowDefinition="AppHostOnly" overrideModeDefault="Deny" />
<section name="listenerAdapters" allowDefinition="AppHostOnly" overrideModeDefault="Deny" />
<section name="log" allowDefinition="AppHostOnly" overrideModeDefault="Deny" />
<section name="serviceAutoStartProviders" allowDefinition="AppHostOnly" overrideModeDefault="Deny" />
<section name="sites" allowDefinition="AppHostOnly" overrideModeDefault="Deny" />
<section name="webLimits" allowDefinition="AppHostOnly" overrideModeDefault="Deny" />
</sectionGroup>
Notice the allowDefinition="AppHostOnly"? That's basically telling you that these settings can't be configured in web.config.
The scope of how feature delegation works is far too wide to cover in an answer so I suggest you read the article linked to above.
It sounds like you are trying to build a generic tool to manage configuration, and so you might want to consider to follow a similar pattern that IIS Manager follows; in short, it always tries to save configuration to the deepest path possible. What this means is that it will always Commit it to the place where it can, by looking at if the section is locked or not. It uses Managed code (Microsoft.Web.Administration), but you can access the same data from C++ using AppHostElement.GetMetadata("isLocked").
By the way if you are using C++, I would STRONGLY recommend using AHADMIN directly (and not WMI, or anything else), in particular IAppHostWritableAdminManager.
So the algorithm would be, set the CommitPath to the same value as the GetAdminSection configuration path specified. Then check for IsLocked, if it is then remove the last "path part" (trim starting the last '/'), and read again till you find the place where the section is unlocked. That is the deepest place where you can save it. Also, you will need to switch to MACHINE/WEBROOT at some point if it is a system.web section. IsLocked will respect things like Section Definition allow location, and other things that are required. If you want to make it bullet proof you would even need to check for attribute-level locking, but I think that is quite advanced.

Azure connection string best practices

I have an application that I am just migrating to Azure. Currently I use web.config transformation to manage changing the database connecting string dev/staging/prod environments. How is it best to manage these multiple connection strings in Azure?
In cases where it doesn't matter if the developer can see production credentials, you can use the built-in Visual Studio 10 config transformations. If this is what you're looking for, follow these steps:
1.Navigate to your Azure project folder in file explorer
2. Make a copy of ServiceConfiguration.cscfg
3. Rename copy to ServiceConfiguration.Base.cscfg
4. For each build configuration (e.g. Dev, Staging, Production), create a ServiceConfiguration.<build config name>.cscfg file. In these files, you can use the normal config transformation syntax
5. Open your .ccproj file in a text editor
6. Find the following node,
<ItemGroup>
<ServiceDefinition Include="ServiceDefinition.csdef" />
<ServiceConfiguration Include="ServiceConfiguration.cscfg" />
</ItemGroup>
and replace it with this (you will have to edit this block to match your build configs):
<ItemGroup>
<ServiceDefinition Include="ServiceDefinition.csdef" />
<ServiceConfiguration Include="ServiceConfiguration.cscfg" />
<None Include="ServiceConfiguration.Base.cscfg">
<DependentUpon>ServiceConfiguration.cscfg</DependentUpon>
</None>
<None Include="ServiceConfiguration.Dev.cscfg">
<DependentUpon>ServiceConfiguration.cscfg</DependentUpon>
</None>
<None Include="ServiceConfiguration.Staging.cscfg">
<DependentUpon>ServiceConfiguration.cscfg</DependentUpon>
</None>
<None Include="ServiceConfiguration.Production.cscfg">
<DependentUpon>ServiceConfiguration.cscfg</DependentUpon>
</None>
</ItemGroup>
7.Add the following at the end of the .ccproj file, just above </Project>:
<Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v10.0\Web\Microsoft.Web.Publishing.targets" />
<Target Name="BeforeBuild">
<TransformXml Source="ServiceConfiguration.Base.cscfg" Transform="ServiceConfiguration.$(Configuration).cscfg" Destination="ServiceConfiguration.cscfg" />
</Target>
8.If you're using a CI server that doesn't have Visual Studio 10 installed, you'll probably have to copy the C:\Program Files\MSBuild\Microsoft\VisualStudio\v10.0\Web folder and its contents from a development machine to the server.
Update: As #SolarSteve noted, you might have to add a namespace to your ServiceConfiguration.*.cscfg files. Here's an example of ServiceConfiguration.Base.cscfg:
<sc:ServiceConfiguration serviceName="MyServiceName" osFamily="1" osVersion="*" xmlns:sc="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<sc:Role name="MyRoleName">
<sc:Instances count="1" />
<sc:ConfigurationSettings>
<sc:Setting name="DataConnectionString" value="xxx" />
</sc:ConfigurationSettings>
</sc:Role>
</sc:ServiceConfiguration>
Personally we:
Dropped web config transformations completely.
Setting is retrieved from cscfg.
Development version of cscfg points to local development environment (that's stored in version control).
While deploying to production, we supply secure credentials for production SQL Azure and storage.
For sample of the settings management class that scans application settings and cloud environment for configuration values, you can check out open source Lokad.CQRS for Windows Azure project (see CloudSettingsProvider)
You can use CloudConfigurationManager in Azure SDK 1.7 http://msdn.microsoft.com/en-us/LIBRARY/microsoft.windowsazure.cloudconfigurationmanager
This starts by looking in the ServiceConfiguration.cscfg e.g. ServiceConfiguration.Cloud.cscfg for config setting. If it isn't there it falls back to web.config and app.config
For example
CloudConfigurationManager.GetSetting("StorageConnectionString")
Will look in the appropriate cscfgfile for StorageConnectionString setting, then it will search the web.config and then app.config.
We have a number of environments (local dev inside dev fabric, local dev outside dev fabric, testing, release which has 2 versions: release/prod and release/staging and 20 projects some of which need some variability in configure settings. We solved this problem by creating a tiny "config" project, included subfolders there that match the environments. We copy files from the subfolder depending on which build we're doing into root folder of the config project, during every compile.
All other projects link to the config project for .config files. We also use partial config files to keep the insanity of repeating the same info all the time across various environments.
Hope this helps
I had the same requirement for transforming ServiceConfiguration.
I went with the answer from jmac (thank you!), but had trouble with the namespace in the Base version:
<ServiceConfiguration serviceName="TestCloud2" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" osFamily="1" osVersion="*">
after a bit more poking around found this by Andrew Patterson (Thank You).
so my resulting transform file:
<asc:ServiceConfiguration serviceName="TestCloud2" xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform" xmlns:asc="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" osFamily="1" osVersion="*">
<asc:Role name="WebRole1">
<asc:Instances count="1" />
<asc:ConfigurationSettings>
<asc:Setting name="LoggingStorage" value="UseDevelopmentStorage=true" xdt:Transform="SetAttributes" xdt:Locator="Match(name)"/>
</asc:ConfigurationSettings>
</asc:Role>

Resources