Azure connection string best practices - azure

I have an application that I am just migrating to Azure. Currently I use web.config transformation to manage changing the database connecting string dev/staging/prod environments. How is it best to manage these multiple connection strings in Azure?

In cases where it doesn't matter if the developer can see production credentials, you can use the built-in Visual Studio 10 config transformations. If this is what you're looking for, follow these steps:
1.Navigate to your Azure project folder in file explorer
2. Make a copy of ServiceConfiguration.cscfg
3. Rename copy to ServiceConfiguration.Base.cscfg
4. For each build configuration (e.g. Dev, Staging, Production), create a ServiceConfiguration.<build config name>.cscfg file. In these files, you can use the normal config transformation syntax
5. Open your .ccproj file in a text editor
6. Find the following node,
<ItemGroup>
<ServiceDefinition Include="ServiceDefinition.csdef" />
<ServiceConfiguration Include="ServiceConfiguration.cscfg" />
</ItemGroup>
and replace it with this (you will have to edit this block to match your build configs):
<ItemGroup>
<ServiceDefinition Include="ServiceDefinition.csdef" />
<ServiceConfiguration Include="ServiceConfiguration.cscfg" />
<None Include="ServiceConfiguration.Base.cscfg">
<DependentUpon>ServiceConfiguration.cscfg</DependentUpon>
</None>
<None Include="ServiceConfiguration.Dev.cscfg">
<DependentUpon>ServiceConfiguration.cscfg</DependentUpon>
</None>
<None Include="ServiceConfiguration.Staging.cscfg">
<DependentUpon>ServiceConfiguration.cscfg</DependentUpon>
</None>
<None Include="ServiceConfiguration.Production.cscfg">
<DependentUpon>ServiceConfiguration.cscfg</DependentUpon>
</None>
</ItemGroup>
7.Add the following at the end of the .ccproj file, just above </Project>:
<Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v10.0\Web\Microsoft.Web.Publishing.targets" />
<Target Name="BeforeBuild">
<TransformXml Source="ServiceConfiguration.Base.cscfg" Transform="ServiceConfiguration.$(Configuration).cscfg" Destination="ServiceConfiguration.cscfg" />
</Target>
8.If you're using a CI server that doesn't have Visual Studio 10 installed, you'll probably have to copy the C:\Program Files\MSBuild\Microsoft\VisualStudio\v10.0\Web folder and its contents from a development machine to the server.
Update: As #SolarSteve noted, you might have to add a namespace to your ServiceConfiguration.*.cscfg files. Here's an example of ServiceConfiguration.Base.cscfg:
<sc:ServiceConfiguration serviceName="MyServiceName" osFamily="1" osVersion="*" xmlns:sc="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<sc:Role name="MyRoleName">
<sc:Instances count="1" />
<sc:ConfigurationSettings>
<sc:Setting name="DataConnectionString" value="xxx" />
</sc:ConfigurationSettings>
</sc:Role>
</sc:ServiceConfiguration>

Personally we:
Dropped web config transformations completely.
Setting is retrieved from cscfg.
Development version of cscfg points to local development environment (that's stored in version control).
While deploying to production, we supply secure credentials for production SQL Azure and storage.
For sample of the settings management class that scans application settings and cloud environment for configuration values, you can check out open source Lokad.CQRS for Windows Azure project (see CloudSettingsProvider)

You can use CloudConfigurationManager in Azure SDK 1.7 http://msdn.microsoft.com/en-us/LIBRARY/microsoft.windowsazure.cloudconfigurationmanager
This starts by looking in the ServiceConfiguration.cscfg e.g. ServiceConfiguration.Cloud.cscfg for config setting. If it isn't there it falls back to web.config and app.config
For example
CloudConfigurationManager.GetSetting("StorageConnectionString")
Will look in the appropriate cscfgfile for StorageConnectionString setting, then it will search the web.config and then app.config.

We have a number of environments (local dev inside dev fabric, local dev outside dev fabric, testing, release which has 2 versions: release/prod and release/staging and 20 projects some of which need some variability in configure settings. We solved this problem by creating a tiny "config" project, included subfolders there that match the environments. We copy files from the subfolder depending on which build we're doing into root folder of the config project, during every compile.
All other projects link to the config project for .config files. We also use partial config files to keep the insanity of repeating the same info all the time across various environments.
Hope this helps

I had the same requirement for transforming ServiceConfiguration.
I went with the answer from jmac (thank you!), but had trouble with the namespace in the Base version:
<ServiceConfiguration serviceName="TestCloud2" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" osFamily="1" osVersion="*">
after a bit more poking around found this by Andrew Patterson (Thank You).
so my resulting transform file:
<asc:ServiceConfiguration serviceName="TestCloud2" xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform" xmlns:asc="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" osFamily="1" osVersion="*">
<asc:Role name="WebRole1">
<asc:Instances count="1" />
<asc:ConfigurationSettings>
<asc:Setting name="LoggingStorage" value="UseDevelopmentStorage=true" xdt:Transform="SetAttributes" xdt:Locator="Match(name)"/>
</asc:ConfigurationSettings>
</asc:Role>

Related

Azure function returns error: No job functions found. Try making your job classes and methods public

I have created an Azure Function application in Visual Studio 2017 and can publish it to Azure without any issues using the Visual Studio publish functionality (right click project then select Publish).
As deploying from a developers instance of Visual Studio isn't an ideal continuous integration strategy, I have created a deployment pipeline with TeamCity and Octopus where TeamCity builds the Azure function application and Octopus uses the WAWSDeploy application to deploy the Azure Function files to Azure. The deployment works fine and I when I view the Azure function files when deployed via WAWSDeploy, the files are exactly the same as when I publish the Azure Function application from Visual Studio.
However I get the errors No job functions found. Try making your job classes and methods public. and Invalid script file name configuration. The 'scriptFile' property is set to a file that does not exist. when I deploy (by viewing the Azure Function application logs) from WAWSDeploy. This doesn't appear to be a WAWSDeploy issue but it looks like the Visual Studio publish function is doing something I'm missing. Any ideas?
Folder structure of Azure function files:
Issue was caused by having an incorrect scriptFile value in the function.json file. When I published the Azure function from Visual Studio, this value is set correctly but when I build the Azure function and push the files to the Azure function application manually, the scriptFile value is missing the bin folder in the path to my function dll file. During my build process, I now hack the scriptFile value to set it correctly.
I faced similar issues and finally managed to found the root cause. You may fixed this problem, but putting this here for anyone else facing this problem.
Ensure you have this following block of element in your .csproj in of a Function App
<None Update="host.json">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
I got this problem when i excluded host.json from the project and then created one again (after realizing this is needed for sure).
valid .csproj should look something like this
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net461</TargetFramework>
</PropertyGroup>
...
...
...
<ItemGroup>
<None Update="host.json">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Update="local.settings.json">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<CopyToPublishDirectory>Never</CopyToPublishDirectory>
</None>
</ItemGroup>
</Project>
Hopefully Adding that will solve this issue. Let me know if this helps :)
I faced the same issue,
I deleted the bin and obj folders and rebuilt the project and it works perfectly then.

Refer a connectionstring from web config in another config file

I am using NLog to write logs to my database,
I have created a file NLog.config which is writing logs to a text file as of now.
To write the logs to a database, I am following this tutorial.
However, the connectionstrings for diferrent environments can be only modified in Web.config. (I am using Azure App services). Is there any way I can refer the connection string from web.config in NLog.config.
TIA
If you not using ASP.NET Core (but "full" ASP.NET), you could use ${appsetting:name=..}
Install NLog.Extended with Nuget and use ${appsetting:name=..} in your config file.
e.g.
<target name="database"
type="Database"
connectionString="${appsetting:name=myConnectionString}" />
See also the ${appsetting} documentation
NB: It can only read <appSettings> and not <connectionStrings>

Service Fabric Default Publish Profile other than Local.xml

Our company is developing our new applications using Service Fabric.
A common problem we have, multiple developers use queues, databases, storages that are on remote servers, and each one has different configuration for this, all the settings are stored on ApplicationParameters file per environment, for Local development there is a single Local.5Node.xml. It is very common developers checkin their credentials and overwrite others when we get latest version of these files.
I'm trying to customize the ServiceFabric deployment script 'Deploy-FabricApplication.ps1' to use a custom PublishProfile depending on windows credentials of logged user. I can achieve that updating the deployment file, it works well when we deploy using the publish, but it seems that the default behavior of ServiceFabric when we hit F5(debug) is overwrite the parameters with a specific Local.5Node.xml application parameters.
I explored all service fabric .ps1 files and couldn't find where this is defined. I guess this is defined on .targets file, so I don't know how can I avoid this default behaviour.
Is there any other approach to use custom PublishProfiles on local development machines other than Local.5Node.xml?
I actually just ran into this with setting up some team specific environments. I borrowed information from the following sources:
Web Config Transformation
Replace String In File With MSBUILD
I added multiple parameters files based on what was needed for the different teams. Each one containing their specific resource settings.
I also added a Local.1Node.Template.xml and Local.5Node.Template.xml. I even removed the Local.1Node.xml and Local.5Node.xml from source control and set them up to be ignored while leaving them in the projects so that Visual Studio doesn't think they are truly missing. The contents of the 1Node (5Node is the same except for replacing 1Node with 5Node) are as follows:
<?xml version="1.0" encoding="utf-8"?>
<PublishProfile xmlns="http://schemas.microsoft.com/2015/05/fabrictools">
<ClusterConnectionParameters />
<ApplicationParameterFile Path="..\ApplicationParameters\Local.1Node.$(Configuration).xml" />
</PublishProfile>
I then edited the sfproj file for the Service Fabric project to contain the following MSBuild Task and Target:
<UsingTask TaskName="ReplaceFileText" TaskFactory="CodeTaskFactory" AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.v4.0.dll">
<ParameterGroup>
<InputFilename ParameterType="System.String" Required="true" />
<OutputFilename ParameterType="System.String" Required="true" />
<MatchExpression ParameterType="System.String" Required="true" />
<ReplacementText ParameterType="System.String" Required="true" />
</ParameterGroup>
<Task>
<Reference Include="System.Core" />
<Using Namespace="System" />
<Using Namespace="System.IO" />
<Using Namespace="System.Text.RegularExpressions" />
<Code Type="Fragment" Language="cs">
<![CDATA[
File.WriteAllText(
OutputFilename,
Regex.Replace(File.ReadAllText(InputFilename), MatchExpression, ReplacementText)
);
]]>
</Code>
</Task>
</UsingTask>
<Target Name="UpdateProfile" BeforeTargets="UpdateServiceFabricApplicationManifest">
<ReplaceFileText InputFilename="PublishProfiles\Local.1Node.Template.xml" OutputFilename="PublishProfiles\Local.1Node.xml" MatchExpression="\$\(Configuration\)" ReplacementText="$(Configuration)" />
<ReplaceFileText InputFilename="PublishProfiles\Local.5Node.Template.xml" OutputFilename="PublishProfiles\Local.5Node.xml" MatchExpression="\$\(Configuration\)" ReplacementText="$(Configuration)" />
</Target>
The final step was to setup the different Build Configurations for the teams. I created a FT1-Debug through FT6-Debug based on the Debug configuration in the Service Fabric Service project and the Service Fabric Host project. I left all of my other projects alone.
At this point everyone on the different teams can debug locally with the correct configuration for the cluster they are doing work in just by changing the Build Configuration and pressing F5 to debug.
The VS extension for Service Fabric define a hard coded publish profile when we debug the solution using Visual Studio, it check how many nodes my cluster has and create a link to Local.5Node.xml and Local.1Node.xml depending how many nodes my cluster have.
To accomplish the same results, we end up using custom Application Parameters per developer and each developer update the Publish Profile (Local.5node.xml) to point to their respective Application parameter files.
It is not automated as the required feature, but can solve the main problem.

how to use msbuild properties in sqlproj (SQL Server 2012) script

I just upgraded my existing SQL Server 2008 r2 .dbproj to a SQL Server 2012 .sqlproj (using SQL Server Data Tools).
Previously, I was able to define a SQLCMD variable in my project, and then define the value by editing the project file to use msbuild values by adding the following element:
<ItemGroup>
<SqlCommandVariableOverride Include="ProjectDirectory=$(MSBuildProjectDirectory)" />
</ItemGroup>
Which I could then use in my PostDeployment script like this:
SELECT * INTO dbo.MyTable FROM dbo.MyTable WHERE 1=2
BULK INSERT dbo.MyTable
FROM '$(ProjectDirectory)\data\dbo.MyTable.dat'
WITH (DATAFILETYPE = 'widenative')
However, after the upgrade, this no longer seems to work.
I have tried adding that same entry to the new sqlproj, but the Publish functionality doesn't seem to pick it up and wants me to supply a value. If I supply $(MSBuildProjectDirectory), that is interpreted literally and fails.
Under the new regime, what is the mechanism for specifying a local filepath and/or using msbuild values?
In a sql server 2012 sqlproj (SSDT database project) you use publishing profiles. You can start off by right-clicking your database project and choosing 'Publish'.
You can then set desired options and save these in a so-called publishing profile in your project. Double-clicking this profile launches the publishing wizard with the correct options set.
In your publish profile you can include hard-coded values for sqlcmd variables:
<ItemGroup>
<SqlCmdVariable Include="ProjectDirectory">
<Value>UNKNOWN</Value>
</SqlCmdVariable>
</ItemGroup>
If desired, you can update these with dynamic values during build. In your msbuild project:
<Target Name="SetProjectDirectoryInPublishXml">
<ItemGroup>
<Namespaces Include="nsMsbuild">
<Prefix>nsMsbuild</Prefix>
<Uri>http://schemas.microsoft.com/developer/msbuild/2003</Uri>
</Namespaces>
</ItemGroup>
<ItemGroup>
<SSDTPublishFiles Include="$(SolutionBinFolder)\**\*.publish.xml" />
</ItemGroup>
<MSBuild.ExtensionPack.Xml.XmlFile Condition="%(SSDTPublishFiles.Identity) != ''"
TaskAction="UpdateElement"
File="%(SSDTPublishFiles.Identity)"
Namespaces="#(Namespaces)"
XPath="//nsMsbuild:SqlCmdVariable[#Include='ProjectDirectory']/nsMsbuild:Value"
InnerText="$(MSBuildProjectDirectory)"/>
</Target>
This requires an extension to update the XML. I use the msbuild extension pack.
Credits for this mechanism go to Jamie Thomson

Integrate YuiCompressor into TFS build

I Already deploy my website using TFS Build template, now i want to add css and js minify and compression using yuicompressor port for .net, i have follow all instruction at Yui compressor site in Codeplex but i have no results. I need some example or any other probed way to use yuicompressor in TFS build proccess.
this is the portion of configuration related with yui-compressor that i have append to my project file
<UsingTask
TaskName="CompressorTask"
AssemblyFile="$(MSBuildProjectDirectory)\..\packages\YUICompressor.NET-MsBuild-Task.1.7.0.0\lib\net35\Yahoo.Yui.Compressor.MsBuildTask.dll" />
<PropertyGroup>
<SourceLocation>$(MSBuildProjectDirectory)</SourceLocation>
<CssOutputFile>$(SourceLocation)\Content\sylessheetfinal.css</CssOutputFile>
<JavaScriptOutputFile>$(SourceLocation)\Scripts\javascriptfinal.js</JavaScriptOutputFile>
</PropertyGroup>
<Target Name="MyTaskTarget">
<ItemGroup>
<!-- All the files. They will be handled (I assume) in alphabetically. -->
<CssFiles Include="$(SourceLocation)\Content\*.css" />
<JavaScriptFiles Include="$(SourceLocation)\Scripts\*.js" />
</ItemGroup>
<CompressorTask
CssFiles="#(CssFiles)"
DeleteCssFiles="false"
CssOutputFile="$(CssOutputFile)"
CssCompressionType="YuiStockCompression"
JavaScriptCompressionType="YuiStockCompression"
JavaScriptFiles="#(JavaScriptFiles)"
ObfuscateJavaScript="True"
PreserveAllSemicolons="False"
DisableOptimizations="Nope"
EncodingType="Default"
DeleteJavaScriptFiles="false"
LineBreakPosition="-1"
JavaScriptOutputFile="$(JavaScriptOutputFile)"
LoggingType="ALittleBit"
ThreadCulture="en-au"
IsEvalIgnored="false"/>
</Target>
Are you using Team Build on a Team Foundation Server? The instructions on that web site are for local developer's machine as they mention invoking msbuild.exe on the command-line in post-build event.
Note that the project file (i.e. csproj) is using msbuild xml syntax so having msbuild.exe in post-build is excessive even for local developer's build - so what you may need to do is to copy parts of their msbuild.xml in your project file, if you are using web application project

Resources