Error Accessing NuGet from an Inline Task - visual-studio-2012

Question

In Visual Studio 2012 I made an MsBuild inline task that works, until I uncomment a single line of code. When uncommented it fails with this message:
C:\Build\MyScript.targets(49,5): error MSB4018: The "MyCustomGetPath" task failed unexpectedly.
C:\Build\MyScript.targets(49,5): error MSB4018: System.IO.FileNotFoundException: Could not load file or assembly 'NuGet, Version=2.7.40906.75, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.
C:\Build\MyScript.targets(49,5): error MSB4018: File name: 'NuGet, Version=2.7.40906.75, Culture=neutral, PublicKeyToken=null'
C:\Build\MyScript.targets(49,5): error MSB4018: at InlineCode.GetNuGetRepositoryPath.Execute()
C:\Build\MyScript.targets(49,5): error MSB4018: at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
C:\Build\MyScript.targets(49,5): error MSB4018: at Microsoft.Build.BackEnd.TaskBuilder.<ExecuteInstantiatedTask>d__20.MoveNext()
C:\Build\MyScript.targets(49,5): error MSB4018:
C:\Build\MyScript.targets(49,5): error MSB4018: === Pre-bind state information ===
C:\Build\MyScript.targets(49,5): error MSB4018: LOG: User = Biz\baria2
C:\Build\MyScript.targets(49,5): error MSB4018: LOG: DisplayName = NuGet, Version=2.7.40906.75, Culture=neutral, PublicKeyToken=null
C:\Build\MyScript.targets(49,5): error MSB4018: (Fully-specified)
C:\Build\MyScript.targets(49,5): error MSB4018: LOG: Appbase = file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/
C:\Build\MyScript.targets(49,5): error MSB4018: LOG: Initial PrivatePath = NULL
C:\Build\MyScript.targets(49,5): error MSB4018: Calling assembly : (Unknown).
C:\Build\MyScript.targets(49,5): error MSB4018: ===
C:\Build\MyScript.targets(49,5): error MSB4018: LOG: This bind starts in default load context.
C:\Build\MyScript.targets(49,5): error MSB4018: LOG: Using application configuration file: C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe.Config
C:\Build\MyScript.targets(49,5): error MSB4018: LOG: Using host configuration file:
C:\Build\MyScript.targets(49,5): error MSB4018: LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
C:\Build\MyScript.targets(49,5): error MSB4018: LOG: Policy not being applied to reference at this time (private, custom, partial, or location-based assembly bind).
C:\Build\MyScript.targets(49,5): error MSB4018: LOG: Attempting download of new URL file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/NuGet.DLL.
C:\Build\MyScript.targets(49,5): error MSB4018: LOG: Attempting download of new URL file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/NuGet/NuGet.DLL.
C:\Build\MyScript.targets(49,5): error MSB4018: LOG: Attempting download of new URL file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/NuGet.EXE.
C:\Build\MyScript.targets(49,5): error MSB4018: LOG: Attempting download of new URL file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/NuGet/NuGet.EXE.
C:\Build\MyScript.targets(49,5): error MSB4018:
Done Building Project "C:\Build\MyScript.targets" (default targets) -- FAILED.
The command I am executing is:
C:\Build\>msbuild MyScript.targets
Here is the inline task:
<UsingTask TaskName="MyCustomGetPath" TaskFactory="CodeTaskFactory" AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.v4.0.dll">
<ParameterGroup>
<config ParameterType="System.String" Required="true" />
<repositoryPath ParameterType="System.String" Output="true" />
</ParameterGroup>
<Task>
<Reference Include="$(NuGetExePath)" />
<Using Namespace="NuGet" />
<Using Namespace="System.IO" />
<Code Type="Fragment" Language="cs">
<![CDATA[
ISettings settings = null;
string directoryName = Path.GetDirectoryName(Path.GetFullPath(config));
string fileName = null;
if (File.Exists(config))
{
fileName = Path.GetFileName(config);
}
//var temp = new NullLogger();
//MessageLevel msg = MessageLevel.Warning;
//PhysicalFileSystem physicalFileSystem = new PhysicalFileSystem(directoryName);
repositoryPath = directoryName;
]]>
</Code>
</Task>
</UsingTask>
Yes, this code is referencing the NuGet.exe (version 2.7 in this case). There are three commented lines above; uncommenting any one of them will cause the error. Overall I suspect this is fusion related. That reasonably makes sense, but then what are my options to fix this?
Update
This post seems to be the same problem.