SSIS package works from SSMS but not from agent job - excel

I've an SSIS package to load excel file from network drive. It's designed to load content and then move the file to archived folder.
Everything works good when the following SQL statement runs in SSMS window.
However when it's copied to SQL agent job and executes from there, the file is neither loaded nor moved. But it shows "successful" from the agent log.
The same thing also happened to "SSIS job" instead of T-SQL job, even with proxy of windows account.(same account as ssms login)
Declare #execution_id bigint
EXEC [SSISDB].[catalog].[create_execution] #package_name=N'SG_Excel.dtsx', #execution_id=#execution_id OUTPUT, #folder_name=N'ETL', #project_name=N'Report', #use32bitruntime=True, #reference_id=Null
Select #execution_id
DECLARE #var0 smallint = 1
EXEC [SSISDB].[catalog].[set_execution_parameter_value] #execution_id, #object_type=50, #parameter_name=N'LOGGING_LEVEL', #parameter_value=#var0
EXEC [SSISDB].[catalog].[start_execution] #execution_id
GO
P.S. At first relative path of network drive is applied, then switched to absolute path(\\server\folder). It's not solving the issue.

SSIS Package Jobs run under the context of the SQL Server Agent. What Account is setup to run the SQL Server Agent on the SQL Server? It may need to be run as a Domain account that has access to the network share.
Or you can copy the Excel file to local folder on the SQL Server, so the Package can access the file there.

Personally I avoid the File System Task - I have found it unreliable. I would replace that with a Script Task, and use .NET methods from the System.IO namespace e.g. File.Move. These are way more reliable and have mature error handling.
Here's a starting point for the System.IO namespace:
https://msdn.microsoft.com/en-us/library/ms404278.aspx
Be sure to select the relevant .NET version using the Other Versions link.

When I have seen things like this in the past it's been that my package isn't accessing the path I thought it was at run time, its looking somewhere else, finding an empty folder & exiting with success.
SSIS can have a nasty habit of going back to variable defaults . It may be looking at a different path you used in dev? Maybe hard code all path values as a test? or put in break points & double check the run time values of all variables & parameters.
Other long shots may be:
Name resolution, are you sure the network name is resolving correctly at runtime?
32/64 bit issues. Dev tends to run 32 bit, live may be 64 bit. May interfere with file paths? Maybe force to 32 bit at run time?

There is issue with sql statement not having statement terminator (;) that is causing issue.
Declare #execution_id bigint ;
EXEC [SSISDB].[catalog].[create_execution] #package_name=N'SG_Excel.dtsx', #execution_id=#execution_id OUTPUT, #folder_name=N'ETL', #project_name=N'Report', #use32bitruntime=True, #reference_id=Null ;
Select #execution_id ;
DECLARE #var0 smallint = 1 ;
EXEC [SSISDB].[catalog].[set_execution_parameter_value] #execution_id, #object_type=50, #parameter_name=N'LOGGING_LEVEL', #parameter_value=#var0 ;
EXEC [SSISDB].[catalog].[start_execution] #execution_id ;
GO
I have faced similar issue in service broker ..

Related

Snakemake cannot write metadata

I have troubles to get snakemake-minimal=7.8.5 to run on Windows 10. I can execute rules, but snakemake terminates due to an error regarding the metadata:
Failed to set marker file for job started ([Errno 2] No such file or directory: 'C:\\test\\project\\.snakemake\\incomplete\\cnVucy9leHBlcmltZW50XzAzL2RmX2ludGVuc2l0aWVzX3Byb3RlaW5Hcm91cHNfbG9uZ18yMDE3XzIwMThfMjAxOV8yMDIwX04wNTAxNV9NMDQ1NDcvUV9FeGFjdGl2ZV9IRl9YX09yYml0cmFwX0V4YWN0aXZlX1Nlcmllc19zbG90XyM2MDcwLzE0X2V4cGVyaW1lbnRfMDNfZGF0YS5pcHluYg=='). Snakemake will work, but cannot ensure that output files are complete in case of a kill signal or power loss. Please ensure write permissions for the directory C:\test\project\.snakemake
I tried to troubleshoot doing the following
change the folders: Documents, User folder, and like the above in the root folder of my c drive
I tried to manipulate the security settings: Controlled folder or RandsomWare Access, see discussion -> it is deactivated
If I erase the .snakemake it is re-creating upon execution, so I assume I have write access. However, some security setting is disallowing the long filename with the hash
I tried the same workflow on a different Windows 10 machine and there I don't get the error, so I assume it is some windows issue.
Did anyone encounter the same error and found a solution?
I agree it is due to the length of the filename. It seems the default max filename length is 260. The file you pasted has a length of 262. You can edit the registry to allow longer filenames. Also consider opening an issue in snakemake to improve the documentation or otherwise address this issue for windows machines.

Azure ML release bug AZUREML_COMPUTE_USE COMMON_RUNTIME

On 2021-10-13 in our application in Azure ML platform we get this new error that causes failures in pipeline steps - python module import failures - warning stack <- warning that leads to pipeline runtime error
we needed to set it to false. Why is it failing? What exactly are exact (and long term) consequences when opting out? Also, Azure ML users - do you think it was rolled out appropriately?
Try to add into your envirnoment new variable like this:
environment.environment_variables = {"AZUREML_COMPUTE_USE_COMMON_RUNTIME":"false"}
Long term (throughout 2022), AzureML will be fully migrating to the new Common Runtime on AmlCompute. Short term, this change is a large undertaking, and we're on the lookout for tricky functionality of the old Compute Runtime we're not yet handling correctly.
One small note on disabling Common Runtime, it can be more efficient (avoids an Environment rebuild) to add the environment variable directly to the RunConfig:
run_config.environment_variables["AZUREML_COMPUTE_USE_COMMON_RUNTIME"] = "false"
We'd like to get more details about the import failures, so we can fix the regression. Are you setting the PYTHONPATH environment variable to make your custom scripts importable? If so, this is something we're aware isn't working as expected and are looking to fix it within the next two weeks.
We identified the issue and have rolled out a hotfix on our end addressing the issue. There are two problems that could've caused the import issue. One is that we are overwriting the PYTHONPATH environment variable. The second is that we are not adding the python script's containing directory to python's module search path if the containing directory is not the current working directory.
It would be great if you can please try again without setting the AZUREML_COMPUTE_USE_COMMON_RUNTIME environment variable and see if the problem is still there. If it is, please reply to either Lucas's thread or mine with a minimal repro or description of where the module you are trying to import is located at in relation to the script being run and the root of the snapshot (which is the current working directory).

Code Behind RStudio Server Export Function

I am currently using RStudio-server on Linux redhat. One nice feature of RStudio-server is that I can export from the server to my Windows desktop. Does anyone know the code behind the export drop-down?
The export function can be found via the Files tab:
(More >> Export...)
I would like use code to automate the exporting of objects. I figured I should be able to perform this export using the system function, but I am having trouble.
Thanks for any help.
I think this post might help you,
Spacedman explains that you can trigger the export by the use of the R function "browseURL", with the URL parameter replaced by the ftp path to the file.
If you absolutely want to trigger this export with a system command, perhaps you could create an R script taking as parameter the file to export and launch that script with the system() function =) Although I can't see clearly the advantages of such a process.
[edit] : After having tried it today, I realise my answer wasn't complete :
If you try the function browseURL on files such as "whateverRscript.r", it will display it in a tab of your browser, rather than trigger the download.
In order to actually make your browser download this kind of file, maybe you can zip it first.
To complete the automation process, just change the parameters of your browser such that it won't "ask everytimes where to stock the downloaded files"
This is what worked for me: run it on Server side. Working browser is required (I used Chrome)
my_data_file_name <- "data.RData"
# set file name
save(Data, file=my_data_file_name)
# save data to file
current_dir <- getwd()
# capture current working directory on server
my_export_file_path <- paste0(current_dir, '/', my_data_file_name)
# create a path for file to export
browseURL(my_export_file_path)
# export to local disk using browser's capabilities

VS2012 & 2013: Can't publish Services project - specified path is too long

I have a VS2012 solution, containing 10 projects, and suddenly, I can no longer publish my Services project to any folder.
When I try to publish to D:\temp, I get this error:
The expression "[System.IO.Path]::GetFullPath(obj\Release%25252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252528Prod%25252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252529\)" cannot be evaluated. The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters. C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\Web\Microsoft.Web.Publishing.targets
Huh ?
VS2012 (with update 4) seems to have taken my configuration name "Release(Prod)" and completely messed it up, causing the GetFullPath to produce too long a path name.
How the heck can I fix this ?
Out of desperation, I tried to build and publish the same project in VS2013 - and it had the same error message.
One of my colleagues said he'd seen the same thing, but had fixed it by removing the spaces from his configuration name. I tried this, which is why my configuration name is now "Release(Prod)" rather than "Release (Prod)", but it made no difference.
I did also open the file which this error is suggesting is the cause of the error:
C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\Web\Microsoft.Web.Publishing.targets
..and noticed that there's something in there concerning the AnyCPU platform name. I have tried getting my Services project to use "AnyCPU" and "Any CPU" (depressed sigh) but neither seems to make any difference.
<PropertyGroup Condition=" '$(IntermediateOutputPath)' == '' ">
<IntermediateOutputPath Condition=" '$(PlatformName)' == 'AnyCPU' Or '$(PlatformName)' == ''">$(BaseIntermediateOutputPath)$(Configuration)\</IntermediateOutputPath>
. . .
</PropertyGroup>
Has anyone else seen this issue ?
(A little bit later..)
This is so odd (and frustrating).
My Solution has 5 configurations - the default Debug and Release ones, plus extra configurations for Test, PreProduction and Production environments.
If I select any of these three configurations containing brackets, I get this ridiculous "The specified path is too long" error, as VS2012 corrupts the pathname (as shown in my first screenshot above).
I can't help wondering... is this some kind of VS2012 bug, handling spaces or brackets in the configuration name ?
I can deploy to a path directly with (, (, )) in the Target Location on tyhe Connection tab when publishing to the file system (i.e. not building a path from the configuration name) - but that is not a solution to targeting different locations based on the Configuration.
If you want to keep special characters in the configuration name, but specify a path to the deployment folder that will not cause and issue this post might help: Visual Studio: How to properly build and specify the configurations and platforms for x64 and x86
Specifically play with the settings in here:
In the project properties page, select the various permutations of
Debug/Release and x86/x64 in the solution dropdowns. Make sure the
target processor is set correctly (it should be, but I found instances
when they were not, probably because of my previous attempts). Also,
set the output directory. That should be okay and automatic
(/bin/x86/Debug, etc.). If not, fix.
Looking at what is actually seems to be going on is also potentially useful:
Looking at the numbers inserted:
%25 is an encoded %,
%28 is an encoded (
%29 is an encoded )
Looking at the path:
obj\Release%252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525***28***Prod%252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525***29***)
What I think is happening:
So that seems like a good clue that these are being URL or XML encoded. What appears to be happening is that the ( is being encoded as %28 and then the % is being recursively encoded as %25 - generating an infinite %252525252525252525....
A more interesting question is actually why it stops creating 25's from the %'s with this bug (both times it stops creating 25's at 214 characters including the % and the 28 / 29 - not a very interesting number).
Looking at the file C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\Web\Microsoft.Web.Publishing.targets you reference - it makes sense that these strings are being encoded for XML. I would say this is definitely a bug... I have no suggestions for a fix.
Well, I'm going to accept Matthew's answer as the "Accepted Answer".
Thank you for your help.
This is a really odd problem though, and I'm amazed no one else has reported this elsewhere.
Summary of problem (in case Microsoft is interested, or if anyone tries to Google this issue in the coming years)
With a configuration name of "Release (Prod)", I could happily build my code, run it locally, but when I tried to publish it, even to a local drive, I'd get this message:
It's an odd exception, because the Build did create the "obj\Release (Prod)" folder, without any issues. It's just the Publish which seemed to be looking in the wrong place for it.
Following the advice given in this thread, today I attempted to create a new configuration, with the same settings, but without a space in the name: "Release(SecondProd)". Look what happens:
Interestingly, despite this error, it did create a new configuration with this name.
Anyway, I recreated a new configuration, called it ReleaseProduction, and it worked fine.
Of course, I needed to create new "web.config" Transformation for this name, as this doesn't get automatically copied when you create a new configuration based on an old one.
One last thought (just to confuse matters worse !)
When I posted this plea-for-help, the Services project in my Solution refused to publish to a local drive, but my web site would publish okay.
Today, two days since I last attempted a website Publish, I found that the website also now produces the same GetFullPath exception. Nothing's changed ! We use TFS, I have done a file compare with my project files today against two days ago, and they're identical !
It's a really bizarre bug in VS2012 & VS2013.
Btw, this solution & the projects in it, were originally a VS2010 project. They were upgraded to VS2012 over a year ago, but this Publish problem only started happening recently. I'm not sure if the problem is related to using upgraded VS projects.
Again, thanks for your help.
Now I have some Test, PreProd & Production configurations to recreate !
Maybe I'll grab a beer first..
Summarizing and completing Matthew's answer:
Cause: You have configurations with chars that require URI-encoding - in your case, '(' and ')'.
Workaround: Rename those configs.
What happens: Presumably web deploy URI-encodes the path, replacing % => %25, ( => %28, ) => %29. It does so over and over:
obj\Release(Prod)
obj\Release%28Prod%29
obj\Release%2528Prod%2529
obj\Release%252528Prod%252529
...
Until the path exceeds MAX_PATH=260.
I ran into the same thing and all though it does not resolve the issue I found that if I switch the solution configuration away from a build containing "(" or ")"
Then use the appropriate build in the publish dialog it will not error out.

csrun loses executable from .csx package

I am having a hard time with a seemingly simple Azure program.
My exercise is to create WorkerRole that spawns "helloworld.exe"
- which does just that - prints "hello world" and exits.
I am using Visual Studio to create a project,
then added new folder to project solution "bin2" where I put hello.exe
using menu option "Add Existing Item".
then created local storage bin2 in ServiceDefinition.csdef:
so I can find my executable with RoleEnvironment:
string baseDir = RoleEnvironment.GetLocalResource("bin2").RootPath.Replace('\', '/');
string command = Path.Combine(baseDir, #"hello.exe");
then ran cspack.exe to create .csx directory.
Resulting .csx package got hello.exe in the correct location:
WorkerRole1.csx\roles\WorkerRole1\approot\bin2\hello.exe
then I started local development fabric with csrun.exe and get error from the parent process that bin2/hello.exe is missing.
Do I need to do something else to make csrun to copy hello.exe into "bin2".
Any ideas?
Thank you in advance,
Ivgard
I'm pretty sure I answered this question already (probably on the MSDN forum)? But the local resource you declare will give you a path entirely different from where you're putting your hello.exe. When you add the file to your project, it gets included with the rest of the code for your role. When you look up the local resource, you get a path to an empty directory which you can use to write and read data. Those two are completely separate and unrelated locations.
If you want to find your hello.exe that's under bin2, just look for the relative path, or use %RoleRoot%\approot\bin2 (or maybe it's %RoleRoot%\approot\bin\bin2?).

Resources