I have troubles to get snakemake-minimal=7.8.5 to run on Windows 10. I can execute rules, but snakemake terminates due to an error regarding the metadata:
Failed to set marker file for job started ([Errno 2] No such file or directory: 'C:\\test\\project\\.snakemake\\incomplete\\cnVucy9leHBlcmltZW50XzAzL2RmX2ludGVuc2l0aWVzX3Byb3RlaW5Hcm91cHNfbG9uZ18yMDE3XzIwMThfMjAxOV8yMDIwX04wNTAxNV9NMDQ1NDcvUV9FeGFjdGl2ZV9IRl9YX09yYml0cmFwX0V4YWN0aXZlX1Nlcmllc19zbG90XyM2MDcwLzE0X2V4cGVyaW1lbnRfMDNfZGF0YS5pcHluYg=='). Snakemake will work, but cannot ensure that output files are complete in case of a kill signal or power loss. Please ensure write permissions for the directory C:\test\project\.snakemake
I tried to troubleshoot doing the following
change the folders: Documents, User folder, and like the above in the root folder of my c drive
I tried to manipulate the security settings: Controlled folder or RandsomWare Access, see discussion -> it is deactivated
If I erase the .snakemake it is re-creating upon execution, so I assume I have write access. However, some security setting is disallowing the long filename with the hash
I tried the same workflow on a different Windows 10 machine and there I don't get the error, so I assume it is some windows issue.
Did anyone encounter the same error and found a solution?
I agree it is due to the length of the filename. It seems the default max filename length is 260. The file you pasted has a length of 262. You can edit the registry to allow longer filenames. Also consider opening an issue in snakemake to improve the documentation or otherwise address this issue for windows machines.
Related
Repo for all code I've been using is updated here . When I run the requestor script it exits with a runtime error 2 (File not found). I am not sure how to further debug this or fix it. So far I've converted my code over to a python slim docker image to better mirror the example. It also works for me when I spin up a docker image that typing and running "/golem/work/imageclassifier.py --trainmodel" works from root. I switched all my code to use absolute paths. I also did make sure the shebang (#!) uses linux end of file characters rather than windows before which was giving me errors. Fixed a bug where my script returns error code 2 when called with no args to now pass.
clf.fit(trainDataGlobal, trainLabelsGlobal)
pkl_file = "classifier.pkl"
with open(pkl_file, 'wb') as file:
pickle.dump(clf, file)
is the only piece I could think of that causes the issue, but as far as I can tell this is the proper way to pickle something in python. Requestor script is also heavily based on the simple service example and I tried to mirror my design to that. I just need help in getting more information while debugging, or guidance on how to move forward from here
I am trying to run a model in GAMS (using the Windows GAMS 64, version 24.5). The model .gms file is located on a Z drive that is mapped to a SharePoint folder. We use this setup all the time with other programming languages without problem. But when I try to run or save my model, I get the following error:
Unable to save file:
Z:\[my file path]\mymodel.gms
Error 123: The filename, directory name, or volume label syntax is incorrect
When I try to run the file, I also get the following message in the process window:
*** Error: Error writing GAMSNext: Invalid argument
Exit code = 113
SharePoint is really useful in tracking versions and letting the teams collaborate using a tool they are already familiar with (and our IT dept will not let us install git or similar). So I'd like to figure out how to get this to work, if possible, and why it is only a problem in GAMS.
Any insights? In particular, anyone know what the GAMSNext file that GAMS seems to be trying to write is? Also, why is GAMS perfectly able to save files to SharePoint when I create my project file (the .gpr file was, in fact, saved just fine), but not when I try to save a change to my .gms file? What is the difference in the GAMS save process for a .gms file and a .gpr file?
I have seen this issue before - GAMS not running a file on a network path/drive and give a 'UNC path error' or so.
If I recall, the issue was solved when we took the project file out of the "My Documents" folder. If it is a shared drive created for windows, you may have a "My Documents" folder which has a path with "Documents". Create your GAMS folder from the root and create your .gpj and .gms files there.
Hope this helps.
I've an SSIS package to load excel file from network drive. It's designed to load content and then move the file to archived folder.
Everything works good when the following SQL statement runs in SSMS window.
However when it's copied to SQL agent job and executes from there, the file is neither loaded nor moved. But it shows "successful" from the agent log.
The same thing also happened to "SSIS job" instead of T-SQL job, even with proxy of windows account.(same account as ssms login)
Declare #execution_id bigint
EXEC [SSISDB].[catalog].[create_execution] #package_name=N'SG_Excel.dtsx', #execution_id=#execution_id OUTPUT, #folder_name=N'ETL', #project_name=N'Report', #use32bitruntime=True, #reference_id=Null
Select #execution_id
DECLARE #var0 smallint = 1
EXEC [SSISDB].[catalog].[set_execution_parameter_value] #execution_id, #object_type=50, #parameter_name=N'LOGGING_LEVEL', #parameter_value=#var0
EXEC [SSISDB].[catalog].[start_execution] #execution_id
GO
P.S. At first relative path of network drive is applied, then switched to absolute path(\\server\folder). It's not solving the issue.
SSIS Package Jobs run under the context of the SQL Server Agent. What Account is setup to run the SQL Server Agent on the SQL Server? It may need to be run as a Domain account that has access to the network share.
Or you can copy the Excel file to local folder on the SQL Server, so the Package can access the file there.
Personally I avoid the File System Task - I have found it unreliable. I would replace that with a Script Task, and use .NET methods from the System.IO namespace e.g. File.Move. These are way more reliable and have mature error handling.
Here's a starting point for the System.IO namespace:
https://msdn.microsoft.com/en-us/library/ms404278.aspx
Be sure to select the relevant .NET version using the Other Versions link.
When I have seen things like this in the past it's been that my package isn't accessing the path I thought it was at run time, its looking somewhere else, finding an empty folder & exiting with success.
SSIS can have a nasty habit of going back to variable defaults . It may be looking at a different path you used in dev? Maybe hard code all path values as a test? or put in break points & double check the run time values of all variables & parameters.
Other long shots may be:
Name resolution, are you sure the network name is resolving correctly at runtime?
32/64 bit issues. Dev tends to run 32 bit, live may be 64 bit. May interfere with file paths? Maybe force to 32 bit at run time?
There is issue with sql statement not having statement terminator (;) that is causing issue.
Declare #execution_id bigint ;
EXEC [SSISDB].[catalog].[create_execution] #package_name=N'SG_Excel.dtsx', #execution_id=#execution_id OUTPUT, #folder_name=N'ETL', #project_name=N'Report', #use32bitruntime=True, #reference_id=Null ;
Select #execution_id ;
DECLARE #var0 smallint = 1 ;
EXEC [SSISDB].[catalog].[set_execution_parameter_value] #execution_id, #object_type=50, #parameter_name=N'LOGGING_LEVEL', #parameter_value=#var0 ;
EXEC [SSISDB].[catalog].[start_execution] #execution_id ;
GO
I have faced similar issue in service broker ..
everyone,
when I deploy my package to a linux environment, I met this error:
.../Linux-2.6c2.5-i686/Ncurses/Ncurses-15766.0-0/lib/libncurses.so.5 is encountered a second time at /apollo/_env/FBAMerchantAutoRemovalDaemon-swit1na.1755067.237551097.1107633519/perl/lib/perl5.8-dist/File/Find.pm line 542.
though I read the perl script, I have no idea what is wrong. I suspect my environment is tainted. Does anyone have idea what is wrong and how can I debug this problem? Thanks a lot in advance!
Zhe
From perldoc File::Find
follow
Causes symbolic links to be followed. Since directory trees with symbolic links (followed) may contain files more than once and may even have cycles, a hash has to be built up with an entry for each file. This might be expensive both in space and time for a large directory tree. See "follow_fast" and "follow_skip" below. If either follow or follow_fast is in effect:
It is guaranteed that an lstat has been called before the user's wanted() function is called. This enables fast file checks involving _. Note that this guarantee no longer holds if follow or follow_fast are not set.
There is a variable $File::Find::fullname which holds the absolute pathname of the file with all symbolic links resolved. If the link is a dangling symbolic link, then fullname will be set to undef.
So, if, for the purposes of your application, if it is OK to follow symlinks, invoke find with the follow option set:
find({ wanted => \&process, follow => 1 }, $dir);
Or, consider if one of the other follow_skip behaviors is more appropriate for your application:
follow_skip
follow_skip==1, which is the default, causes all files which are neither directories nor symbolic links to be ignored if they are about to be processed a second time. If a directory or a symbolic link are about to be processed a second time, File::Find dies.
follow_skip==0 causes File::Find to die if any file is about to be processed a second time.
follow_skip==2 causes File::Find to ignore any duplicate files and directories but to proceed normally otherwise.
It may be that follow_skip => 2 is more appropriate for your application. Only you can make that decision.
I have a VS2012 solution, containing 10 projects, and suddenly, I can no longer publish my Services project to any folder.
When I try to publish to D:\temp, I get this error:
The expression "[System.IO.Path]::GetFullPath(obj\Release%25252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252528Prod%25252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252529\)" cannot be evaluated. The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters. C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\Web\Microsoft.Web.Publishing.targets
Huh ?
VS2012 (with update 4) seems to have taken my configuration name "Release(Prod)" and completely messed it up, causing the GetFullPath to produce too long a path name.
How the heck can I fix this ?
Out of desperation, I tried to build and publish the same project in VS2013 - and it had the same error message.
One of my colleagues said he'd seen the same thing, but had fixed it by removing the spaces from his configuration name. I tried this, which is why my configuration name is now "Release(Prod)" rather than "Release (Prod)", but it made no difference.
I did also open the file which this error is suggesting is the cause of the error:
C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\Web\Microsoft.Web.Publishing.targets
..and noticed that there's something in there concerning the AnyCPU platform name. I have tried getting my Services project to use "AnyCPU" and "Any CPU" (depressed sigh) but neither seems to make any difference.
<PropertyGroup Condition=" '$(IntermediateOutputPath)' == '' ">
<IntermediateOutputPath Condition=" '$(PlatformName)' == 'AnyCPU' Or '$(PlatformName)' == ''">$(BaseIntermediateOutputPath)$(Configuration)\</IntermediateOutputPath>
. . .
</PropertyGroup>
Has anyone else seen this issue ?
(A little bit later..)
This is so odd (and frustrating).
My Solution has 5 configurations - the default Debug and Release ones, plus extra configurations for Test, PreProduction and Production environments.
If I select any of these three configurations containing brackets, I get this ridiculous "The specified path is too long" error, as VS2012 corrupts the pathname (as shown in my first screenshot above).
I can't help wondering... is this some kind of VS2012 bug, handling spaces or brackets in the configuration name ?
I can deploy to a path directly with (, (, )) in the Target Location on tyhe Connection tab when publishing to the file system (i.e. not building a path from the configuration name) - but that is not a solution to targeting different locations based on the Configuration.
If you want to keep special characters in the configuration name, but specify a path to the deployment folder that will not cause and issue this post might help: Visual Studio: How to properly build and specify the configurations and platforms for x64 and x86
Specifically play with the settings in here:
In the project properties page, select the various permutations of
Debug/Release and x86/x64 in the solution dropdowns. Make sure the
target processor is set correctly (it should be, but I found instances
when they were not, probably because of my previous attempts). Also,
set the output directory. That should be okay and automatic
(/bin/x86/Debug, etc.). If not, fix.
Looking at what is actually seems to be going on is also potentially useful:
Looking at the numbers inserted:
%25 is an encoded %,
%28 is an encoded (
%29 is an encoded )
Looking at the path:
obj\Release%252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525***28***Prod%252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525***29***)
What I think is happening:
So that seems like a good clue that these are being URL or XML encoded. What appears to be happening is that the ( is being encoded as %28 and then the % is being recursively encoded as %25 - generating an infinite %252525252525252525....
A more interesting question is actually why it stops creating 25's from the %'s with this bug (both times it stops creating 25's at 214 characters including the % and the 28 / 29 - not a very interesting number).
Looking at the file C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\Web\Microsoft.Web.Publishing.targets you reference - it makes sense that these strings are being encoded for XML. I would say this is definitely a bug... I have no suggestions for a fix.
Well, I'm going to accept Matthew's answer as the "Accepted Answer".
Thank you for your help.
This is a really odd problem though, and I'm amazed no one else has reported this elsewhere.
Summary of problem (in case Microsoft is interested, or if anyone tries to Google this issue in the coming years)
With a configuration name of "Release (Prod)", I could happily build my code, run it locally, but when I tried to publish it, even to a local drive, I'd get this message:
It's an odd exception, because the Build did create the "obj\Release (Prod)" folder, without any issues. It's just the Publish which seemed to be looking in the wrong place for it.
Following the advice given in this thread, today I attempted to create a new configuration, with the same settings, but without a space in the name: "Release(SecondProd)". Look what happens:
Interestingly, despite this error, it did create a new configuration with this name.
Anyway, I recreated a new configuration, called it ReleaseProduction, and it worked fine.
Of course, I needed to create new "web.config" Transformation for this name, as this doesn't get automatically copied when you create a new configuration based on an old one.
One last thought (just to confuse matters worse !)
When I posted this plea-for-help, the Services project in my Solution refused to publish to a local drive, but my web site would publish okay.
Today, two days since I last attempted a website Publish, I found that the website also now produces the same GetFullPath exception. Nothing's changed ! We use TFS, I have done a file compare with my project files today against two days ago, and they're identical !
It's a really bizarre bug in VS2012 & VS2013.
Btw, this solution & the projects in it, were originally a VS2010 project. They were upgraded to VS2012 over a year ago, but this Publish problem only started happening recently. I'm not sure if the problem is related to using upgraded VS projects.
Again, thanks for your help.
Now I have some Test, PreProd & Production configurations to recreate !
Maybe I'll grab a beer first..
Summarizing and completing Matthew's answer:
Cause: You have configurations with chars that require URI-encoding - in your case, '(' and ')'.
Workaround: Rename those configs.
What happens: Presumably web deploy URI-encodes the path, replacing % => %25, ( => %28, ) => %29. It does so over and over:
obj\Release(Prod)
obj\Release%28Prod%29
obj\Release%2528Prod%2529
obj\Release%252528Prod%252529
...
Until the path exceeds MAX_PATH=260.
I ran into the same thing and all though it does not resolve the issue I found that if I switch the solution configuration away from a build containing "(" or ")"
Then use the appropriate build in the publish dialog it will not error out.