Python tempfile.TemporaryFile hangs on Windows when no write privilege - python-3.x

My environment is Python 3.7.2, running on Windows 10. I'm working on a directory-selection widget, and I'm looking for the cleanest+surest method to test whether the selected directory path allows write privilege.
Previously I'd been opening a named file by the usual open() method, writing a few bytes to it, then deleting it -- putting the whole thing in a try-except block. This was OK but it ran the risk of leaving behind an unwanted file. Recently I came across the documentation for tempfile.TemporaryFile(), and this seemed cleaner way to get the same result, with no risk of leaving junk files on the system.
The problem is, tempfile.TemporaryFile() hangs on my system when it's given a dir parameter that's a read-only folder. I've googled around and found this very old bug, but it was written against Python 2.4 and was fixed long ago.
Here's a test script I put together to illustrate the issue. (Note that I've omitted the file-delete that my actual app performs, as it's not relevant for the illustration.)
import os, tempfile
def normOpen(checkPath):
try:
with open(os.path.join(checkPath,'x.txt'),'wb') as tf:
tf.write(b'ABC')
except Exception as e:
print('Write disabled for '+checkPath)
print(str(e))
else:
print('Write enabled for '+checkPath)
def tfOpen(checkPath):
try:
with tempfile.TemporaryFile(dir=checkPath) as tf:
tf.write(b'ABC')
except Exception as e:
print('Write disabled for '+checkPath)
print(str(e))
else:
print('Write enabled for '+checkPath)
tryPath1 = 'C:\\JDM\\Dev_Python\\TMPV\\canwrite' #Full control path
tryPath2 = 'C:\\JDM\\Dev_Python\\TMPV\\nowrite' #Read-only path
print('First method - normal file-open')
normOpen(tryPath1)
normOpen(tryPath2)
print('Second method - TemporaryFile')
tfOpen(tryPath1)
tfOpen(tryPath2)
When I run this script, it hangs on the last line and just sits there (Task Manager shows Python consuming about 10-15% CPU).
Does anyone know what the problem might be? Particularly is this a Python bug, or is there something wrong with my usage of TemporaryFile?
In case it helps, below is the specific privileges Windows shows for each of these folders:

A deeper dive than I'd initially done, turned up the answer. This is indeed a Python bug, reported some time ago but which remains to be addressed.
The comments from eryksun describe the details -- and it's what prompted me to take a closer look at the Python bug database -- so ultimately that's where credit is due. I'm just filling it in here to get the question answered and closed out.
The bug affects only Windows environments, but unfortunately it has the result of rendering tempfile.TemporaryFile unusable on Windows for this common use case.

Related

7687 INFO: Extra DLL search directories (path): stuck at this part in pyinstaller , what solution?

[
how to solve it, i've been trying for days
](https://i.stack.imgur.com/gOSQ6.jpg)
enter image description here
how to solve it, i've been trying for days
Had the same issue, pyinstaller stuck at INFO: Extra DLL search directories (PATH)
Fixed by patching Lib\site-packages\pyinstaller\isolated\_parent.py on line #233 (see original code on github
try:
self._child.wait(timeout=5)
except subprocess.TimeoutExpired:
self._child.kill()
For some reason the _child.wait() in the original code was hanging forever.
Possibly due to deadlock with PIPE when using .wait() as described here

Golem Task respons back with runtime error 2, can't determine the cause

Repo for all code I've been using is updated here . When I run the requestor script it exits with a runtime error 2 (File not found). I am not sure how to further debug this or fix it. So far I've converted my code over to a python slim docker image to better mirror the example. It also works for me when I spin up a docker image that typing and running "/golem/work/imageclassifier.py --trainmodel" works from root. I switched all my code to use absolute paths. I also did make sure the shebang (#!) uses linux end of file characters rather than windows before which was giving me errors. Fixed a bug where my script returns error code 2 when called with no args to now pass.
clf.fit(trainDataGlobal, trainLabelsGlobal)
pkl_file = "classifier.pkl"
with open(pkl_file, 'wb') as file:
pickle.dump(clf, file)
is the only piece I could think of that causes the issue, but as far as I can tell this is the proper way to pickle something in python. Requestor script is also heavily based on the simple service example and I tried to mirror my design to that. I just need help in getting more information while debugging, or guidance on how to move forward from here

create exception when python command generates a program.exe has stopped working type error

I am facing a problem with a program i am developing in Python 3.6 under Windows 10.
One particular command generates an unknown error, windows throws a 'program.exe has stopped working' message and the program exits.
The command is a 3d-model loader that is part of another python package (Panda3D). The crash is always associated with this command (and more particularly with a specific dll of the loader) and a particular file that it tries to open.
Since i cannot locate and therefore solve the faults in the dll (probably there is a bug there) i would like to just pass the problematic file and continue to the next one. But since python exits and i do not know the error type, the typical try, except does not work.
So, i would like to know if there is a way to predict this type of behavior in my code and prevent the program from exiting.
Many thanks for any help.
The pop-up "Program.exe has stopped working." can be caused by a variety of things and therefor there is no "one size fits all" type solution. But if you're certain that your problem is cause by a specific line of code you can always try something along the lines of :
try:
loader.loadModel("c/path/to/your/file")
except Exception as e:
print(e.message, e.args)
# your error-handling code here
Make sure the file path that you're giving to loadModel respects the following :
# WRONG:
loader.loadModel("c:\\Program Files\\My Game\\Models\\Model1.egg")
# RIGHT:
loader.loadModel("/c/Program Files/My Game/Models/Model1.egg")
Source : pandas3d official documentation

SocketNER freezes in Stanford NER

Or, The Mystery of Line #2598.
I was trying to run Stanford-NER on a local server. I wrote some scripts, using a bit of PyNER. I then downloaded some of Sherlock Holmes' books off Project Gutenberg, and started running the program. Set up a server on the PC with a 1000MB RAM and used this
tagger = ner.SocketNER(host='192.168.xxx.xxx', port=xxxx)
The code runs smoothly, till it reaches line#2598. After that, without raising any error, or even terminating, the program does not proceed. I thought it would be an issue with the processing capacity/memory of the server, so I set up a friend's PC with the same configurations, and ran the script to hit both the servers alternately.
if count % 2:
get_entities(str(l), tagger_1)
else:
get_entities(str(l), tagger_2)
Still, it froze at Line#2598. Not the one to give up, I did this
if count % 2:
tagger_1 = ner.SocketNER(host='192.168.xxx.xxx', port=xx)#k
get_entities(str(l), tagger_1)
del tagger_1
else:
tagger_2 = ner.SocketNER(host='192.168.xxx.yyy', port=xx)#v
get_entities(str(l), tagger_2)
del tagger_2
Still, Line #2598 freezes. On another book, it is line#2212, so it's not the number of times I connect. Neither is it the number of characters. The lack of errors baffles me. Is there an underlying limit enforced on the network? Thank you for any hint. The issue stays the same for IDLE as well as ST3.

Runtime error R6034 in embedded Python application

I am working on an application which uses Boost.Python to embed the Python interpreter. This is used to run user-generated "scripts" which interact with the main program.
Unfortunately, one user is reporting runtime error R6034 when he tries to run a script. The main program starts up fine, but I think the problem may be occurring when python27.dll is loaded.
I am using Visual Studio 2005, Python 2.7, and Boost.Python 1.46.1. The problem occurs only on one user's machine. I've dealt with manifest issues before, and managed to resolve them, but in this case I'm at a bit of a loss.
Has anyone else run into a similar problem? Were you able to solve it? How?
The problem was caused by third-party software that had added itself to the path and installed msvcr90.dll in its program folder. In this case, the problem was caused by Intel's iCLS Client.
Here's how to find the problem in similar situations:
Download Process Explorer here.
Start your application and reproduce runtime error R6034.
Start Process Explorer. In the "View" menu go to "Lower Pane View" and choose "DLLs".
In the top pane, locate your application and click on it. The bottom pane should show a list of DLLS loaded for your application.
Locate "msvcr??.dll" in the list. There should be several. Look for the one that is not in the "winsxs" folder, and make a note of it.
Now, check the path just before your application runs. If it includes the folder you noted in step 5, you've probably found the culprit.
How to fix the problem? You'll have to remove the offending entry from the path before running your program. In my case, I don't need anything else in the path, so I wrote a simple batch file that looks like this:
path=
myprogram.exe
That's it. The batch file simply clears the path before my program runs, so that the conflicting runtime DLL is not found.
This post elaborates on #Micheal Cooper and #frmdstryr and gives a better alternative than my earlier answer.
You can put the following in front of a python script to purge the problematic entries.
import os, re
path = os.environ['PATH'].split(';')
def is_problem(folder):
try:
for item in os.listdir(folder):
if re.match(r'msvcr\d\d\.dll', item):
return True
except:
pass
return False
path = [folder for folder in path if not is_problem(folder)]
os.environ['PATH'] = ';'.join(path)
For the vim with YouCompleteMe case, you can put the following at the top of your vimrc:
python << EOF
import os, re
path = os.environ['PATH'].split(';')
def is_problem(folder):
try:
for item in os.listdir(folder):
if re.match(r'msvcr\d\d\.dll', item):
return True
except:
pass
return False
path = [folder for folder in path if not is_problem(folder)]
os.environ['PATH'] = ';'.join(path)
EOF
A more general solution is:
import os
os.environ['path'] = ";".join(
[path for path in os.environ['path'].split(";")
if "msvcr90.dll" not in map((lambda x:x.lower()), os.listdir(path))])
(I had the same problem with VanDyke SecureCRT)
(This might be better as a comment than a full answer, but my dusty SO acct. doesn't yet have enough rep for that.)
Like the OP I was also using an embedded Python 2.7 and some other native assemblies.
Complicating this nicely was the fact that my application was a med-large .Net solution running on top of 64-Bit IIS Express (VS2013).
I tried Dependency Walker (great program, but too out of date to help with this), and Process Monitor (ProcMon -- which probably did find some hints, but even though I was using filters the problems were buried in thousands of unrelated operations, better filters may have helped).
However, MANY THANKS to Michael Cooper! Your steps and Process Explorer (procexp) got me quickly to a solution that had been dodging me all day.
I can add a couple of notes to Michael's excellent post.
I ignored (i.e. left unchanged) not just the \WinSxS\... folder but also the \System32\... folder.
Ultimately I found msvcr90.dll being pulled in from:
C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x64
Going through my Path I found the above and another, similar directory which seemed to contain 32-bit versions. I removed both of these, restarted and... STILL had the problem.
So, I followed Michael's steps once more, and, discovered another msvcr90.dll was now being loaded from:
C:\Program Files\Intel\iCLS Client\
Going through my Path again, I found the above and an (x86) version of this directory as well. So, I removed both of those, applied the changes, restarted VS2013 and...
No more R6034 Error!
I can't help but feel frustrated with Intel for doing this. I had actually found elsewhere online a tip about removing iCLS Client from the Path. I tried that, but the symptom was the same, so, I thought that wasn't the problem. Sadly iCLS Client and OpenCL SDK were tag-teaming my iisexpress. If I was lucky enough to remove either one, the R6034 error remained. I had to excise both of them in order to cure the problem.
Thanks again to Michael Cooper and everyone else for your help!
Using Michael's answer above, I was able to resolve this without a bat file by adding:
import os
# Remove CLS Client from system path
if os.environ['PATH'].find("iCLS Client")>=0:
os.environ['PATH'] = "".join([it for it in os.environ['PATH'].split(";") if not it.find("iCLS Client")>0])
to the main python file of the application. It just makes sure system path didn't include the paths that were causing the issue before the libraries that loaded the dll's were imported.
Thanks!
This post elaborates on #Micheal Cooper and #frmdstryr.
Once you identified the problematic PATH entries, you can put the following in front of a
python script, assuming here that iCLS Client and CMake are problematic.
import os
for forbidden_substring in ['iCLS Client', 'CMake']:
os.environ['PATH'] = ';'.join([item for item in os.environ['PATH'].split(';')
if not item.lower().find(forbidden_substring.lower()) >= 0])
Concerning the vim with YouCompleteMe case, you can put the following at the top of your vimrc:
python << EOF
import os
for forbidden_substring in ['iCLS Client', 'CMake']:
os.environ['PATH'] = ';'.join([item for item in os.environ['PATH'].split(';')
if not item.lower().find(forbidden_substring.lower()) >= 0])
EOF
If none of these solutions is applicable for you, you can try to remove the problem causing
entries from you PATH manually, but you want to make sure you don't break anything else on your
system that depends on these PATH entries. So, for instance, for CMake you could try to remove
its PATH entry, and only put a symlink (or the like) pointing to the cmake.exe binary into some
other directory that is in your PATH, to make sure cmake is still runnable from anywhere.
Thanks for the solution. I just little modified this sample code as the path variable in my system contains the string "ICLS CLIENT" instead of "iCLS Client"
import os
# print os.environ['PATH']
# Remove CLS Client from system path
if os.environ['PATH'].find("iCLS Client") >= 0 or os.environ['PATH'].find("ICLS CLIENT") >= 0:
os.environ['PATH'] = "".join([it for it in os.environ['PATH'].split(";") if not (it.find("iCLS Client")>0 or it.find("ICLS CLIENT")>0)])
I also had the same problem with embedding Python27.dll from a C-program using the Universal-CRT.
A <PYTHON_ROOT>\msvcr90.dll was the offender. And <PYTHON_ROOT> is off-course in my PATH. AFAICS the only users of msvcr90.dll are the PyWin32 modules
<PYTHON_ROOT>\lib\site-packages\win32\win32*.pyd.
The fix was just move <PYTHON_ROOT>\msvcr90.dll to that directory.
PS. PyWin32 still has this as an issue 7 years later!
In my case the rebuilding of linked libraries and the main project with similar "Runtime execution libraries" project setting helped. Hope that will be usefull for anybody.
In my case, I realised the problem was coming when, after compiling the app into an exe file, I would rename that file. So leaving the original name of the exe file doesn't show the error.
The discussion on this page involves doing things way far advanced above me. (I don't code.) Nevertheless, I ran Process Explorer as the recommended diagnostic. I found that another program uses and needs msvcr90.dll in it's program folder. Not understanding anything else being discussed here, as a wild guess I temporarily moved the dll to a neighboring program folder.
Problem solved. End of Runtime error message.
(I moved the dll back when I was finished with the program generating the error message.)
Thank you all for your help and ideas.
Check any library having user specified path by Process Explorer. It is not necessary must be msvcr??.dll
I solved same problem except I run Python 3. Present solutions not helped because they not indicate unusual paths of msvcr90.dll. I debug code step by step inside till error dialog appears after rows (called when my code was importing PyTables module):
import ctypes
ctypes.cdll.LoadLibrary('libbz2.dll')
Then Process Explorer helps to find path to old libbz2.dll caused the problem (steps 3, 4 of #Micheal Cooper algorithm)
Adding this answer for who is still looking for a solution. ESRI released a patch for this error. Just download the patch from their website (no login required), install it and it will solve the problem. I downloaded the patch for 10.4.1 but there are maybe patches for other versions also.

Resources