I have a Python app using Pygraphviz that works fine locally, but on the server the draw function throws an error. It happens in make_svg. The following lines are the relevant part of the errors I get. (The full trail is here.)
File "/path/to/app/utils/make_svg.py", line 17, in make_svg
prog='dot'
File "/path/to/pygraphviz/agraph.py", line 1477, in draw
fh = self._get_fh(path, 'w+b')
File "/path/to/pygraphviz/agraph.py", line 1506, in _get_fh
fh = open(path, mode=mode)
FileNotFoundError: [Errno 2] No such file or directory: 'app/svg_files/nope.svg'
Logging type(g) gives <class 'pygraphviz.agraph.AGraph'> as expected.
I work in a virtualenv in a mod_wsgi 4.6.5/Python3.7 environment on a Webfaction server.
Locally I use a virtualenv with Python 3.5.
The version of Pygraphviz is 1.3.1.(First I had 1.5 on the server. The error was exactly the same, except for the line numbers.)
What can I do?
The same error is described in this bug report from last year.
I don't get which directory I am supposed to create. svg_files exists and has rights 777.
The draw function at the end of make_svg should create the SVG.(And at the end of extract_coordinates_from_svg the file is removed again.)The file name is a hash created in connected_dag (svg_name).
On the server app/svg_files seems not to describe the same place as locally.
I defined the path unambiguously, and now it works.
file_path = '{grandparent}/svg_files/{name}.svg'.format(
grandparent=os.path.dirname(os.path.dirname(__file__)),
name=name
)
g.draw(file_path, prog='dot')
Related
I wish I could use python to execute the Excel macro, so I tried to use the package xlwings to implement it.
The OS of my laptop is macOS Catalina (ver.: 10.15.7), my compiler is PyCharm (ver.: 2021.2.3), my Python version is 3.8.8, I used Anaconda (Ver.: 22.11.1) as my interpreter, my excel version is 16.66.1 (Microsoft Excel for Mac).
I faced the error "Command Error -1743: The User has declined permission" when I tried to use this package originally, and I solved this issue by installing an old compiler & using the old version of the compiler to run my code. My privacy setting for automation in the app Setting was shown below: (This is NOT the question I want to ask, but I'm not sure if it is also related to the issue I faced, so I still attached it here. I had uninstalled the old version of my compiler already.)
I wish I could implement an existing macro (called Hi) in an existing Excel file (called [StakeOverflow]HelloWorld.xlsm) through Python (represented as MY_PYTHON_FILE.py below), like the snapshot below:
My Excel file and my python code were stored on OneDrive. My macro code was shown below:
Sub Hi()
MsgBox "Good morning!"
End Sub
My python code was shown below:
import xlwings as xw
import time
wb = xw.Book('/Users/<MY NAME>/OneDrive/MY PATH DETAILS/[StakeOverflow]HelloWorld.xlsm')
time.sleep(10)
app = wb.app
macro_vba = app.macro("Hi")
macro_vba()
The code looks really simple, but I still faced the error. My Excel was not opened automatically, and I even could not open the Excel file manually thereafter. The error was shown below:
/Users/.../.conda/envs/Program/bin/python "/Users/.../OneDrive/.../MY_PYTHON_FILE.py"
Traceback (most recent call last):
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/xlwings/main.py", line 4914, in open
impl= self.impl(name)
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/xlwings/_xlmac.py", line 366, in __call__
raise KeyError(name_or_index)
KeyError: '[stakeoverflow] helloworld.xlsm'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/aeosa/appscript/reference.py", line 482, in __call___
return self.AS_appdata.target().event (self._code, params, atts, codecs=self.AS_appdata).send(timeout, sendflags)
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/aeosa/aem/aemsend.py", line 92, in send
raise EventError(errornum, errormsg, eventresult)
aem.aemsend.EventError: Command failed: Parameter error. (-50)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/.../OneDrive/.../MY_PYTHON_FILE.py", line 18, in <module>
wb = xw.Book('/Users/.../OneDrive/.../[stakeoverflow] helloworld.xlsm')
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/xlwings/main.py", line 876, in __init__
impl= app.books.open(
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/xlwings/main.py", line 4921, in open
impl = self.impl.open(
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/xlwings/ xlmac.py", line 420, in open
self.app.xl.open_workbook(
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/aeosa/appscript/reference.py", line 518, in __call__
raise CommandError(self, (args, kargs), e, self.AS_appdata) from e
appscript.reference.CommandError: Command failed:
OSERROR: -50
MESSAGE: Parameter error.
COMMAND: app(pid=1647).open_workbook (workbook_file_name='/users/.../onedrive/.../[stakeoverflow] helloworld.xlsm', update_links=k.do_not_update_links, read_only=None, format=None, password=None, write_reserved_password=None, ignore_read_only_recommended=None, origin=None, delimiter=None, editable=None, notify=None, converter=None, add_to_mru=None, timeout=-1)
Process finished with exit code 1
I tried to use terminal to run my python code, but I could not solve the problem, either.
I tried to open the Excel file manually thereafter, the error was shown below:
Excel cannot open the file ’[StakeOverflow]HelloWorld.xlsm’
because the file format or file extension is not valid. Verify
that the file has not been corrupted and that the file
extension matches the format of the file.
I tried to Google this error, but few solutions was found. It seems that it is related to the issue of external storage location. I tried to create another Excel file with the same name & macro code on my desktop and try again, and the error would disappear. (However, our company stored the files on OneDrive, so I wish I could utilise the file online.)
Just wondering if anyone here faced this situation before?
I found the answer by myself today. The issue is related to the naming issue rather than the permission issue.
As we could see that the error is KeyError: '[stakeoverflow] helloworld.xlsm', which implies that the problem is here. (Maybe because the system could not find the Excel file with this name.)
I tried to change the name from [stakeoverflow] helloworld.xlsm to helloworld.xlsm, and the error was gone. It seems that when xlwings want to open an Excel file, it would check the validity of the file name and change the name into smaller cases. If the file name contains special characters which are not allowed (e.g., "[]"), then the error would occur.
Notice that I could store the Excel file with these special characters in our laptop & OneDrive, but xlwings did not accept them.
Hope it is helpful to those who face this issue when using xlwings!
I am trying to run this code locally:
https://gist.github.com/shashank524/74d8f46d5de633b84e2265fcc34774de#file-tabledetection-ipynb
After installing required packages, when I am trying to run this line:
import layoutparser as lp
# PubLayNet
model = layoutparser.Detectron2LayoutModel('lp://PubLayNet/faster_rcnn_R_50_FPN_3x/config',extra_config=["MODEL.ROI_HEADS.SCORE_THRESH_TEST", 0.81],label_map={0: "Text", 1: "Title", 2: "List", 3:"Table", 4:"Figure"})
I receive this error:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\username/.torch/iopath_cache\\s/f3b12qc4hc0yh4m\\config.yml?dl=1.lock'
I looked into the directory and there was no config file available.
I tried to download the config file from here (https://layout-parser.readthedocs.io/en/latest/notes/modelzoo.html) and put it in the directory but it didn't solve the issue!
Even I got a similar error. I tried out manually some work around in Windows.
I am using your case as example: OSError: [Errno 22] Invalid argument: 'C:\Users\username/.torch/iopath_cache\s/f3b12qc4hc0yh4m\config.yml?dl=1.lock'
Please follow the following process.
Navigate to C:\Users\username/.torch/iopath_cache\s/f3b12qc4hc0yh4m\config.yml
Open that config.yaml file
Scroll down to WEIGHTS: https://www.dropbox.com/s/h7th27jfv19rxiy/model_final.pth?dl=1 should be around 265 line.
Copy that link and paste it in your browser, a 'model_final.pth' will be downloaded. Copy this file to your desired folder.
Now replace the path to WEIGHTS: your_desired_folder\model_final.pth
Save it and run the code it works!
But there is a small work around I think before you do this (if you have not done)
iopath work around
https://github.com/Layout-Parser/layout-parser/issues/15 (Github link to the issue)
Hope this helps!
I have recently saved a model into s3 using joblib
model_doc is the model object
import subprocess
import joblib
save_d2v_to_s3_current_doc2vec_model(model_doc,"doc2vec_model")
def save_d2v_to_s3_current_doc2vec_model(model,fname):
model_name = fname
joblib.dump(model,model_name)
s3_base_path = 's3://sd-flikku/datalake/current_doc2vec_model'
path = s3_base_path+'/'+model_name
command = "aws s3 cp {} {}".format(model_name,path).split()
print('saving...'+model_name)
subprocess.call(command)
It was successful, but after that when i try to load the model back from s3 it gives me an error
model = load_d2v("doc2vec_model")
def load_d2v(fname):
model_name = fname
s3_base_path='s3://sd-flikku/datalake/current_doc2vec_model'
path = s3_base_path+'/'+model_name
command = "aws s3 cp {} {}".format(path,model_name).split()
print('loading...'+model_name)
subprocess.call(command)
model=joblib.load(model_name)
return model
This is the error i get:
loading...doc2vec_model
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in load_d2v
File "C:\Users\prane\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 339, in call
with Popen(*popenargs, **kwargs) as p:
File "C:\Users\prane\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 800, in __init__
restore_signals, start_new_session)
File "C:\Users\prane\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 1207, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
I don't even understand why it is saying File not found, this was the path i used to save the model but now i'm unable to get the model back from s3. Please help me!!
I suggest that rather than your generic print() lines, showing your intent, you should print the actual command you've composed, to verify that it makes sense upon observation.
If it does, then also try that exact same aws ... command directly, at the command prompt where you had been launching your python code, to make sure it runs that way. If it doesn't, you may get a more clear error.
Note that the error you're getting doesn't particularly look like it's coming from the aws command, of from the S3 service - which might talk about 'paths' or 'objects'. Rather, it's from the Python subprocess system & Popen' call. I think those are via your call tosubprocess.call(), but for some reason your line-of-code isn't shown. (How are you running the block of code with theload_d2v()`?)
That suggests the file that's no found might be the aws command itself. Are you sure it's installed & runnable from the exact working-directory/environment that your Python is running in, and invoking via subprocess.call()?
(BTW, if my previous answer got you over your sklearn.externals.joblib problem, it'd be good for you to mark the answer as accepted, to save other potential answerers from thinking that's still an unsolved question that's blocking you.)
try to add extension of your model file to your fname if you are confident the model file is there.
e.g. doc2vec_model.h3
Attempting to execute shutil.rmtree(path) on a directory managed by NFS consistently fails. Below you can see that os.rmdir(path) within shutil.rmtree(path) causes the exception. Is there a more robust way for me to achieve the expected result?
It appears that it removes all of the files, yet a hidden .nfs file remains in the directory for a short amount of time. I'm guessing that the process from which I'm calling rmtree has an open file handle to one of the files inside the directory, which, when deleted, apparently causes NFS to write a new hidden file. That would cause os.rmdir to fail on attempting to remove a non-empty directory.
Traceback (most recent call last):
File "/home/me/pre3/lib/python3.6/shutil.py", line 484, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "/home/me/pre3/lib/python3.6/shutil.py", line 482, in rmtree
os.rmdir(path)
OSError: [Errno 39] Directory not empty:
NFS details:
$ nfsstat -m
/home/me/nfs from XXX.YYY.ZZZ:/mnt/path/to/nfs
Flags: rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=50,retrans=2,sec=sys,mountaddr=REDACTED,mountvers=3,mountport=832,mountproto=udp,local_lock=none,addr=REDACTED
I'm using Python 3.6.6 on Ubuntu 16.04.
If the python logging module is logging to the target output directory, it will maintain an open file. A workaround is to call logging.shutdown() first, then called shutil.rmtree(path). This is not a general answer to the broader question, however.
You could try defining an error handler function to be passed to the onerror arg for shutil.rmtree: https://docs.python.org/3/library/shutil.html#shutil.rmtree
def handle_rmtree_err(function, path, excinfo):
...
shutil.rmtree(my_path, onerror=handle_rmtree_err)
There are all sorts of reasons why a process may be holding onto a file, so I can't tell you what the error handler should do exactly.
If you haven't figured out what is holding onto the file, try $ lsof | grep .nfsXXXX.
If all else fails you could time.sleep(secs) and retry shutil.rmtree.
I'm trying to get the largefiles extension working on a mercurial server under Windows Server 2008 / IIS 7.5 with the hgweb.wsgi script.
When I clone a repo with largefiles locally (but using https://domain/, not a file system path) everything gets cloned fine, but when I try it on a different machine I get abort: remotestore: largefile XXXXX is missing
Here's the verbose output:
requesting all changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 177 changes to 177 files
calling hook changegroup.lfiles: <function checkrequireslfiles at 0x0000000002E00358>
updating to branch default
resolving manifests
getting .hglf/path/to.file
...
177 files updated, 0 files merged, 0 files removed, 0 files unresolved
getting changed largefiles
getting path/to.file:c0c81df934cd72ca980dd156984fa15987e3881d
abort: remotestore: largefile c0c81df934cd72ca980dd156984fa15987e3881dis missing
Both machines have the extension working. I've tried disabling the firewall but that didn't help. Do I have to do anything to set up the extension besides adding it to mercurial.ini?
Edit: If I delete the files from the server's AppData\Local\largefiles\ directory, I get the same error when cloning on the server, unless I use a filesystem path to clone, in which case the files are added back to `AppData\Local\largefiles\'
Edit 2: Here's the debug output and traceback:
177 files updated, 0 files merged, 0 files removed, 0 files unresolved
getting changed largefiles
using http://domain
sending capabilities command
getting largefiles: 0/75 lfile (0.00%)
getting path/to.file:64f2c341fb3b1adc7caec0dc9c51a97e51ca6034
sending statlfile command
Traceback (most recent call last):
File "mercurial\dispatch.pyo", line 87, in _runcatch
File "mercurial\dispatch.pyo", line 685, in _dispatch
File "mercurial\dispatch.pyo", line 467, in runcommand
File "mercurial\dispatch.pyo", line 775, in _runcommand
File "mercurial\dispatch.pyo", line 746, in checkargs
File "mercurial\dispatch.pyo", line 682, in <lambda>
File "mercurial\util.pyo", line 463, in check
File "mercurial\commands.pyo", line 1167, in clone
File "mercurial\hg.pyo", line 400, in clone
File "mercurial\extensions.pyo", line 184, in wrap
File "hgext\largefiles\overrides.pyo", line 629, in hgupdate
File "hgext\largefiles\lfcommands.pyo", line 416, in updatelfiles
File "hgext\largefiles\lfcommands.pyo", line 398, in cachelfiles
File "hgext\largefiles\basestore.pyo", line 80, in get
File "hgext\largefiles\remotestore.pyo", line 56, in _getfile
Abort: remotestore: largefile 64f2c341fb3b1adc7caec0dc9c51a97e51ca6034 is missing
The _getfile function throws an exception because the statlfile command returns that the file wasn't found.
I've never used python myself, so I don't know what I'm doing while trying to debug this :D
AFAIK the statlfile command gets executed on the server so I can't debug it from my local machine. I've tried running python -m win32traceutil on the server, but it doesn't show anything. I also tried setting accesslog and errorlog in the server's mercurial config file, but it doesn't generate them.
I run hg through the hgweb.wsgi script, and I have no idea if/how I can get into the python debugger using that, but if I could get the debugger running on the server I could narrow down the problem...
Finally figured it out, the extension tries to write temporary files to %windir%\System32\config\systemprofile\AppData\Local, which was causing permission errors. The call was wrapped in a try-catch block that ended up returning the "file not found" error.
I'm just posting this for anyone else coming into the thread from a search.
There's currently an issue using the largefiles extension in the mercurial python module when hosted via IIS. See this post if you're encountering issues pushing large changesets (or large files) to IIS via TortoiseHg.
The problem ultimlately turns out to be a bug in SSL processing introduced Python 2.7.3 (probably explaining why there are so many unresolve posts of people looking for problems with Mercurial). Rolling back to Python 2.7.2 let me get a little further ahead (blocked at 30Mb pushes instead of 15Mb), but to properly solve the problem I had to install the IISCrypto utility to completely disable transfers over SSLv2.