Attempting to execute shutil.rmtree(path) on a directory managed by NFS consistently fails. Below you can see that os.rmdir(path) within shutil.rmtree(path) causes the exception. Is there a more robust way for me to achieve the expected result?
It appears that it removes all of the files, yet a hidden .nfs file remains in the directory for a short amount of time. I'm guessing that the process from which I'm calling rmtree has an open file handle to one of the files inside the directory, which, when deleted, apparently causes NFS to write a new hidden file. That would cause os.rmdir to fail on attempting to remove a non-empty directory.
Traceback (most recent call last):
File "/home/me/pre3/lib/python3.6/shutil.py", line 484, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "/home/me/pre3/lib/python3.6/shutil.py", line 482, in rmtree
os.rmdir(path)
OSError: [Errno 39] Directory not empty:
NFS details:
$ nfsstat -m
/home/me/nfs from XXX.YYY.ZZZ:/mnt/path/to/nfs
Flags: rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=50,retrans=2,sec=sys,mountaddr=REDACTED,mountvers=3,mountport=832,mountproto=udp,local_lock=none,addr=REDACTED
I'm using Python 3.6.6 on Ubuntu 16.04.
If the python logging module is logging to the target output directory, it will maintain an open file. A workaround is to call logging.shutdown() first, then called shutil.rmtree(path). This is not a general answer to the broader question, however.
You could try defining an error handler function to be passed to the onerror arg for shutil.rmtree: https://docs.python.org/3/library/shutil.html#shutil.rmtree
def handle_rmtree_err(function, path, excinfo):
...
shutil.rmtree(my_path, onerror=handle_rmtree_err)
There are all sorts of reasons why a process may be holding onto a file, so I can't tell you what the error handler should do exactly.
If you haven't figured out what is holding onto the file, try $ lsof | grep .nfsXXXX.
If all else fails you could time.sleep(secs) and retry shutil.rmtree.
Related
I Have installed fasttext module in Python and loaded the model [ 'cc.en.300.bin'].
I already made the data frame format according to the fasttext. and then generating the files
train.to_csv(" ecomm.train",columns=['Category_description'], index= False, header= False)
test.to_csv("ecom.test", columns=['Category_description'], index= False, header= False)
the files created successfully! then when I run this code
import fasttext
mod= fasttext.train_supervised(input='ecomm.train')
I get this error:
Traceback (most recent call last):
File "/Users/rosie/Documents/ProGraMinG/Python/pythonProject/FastText/FastText_overview.py", line 97, in <module>
mod= fasttext.train_supervised(input='ecomm.train')
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/fasttext/FastText.py", line 533, in train_supervised
fasttext.train(ft.f, a)
ValueError: ecomm.train cannot be opened for training!
{ UPDATE } !!!
Used both isfile() and exists() functions to check if the file exists:
path = 'Users/rosie/Documents/ProGraMinG/Python/pythonProject/FastText/ecomm.train'
check_file = os.path.isfile(path)
print("isfile method ",check_file)
check_file = os.path.exists(path)
print("exists method ",check_file)
Both methods returns ' False '.
I also checked if the file is readable or not
doc= open(' ecomm.train', 'r')
print('checking if the file is readable', doc.readable())
However, it returned 'True', now I'm confused. As for the size of the ' ecomm.train', it is 29.4 MB
Are you sure the file is readable, at the simple (local) path 'ecomm.train', from your Python process, given its current local orking directory?
For example, try specifying the file as its full absolute path – on MacOS probably something like /Users/yourusername/yourdirectory/etc/etc/ecomm.train. If that works, the problem was that your Python code's effective directory wasn't what you expected.
Alternatively, if the process that wrote the file was in some way a different user than the later process trying to read it, there might be permission errors.
Totally separate from fasttext, you could check, from the same code that's about to try fasttext operations, if the file is readable (via either the local path, or the absolute path) using a recipe lie that in this other answer: https://stackoverflow.com/a/44213239/130288
Even if it fails, it might give a more-explanatory error.
I wish I could use python to execute the Excel macro, so I tried to use the package xlwings to implement it.
The OS of my laptop is macOS Catalina (ver.: 10.15.7), my compiler is PyCharm (ver.: 2021.2.3), my Python version is 3.8.8, I used Anaconda (Ver.: 22.11.1) as my interpreter, my excel version is 16.66.1 (Microsoft Excel for Mac).
I faced the error "Command Error -1743: The User has declined permission" when I tried to use this package originally, and I solved this issue by installing an old compiler & using the old version of the compiler to run my code. My privacy setting for automation in the app Setting was shown below: (This is NOT the question I want to ask, but I'm not sure if it is also related to the issue I faced, so I still attached it here. I had uninstalled the old version of my compiler already.)
I wish I could implement an existing macro (called Hi) in an existing Excel file (called [StakeOverflow]HelloWorld.xlsm) through Python (represented as MY_PYTHON_FILE.py below), like the snapshot below:
My Excel file and my python code were stored on OneDrive. My macro code was shown below:
Sub Hi()
MsgBox "Good morning!"
End Sub
My python code was shown below:
import xlwings as xw
import time
wb = xw.Book('/Users/<MY NAME>/OneDrive/MY PATH DETAILS/[StakeOverflow]HelloWorld.xlsm')
time.sleep(10)
app = wb.app
macro_vba = app.macro("Hi")
macro_vba()
The code looks really simple, but I still faced the error. My Excel was not opened automatically, and I even could not open the Excel file manually thereafter. The error was shown below:
/Users/.../.conda/envs/Program/bin/python "/Users/.../OneDrive/.../MY_PYTHON_FILE.py"
Traceback (most recent call last):
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/xlwings/main.py", line 4914, in open
impl= self.impl(name)
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/xlwings/_xlmac.py", line 366, in __call__
raise KeyError(name_or_index)
KeyError: '[stakeoverflow] helloworld.xlsm'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/aeosa/appscript/reference.py", line 482, in __call___
return self.AS_appdata.target().event (self._code, params, atts, codecs=self.AS_appdata).send(timeout, sendflags)
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/aeosa/aem/aemsend.py", line 92, in send
raise EventError(errornum, errormsg, eventresult)
aem.aemsend.EventError: Command failed: Parameter error. (-50)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/.../OneDrive/.../MY_PYTHON_FILE.py", line 18, in <module>
wb = xw.Book('/Users/.../OneDrive/.../[stakeoverflow] helloworld.xlsm')
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/xlwings/main.py", line 876, in __init__
impl= app.books.open(
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/xlwings/main.py", line 4921, in open
impl = self.impl.open(
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/xlwings/ xlmac.py", line 420, in open
self.app.xl.open_workbook(
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/aeosa/appscript/reference.py", line 518, in __call__
raise CommandError(self, (args, kargs), e, self.AS_appdata) from e
appscript.reference.CommandError: Command failed:
OSERROR: -50
MESSAGE: Parameter error.
COMMAND: app(pid=1647).open_workbook (workbook_file_name='/users/.../onedrive/.../[stakeoverflow] helloworld.xlsm', update_links=k.do_not_update_links, read_only=None, format=None, password=None, write_reserved_password=None, ignore_read_only_recommended=None, origin=None, delimiter=None, editable=None, notify=None, converter=None, add_to_mru=None, timeout=-1)
Process finished with exit code 1
I tried to use terminal to run my python code, but I could not solve the problem, either.
I tried to open the Excel file manually thereafter, the error was shown below:
Excel cannot open the file ’[StakeOverflow]HelloWorld.xlsm’
because the file format or file extension is not valid. Verify
that the file has not been corrupted and that the file
extension matches the format of the file.
I tried to Google this error, but few solutions was found. It seems that it is related to the issue of external storage location. I tried to create another Excel file with the same name & macro code on my desktop and try again, and the error would disappear. (However, our company stored the files on OneDrive, so I wish I could utilise the file online.)
Just wondering if anyone here faced this situation before?
I found the answer by myself today. The issue is related to the naming issue rather than the permission issue.
As we could see that the error is KeyError: '[stakeoverflow] helloworld.xlsm', which implies that the problem is here. (Maybe because the system could not find the Excel file with this name.)
I tried to change the name from [stakeoverflow] helloworld.xlsm to helloworld.xlsm, and the error was gone. It seems that when xlwings want to open an Excel file, it would check the validity of the file name and change the name into smaller cases. If the file name contains special characters which are not allowed (e.g., "[]"), then the error would occur.
Notice that I could store the Excel file with these special characters in our laptop & OneDrive, but xlwings did not accept them.
Hope it is helpful to those who face this issue when using xlwings!
I tried to make it so that my bot will create a directory inside a directory in the bot to store server data.
I originally just wanted to create the directory without worrying about permissions:
#client.event
async def on_guild_join(guild):
print(f'Recognized that Beatboxer has joined {guild.name}')
guild_path = rf'/guilds/{guild.id}'
if not os.path.exists(guild_path):
os.makedirs(rf'/guilds/{guild.id}')
An error message came up that looked like this:
Ignoring exception in on_guild_join
Traceback (most recent call last):
File "/opt/virtualenvs/python3/lib/python3.8/site-packages/discord/client.py", line 312, in _run_event
await coro(*args, **kwargs)
File "main.py", line 227, in on_guild_join
os.makedirs(rf'guilds/{guild.id}')
File "/usr/lib/python3.8/os.py", line 223, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: 'guilds/727168023101964298'
I then tried adding os.chmod into the code, but, for some reason, still had the same error message.
os.chmod("guilds", 777)
#client.event
async def on_guild_join(guild):
print(f'Recognized that Beatboxer has joined {guild.name}')
guild_path = rf'/guilds/{guild.id}'
if not os.path.exists(guild_path):
os.makedirs(rf'/guilds/{guild.id}')
Also, calling os.chdir and changing the directory into there did not work, and had a similar error message.
os.chmod("guilds", 777)
#client.event
async def on_guild_join(guild):
print(f'Recognized that Beatboxer has joined {guild.name}')
guild_path = rf'/guilds/{guild.id}'
if not os.path.exists(guild_path):
os.chdir('/guilds')
os.makedirs(rf'{guild.id}')
Finally, I attempted one last thing (which obviously still didn't work), which was os.popen, which opens a pipe for a command, allowing it to transfer the output to a file which is editable by other programs (which therefore should not regard what any permissions do):
#client.event
async def on_guild_join(guild):
print(f'Recognized that Beatboxer has joined {guild.name}')
guild_path = rf'/guilds/{guild.id}'
if not os.path.exists(guild_path):
os.popen(os.makedirs(rf'/guilds/{guild.id}'))
All of these attempted codes have very similar error messages, particularly Errno 13. Computer configuration will most likely not work. Please help? Thank you!
The problem is not that the functions you use are wrong.
You are actually giving a directory location you dont have permission to.
There are 2 kinds of paths to folders. Absolute paths and relative paths. In the examples you gave you used absolute path. When you use the absolute path you use the root directory as starting point. The thing with online IDE's is that you often dont have direct access to the root directory. Thus making new directories will give permission errors.
So how do we fix this? I suggest using relative paths instead. To fix this in your code is really easy, instead of doing this:
'/path/to/folder'
Do this:
'./path/to/folder'
By using ./ instead of /. You use the current folder as your starting point instead of the root directory. As you often have access to the current folder this wont give permission errors.
If I open my Yocto project's folder with Sublime under Ubuntu 16.04 and try to build with:
bitbake <image>
I get these errors:
ERROR: Unable to start bitbake server (None)
ERROR: Server log for this session (/local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/build-openstlinuxeglfs-stm32mp1-sw25v00/bitbake-cookerdaemon.log):
--- Starting bitbake server pid 4602 at 2020-02-01 02:59:00.519051 ---
Traceback (most recent call last):
File "/local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/layers/openembedded-core/bitbake/lib/bb/daemonize.py", line 83, in createDaemon
function()
File "/local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/layers/openembedded-core/bitbake/lib/bb/server/process.py", line 469, in _startServer
self.cooker = bb.cooker.BBCooker(self.configuration, self.featureset)
File "/local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/layers/openembedded-core/bitbake/lib/bb/cooker.py", line 210, in __init__
self.initConfigurationData()
File "/local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/layers/openembedded-core/bitbake/lib/bb/cooker.py", line 396, in initConfigurationData
self.add_filewatch(mc.getVar("__base_depends", False), self.configwatcher)
File "/local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/layers/openembedded-core/bitbake/lib/bb/cooker.py", line 306, in add_filewatch
watcher.add_watch(f, self.watchmask, quiet=False)
File "/local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/layers/openembedded-core/bitbake/lib/pyinotify.py", line 1924, in add_watch
raise WatchManagerError(err, ret_)
pyinotify.WatchManagerError: add_watch: cannot watch /local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/build-openstlinuxeglfs-stm32mp1-sw25v00/conf WD=-1, Errno=No space left on device (ENOSPC)
ERROR: No space left on device or exceeds fs.inotify.max_user_watches?
ERROR: To check max_user_watches: sysctl -n fs.inotify.max_user_watches.
ERROR: To modify max_user_watches: sysctl -n -w fs.inotify.max_user_watches=<value>.
ERROR: Root privilege is required to modify max_user_watches.
Closing the editor and issue again the command works correctly.
Other editors (like gedit) does not have this behavior.
I know I can live without Sublime, but I want to understand che cause of the errors.
You're running out of inotify watches. Software such as Sublime and the program you're running here (among others) use inotify watches to detect changes to the file system, such as being able to track when files are changing or when the contents of a directory changes.
There's a (user settable) upper limit to the number of watches that can be in use at once, and the rather cryptic error message you're seeing here is a symptom of the limit being reached and the program failing to obtain a watch.
The default value for the maximum inotify watches may not be set high enough on your system for the software (and volume of files) that you're using, but you can change that if you like.
The output at the bottom of your error diagnotic information shows how you can view/adjust the upper limit. The following question also shows how you could do this as well.
https://unix.stackexchange.com/questions/13751/kernel-inotify-watch-limit-reached
I'm trying to get the largefiles extension working on a mercurial server under Windows Server 2008 / IIS 7.5 with the hgweb.wsgi script.
When I clone a repo with largefiles locally (but using https://domain/, not a file system path) everything gets cloned fine, but when I try it on a different machine I get abort: remotestore: largefile XXXXX is missing
Here's the verbose output:
requesting all changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 177 changes to 177 files
calling hook changegroup.lfiles: <function checkrequireslfiles at 0x0000000002E00358>
updating to branch default
resolving manifests
getting .hglf/path/to.file
...
177 files updated, 0 files merged, 0 files removed, 0 files unresolved
getting changed largefiles
getting path/to.file:c0c81df934cd72ca980dd156984fa15987e3881d
abort: remotestore: largefile c0c81df934cd72ca980dd156984fa15987e3881dis missing
Both machines have the extension working. I've tried disabling the firewall but that didn't help. Do I have to do anything to set up the extension besides adding it to mercurial.ini?
Edit: If I delete the files from the server's AppData\Local\largefiles\ directory, I get the same error when cloning on the server, unless I use a filesystem path to clone, in which case the files are added back to `AppData\Local\largefiles\'
Edit 2: Here's the debug output and traceback:
177 files updated, 0 files merged, 0 files removed, 0 files unresolved
getting changed largefiles
using http://domain
sending capabilities command
getting largefiles: 0/75 lfile (0.00%)
getting path/to.file:64f2c341fb3b1adc7caec0dc9c51a97e51ca6034
sending statlfile command
Traceback (most recent call last):
File "mercurial\dispatch.pyo", line 87, in _runcatch
File "mercurial\dispatch.pyo", line 685, in _dispatch
File "mercurial\dispatch.pyo", line 467, in runcommand
File "mercurial\dispatch.pyo", line 775, in _runcommand
File "mercurial\dispatch.pyo", line 746, in checkargs
File "mercurial\dispatch.pyo", line 682, in <lambda>
File "mercurial\util.pyo", line 463, in check
File "mercurial\commands.pyo", line 1167, in clone
File "mercurial\hg.pyo", line 400, in clone
File "mercurial\extensions.pyo", line 184, in wrap
File "hgext\largefiles\overrides.pyo", line 629, in hgupdate
File "hgext\largefiles\lfcommands.pyo", line 416, in updatelfiles
File "hgext\largefiles\lfcommands.pyo", line 398, in cachelfiles
File "hgext\largefiles\basestore.pyo", line 80, in get
File "hgext\largefiles\remotestore.pyo", line 56, in _getfile
Abort: remotestore: largefile 64f2c341fb3b1adc7caec0dc9c51a97e51ca6034 is missing
The _getfile function throws an exception because the statlfile command returns that the file wasn't found.
I've never used python myself, so I don't know what I'm doing while trying to debug this :D
AFAIK the statlfile command gets executed on the server so I can't debug it from my local machine. I've tried running python -m win32traceutil on the server, but it doesn't show anything. I also tried setting accesslog and errorlog in the server's mercurial config file, but it doesn't generate them.
I run hg through the hgweb.wsgi script, and I have no idea if/how I can get into the python debugger using that, but if I could get the debugger running on the server I could narrow down the problem...
Finally figured it out, the extension tries to write temporary files to %windir%\System32\config\systemprofile\AppData\Local, which was causing permission errors. The call was wrapped in a try-catch block that ended up returning the "file not found" error.
I'm just posting this for anyone else coming into the thread from a search.
There's currently an issue using the largefiles extension in the mercurial python module when hosted via IIS. See this post if you're encountering issues pushing large changesets (or large files) to IIS via TortoiseHg.
The problem ultimlately turns out to be a bug in SSL processing introduced Python 2.7.3 (probably explaining why there are so many unresolve posts of people looking for problems with Mercurial). Rolling back to Python 2.7.2 let me get a little further ahead (blocked at 30Mb pushes instead of 15Mb), but to properly solve the problem I had to install the IISCrypto utility to completely disable transfers over SSLv2.