I'm trying to get the largefiles extension working on a mercurial server under Windows Server 2008 / IIS 7.5 with the hgweb.wsgi script.
When I clone a repo with largefiles locally (but using https://domain/, not a file system path) everything gets cloned fine, but when I try it on a different machine I get abort: remotestore: largefile XXXXX is missing
Here's the verbose output:
requesting all changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 177 changes to 177 files
calling hook changegroup.lfiles: <function checkrequireslfiles at 0x0000000002E00358>
updating to branch default
resolving manifests
getting .hglf/path/to.file
...
177 files updated, 0 files merged, 0 files removed, 0 files unresolved
getting changed largefiles
getting path/to.file:c0c81df934cd72ca980dd156984fa15987e3881d
abort: remotestore: largefile c0c81df934cd72ca980dd156984fa15987e3881dis missing
Both machines have the extension working. I've tried disabling the firewall but that didn't help. Do I have to do anything to set up the extension besides adding it to mercurial.ini?
Edit: If I delete the files from the server's AppData\Local\largefiles\ directory, I get the same error when cloning on the server, unless I use a filesystem path to clone, in which case the files are added back to `AppData\Local\largefiles\'
Edit 2: Here's the debug output and traceback:
177 files updated, 0 files merged, 0 files removed, 0 files unresolved
getting changed largefiles
using http://domain
sending capabilities command
getting largefiles: 0/75 lfile (0.00%)
getting path/to.file:64f2c341fb3b1adc7caec0dc9c51a97e51ca6034
sending statlfile command
Traceback (most recent call last):
File "mercurial\dispatch.pyo", line 87, in _runcatch
File "mercurial\dispatch.pyo", line 685, in _dispatch
File "mercurial\dispatch.pyo", line 467, in runcommand
File "mercurial\dispatch.pyo", line 775, in _runcommand
File "mercurial\dispatch.pyo", line 746, in checkargs
File "mercurial\dispatch.pyo", line 682, in <lambda>
File "mercurial\util.pyo", line 463, in check
File "mercurial\commands.pyo", line 1167, in clone
File "mercurial\hg.pyo", line 400, in clone
File "mercurial\extensions.pyo", line 184, in wrap
File "hgext\largefiles\overrides.pyo", line 629, in hgupdate
File "hgext\largefiles\lfcommands.pyo", line 416, in updatelfiles
File "hgext\largefiles\lfcommands.pyo", line 398, in cachelfiles
File "hgext\largefiles\basestore.pyo", line 80, in get
File "hgext\largefiles\remotestore.pyo", line 56, in _getfile
Abort: remotestore: largefile 64f2c341fb3b1adc7caec0dc9c51a97e51ca6034 is missing
The _getfile function throws an exception because the statlfile command returns that the file wasn't found.
I've never used python myself, so I don't know what I'm doing while trying to debug this :D
AFAIK the statlfile command gets executed on the server so I can't debug it from my local machine. I've tried running python -m win32traceutil on the server, but it doesn't show anything. I also tried setting accesslog and errorlog in the server's mercurial config file, but it doesn't generate them.
I run hg through the hgweb.wsgi script, and I have no idea if/how I can get into the python debugger using that, but if I could get the debugger running on the server I could narrow down the problem...
Finally figured it out, the extension tries to write temporary files to %windir%\System32\config\systemprofile\AppData\Local, which was causing permission errors. The call was wrapped in a try-catch block that ended up returning the "file not found" error.
I'm just posting this for anyone else coming into the thread from a search.
There's currently an issue using the largefiles extension in the mercurial python module when hosted via IIS. See this post if you're encountering issues pushing large changesets (or large files) to IIS via TortoiseHg.
The problem ultimlately turns out to be a bug in SSL processing introduced Python 2.7.3 (probably explaining why there are so many unresolve posts of people looking for problems with Mercurial). Rolling back to Python 2.7.2 let me get a little further ahead (blocked at 30Mb pushes instead of 15Mb), but to properly solve the problem I had to install the IISCrypto utility to completely disable transfers over SSLv2.
Related
I wish I could use python to execute the Excel macro, so I tried to use the package xlwings to implement it.
The OS of my laptop is macOS Catalina (ver.: 10.15.7), my compiler is PyCharm (ver.: 2021.2.3), my Python version is 3.8.8, I used Anaconda (Ver.: 22.11.1) as my interpreter, my excel version is 16.66.1 (Microsoft Excel for Mac).
I faced the error "Command Error -1743: The User has declined permission" when I tried to use this package originally, and I solved this issue by installing an old compiler & using the old version of the compiler to run my code. My privacy setting for automation in the app Setting was shown below: (This is NOT the question I want to ask, but I'm not sure if it is also related to the issue I faced, so I still attached it here. I had uninstalled the old version of my compiler already.)
I wish I could implement an existing macro (called Hi) in an existing Excel file (called [StakeOverflow]HelloWorld.xlsm) through Python (represented as MY_PYTHON_FILE.py below), like the snapshot below:
My Excel file and my python code were stored on OneDrive. My macro code was shown below:
Sub Hi()
MsgBox "Good morning!"
End Sub
My python code was shown below:
import xlwings as xw
import time
wb = xw.Book('/Users/<MY NAME>/OneDrive/MY PATH DETAILS/[StakeOverflow]HelloWorld.xlsm')
time.sleep(10)
app = wb.app
macro_vba = app.macro("Hi")
macro_vba()
The code looks really simple, but I still faced the error. My Excel was not opened automatically, and I even could not open the Excel file manually thereafter. The error was shown below:
/Users/.../.conda/envs/Program/bin/python "/Users/.../OneDrive/.../MY_PYTHON_FILE.py"
Traceback (most recent call last):
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/xlwings/main.py", line 4914, in open
impl= self.impl(name)
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/xlwings/_xlmac.py", line 366, in __call__
raise KeyError(name_or_index)
KeyError: '[stakeoverflow] helloworld.xlsm'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/aeosa/appscript/reference.py", line 482, in __call___
return self.AS_appdata.target().event (self._code, params, atts, codecs=self.AS_appdata).send(timeout, sendflags)
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/aeosa/aem/aemsend.py", line 92, in send
raise EventError(errornum, errormsg, eventresult)
aem.aemsend.EventError: Command failed: Parameter error. (-50)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/.../OneDrive/.../MY_PYTHON_FILE.py", line 18, in <module>
wb = xw.Book('/Users/.../OneDrive/.../[stakeoverflow] helloworld.xlsm')
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/xlwings/main.py", line 876, in __init__
impl= app.books.open(
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/xlwings/main.py", line 4921, in open
impl = self.impl.open(
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/xlwings/ xlmac.py", line 420, in open
self.app.xl.open_workbook(
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/aeosa/appscript/reference.py", line 518, in __call__
raise CommandError(self, (args, kargs), e, self.AS_appdata) from e
appscript.reference.CommandError: Command failed:
OSERROR: -50
MESSAGE: Parameter error.
COMMAND: app(pid=1647).open_workbook (workbook_file_name='/users/.../onedrive/.../[stakeoverflow] helloworld.xlsm', update_links=k.do_not_update_links, read_only=None, format=None, password=None, write_reserved_password=None, ignore_read_only_recommended=None, origin=None, delimiter=None, editable=None, notify=None, converter=None, add_to_mru=None, timeout=-1)
Process finished with exit code 1
I tried to use terminal to run my python code, but I could not solve the problem, either.
I tried to open the Excel file manually thereafter, the error was shown below:
Excel cannot open the file ’[StakeOverflow]HelloWorld.xlsm’
because the file format or file extension is not valid. Verify
that the file has not been corrupted and that the file
extension matches the format of the file.
I tried to Google this error, but few solutions was found. It seems that it is related to the issue of external storage location. I tried to create another Excel file with the same name & macro code on my desktop and try again, and the error would disappear. (However, our company stored the files on OneDrive, so I wish I could utilise the file online.)
Just wondering if anyone here faced this situation before?
I found the answer by myself today. The issue is related to the naming issue rather than the permission issue.
As we could see that the error is KeyError: '[stakeoverflow] helloworld.xlsm', which implies that the problem is here. (Maybe because the system could not find the Excel file with this name.)
I tried to change the name from [stakeoverflow] helloworld.xlsm to helloworld.xlsm, and the error was gone. It seems that when xlwings want to open an Excel file, it would check the validity of the file name and change the name into smaller cases. If the file name contains special characters which are not allowed (e.g., "[]"), then the error would occur.
Notice that I could store the Excel file with these special characters in our laptop & OneDrive, but xlwings did not accept them.
Hope it is helpful to those who face this issue when using xlwings!
This question just recently posted has some useful answers, but it's not the same as mine. I'm running urllib3 1.26.4 and Python 3.7 from an ArcGIS Pro Notebook. I also have Fiddler 4 open because I want to track web traffic while troubleshooting a script. I only get the following error when I have Fiddler open. If I close Fiddler I get <Response [200]>. Is it not possible to use the requests module with Fiddler open? I'm new to Fiddler.
Truncated script:
import requests
#url
idph_data = 'https://idph.illinois.gov/DPHPublicInformation/api/covidVaccine/getVaccineAdministrationCurrent'
#headers
headers = {'user-agent': 'Mozilla/5.0'}
response = requests.get(idph_data, headers=headers, verify=True)
Error:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
In [35]:
Line 4: response = requests.get(idph_data,verify=True)
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\api.py, in get:
Line 76: return request('get', url, params=params, **kwargs)
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\api.py, in request:
Line 61: return session.request(method=method, url=url, **kwargs)
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\sessions.py, in request:
Line 542: resp = self.send(prep, **send_kwargs)
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\sessions.py, in send:
Line 655: r = adapter.send(request, **kwargs)
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\adapters.py, in send:
Line 449: timeout=timeout
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\urllib3\connectionpool.py, in urlopen:
Line 696: self._prepare_proxy(conn)
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\urllib3\connectionpool.py, in _prepare_proxy:
Line 964: conn.connect()
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\urllib3\connection.py, in connect:
Line 359: conn = self._connect_tls_proxy(hostname, conn)
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\urllib3\connection.py, in _connect_tls_proxy:
Line 506: ssl_context=ssl_context,
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\urllib3\util\ssl_.py, in ssl_wrap_socket:
Line 432: ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\urllib3\util\ssl_.py, in _ssl_wrap_socket_impl:
Line 474: return ssl_context.wrap_socket(sock)
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\ssl.py, in wrap_socket:
Line 423: session=session
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\ssl.py, in _create:
Line 827: raise ValueError("check_hostname requires server_hostname")
ValueError: check_hostname requires server_hostname
---------------------------------------------------------------------------
I am running into this issue as well with the environment provided by the current version of ArcGIS Pro. Per a lower-rated answer in the question you linked, I ran pip install urllib3==1.25.11 in the desired environment (in my case a clone of the default), and the issue appears to be resolved.
This is apparently due to a new feature in the urllib3 version provided by ArcGIS Pro. The above command downgrades to a relatively recent, but working, version. This will not be resolved in newer versions of urllib3, but instead, there is currently a pull request pending to fix the underlying issue in Python itself.
By the way, while it's possible to configure pip to be able to run through the fiddler proxy, it's not too easy, so it is best to turn off Fiddler while running any pip commands.
The pertinent bug report is found here. The issue appears to be that there is a very old bug in how Windows system proxy settings are being parsed by CPython / built-in urllib, causing the proxy entry for use with https URLs to always receive a HTTPS prefix (instead of HTTP). Newer version of urllib3 actually support using proxies over HTTPS, which was not previously the case. So before, urllib3 would ignore the prefix, but now, it attempts to use HTTPS to communicate with a HTTP url.
I've updated to requests v. 2.7.0, the latest, and I'm no longer receiving the error. If it was a version-specific issue related to v. 2.25.1, which was what I was using, I'm not sure. I haven't came across any evidence of that.
In a Windows command prompt in the same directory as my Python executable:
python -m pip install requests==2.7.0
Now if I run my original script with Fiddler capturing, I get a HTTP status of 200 and my script no longer gives me the error.
If I open my Yocto project's folder with Sublime under Ubuntu 16.04 and try to build with:
bitbake <image>
I get these errors:
ERROR: Unable to start bitbake server (None)
ERROR: Server log for this session (/local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/build-openstlinuxeglfs-stm32mp1-sw25v00/bitbake-cookerdaemon.log):
--- Starting bitbake server pid 4602 at 2020-02-01 02:59:00.519051 ---
Traceback (most recent call last):
File "/local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/layers/openembedded-core/bitbake/lib/bb/daemonize.py", line 83, in createDaemon
function()
File "/local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/layers/openembedded-core/bitbake/lib/bb/server/process.py", line 469, in _startServer
self.cooker = bb.cooker.BBCooker(self.configuration, self.featureset)
File "/local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/layers/openembedded-core/bitbake/lib/bb/cooker.py", line 210, in __init__
self.initConfigurationData()
File "/local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/layers/openembedded-core/bitbake/lib/bb/cooker.py", line 396, in initConfigurationData
self.add_filewatch(mc.getVar("__base_depends", False), self.configwatcher)
File "/local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/layers/openembedded-core/bitbake/lib/bb/cooker.py", line 306, in add_filewatch
watcher.add_watch(f, self.watchmask, quiet=False)
File "/local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/layers/openembedded-core/bitbake/lib/pyinotify.py", line 1924, in add_watch
raise WatchManagerError(err, ret_)
pyinotify.WatchManagerError: add_watch: cannot watch /local/STM32MP15-Ecosystem-v1.1.0/Distribution-Package/openstlinux-4.19-thud-mp1-19-10-09/build-openstlinuxeglfs-stm32mp1-sw25v00/conf WD=-1, Errno=No space left on device (ENOSPC)
ERROR: No space left on device or exceeds fs.inotify.max_user_watches?
ERROR: To check max_user_watches: sysctl -n fs.inotify.max_user_watches.
ERROR: To modify max_user_watches: sysctl -n -w fs.inotify.max_user_watches=<value>.
ERROR: Root privilege is required to modify max_user_watches.
Closing the editor and issue again the command works correctly.
Other editors (like gedit) does not have this behavior.
I know I can live without Sublime, but I want to understand che cause of the errors.
You're running out of inotify watches. Software such as Sublime and the program you're running here (among others) use inotify watches to detect changes to the file system, such as being able to track when files are changing or when the contents of a directory changes.
There's a (user settable) upper limit to the number of watches that can be in use at once, and the rather cryptic error message you're seeing here is a symptom of the limit being reached and the program failing to obtain a watch.
The default value for the maximum inotify watches may not be set high enough on your system for the software (and volume of files) that you're using, but you can change that if you like.
The output at the bottom of your error diagnotic information shows how you can view/adjust the upper limit. The following question also shows how you could do this as well.
https://unix.stackexchange.com/questions/13751/kernel-inotify-watch-limit-reached
Attempting to execute shutil.rmtree(path) on a directory managed by NFS consistently fails. Below you can see that os.rmdir(path) within shutil.rmtree(path) causes the exception. Is there a more robust way for me to achieve the expected result?
It appears that it removes all of the files, yet a hidden .nfs file remains in the directory for a short amount of time. I'm guessing that the process from which I'm calling rmtree has an open file handle to one of the files inside the directory, which, when deleted, apparently causes NFS to write a new hidden file. That would cause os.rmdir to fail on attempting to remove a non-empty directory.
Traceback (most recent call last):
File "/home/me/pre3/lib/python3.6/shutil.py", line 484, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "/home/me/pre3/lib/python3.6/shutil.py", line 482, in rmtree
os.rmdir(path)
OSError: [Errno 39] Directory not empty:
NFS details:
$ nfsstat -m
/home/me/nfs from XXX.YYY.ZZZ:/mnt/path/to/nfs
Flags: rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=50,retrans=2,sec=sys,mountaddr=REDACTED,mountvers=3,mountport=832,mountproto=udp,local_lock=none,addr=REDACTED
I'm using Python 3.6.6 on Ubuntu 16.04.
If the python logging module is logging to the target output directory, it will maintain an open file. A workaround is to call logging.shutdown() first, then called shutil.rmtree(path). This is not a general answer to the broader question, however.
You could try defining an error handler function to be passed to the onerror arg for shutil.rmtree: https://docs.python.org/3/library/shutil.html#shutil.rmtree
def handle_rmtree_err(function, path, excinfo):
...
shutil.rmtree(my_path, onerror=handle_rmtree_err)
There are all sorts of reasons why a process may be holding onto a file, so I can't tell you what the error handler should do exactly.
If you haven't figured out what is holding onto the file, try $ lsof | grep .nfsXXXX.
If all else fails you could time.sleep(secs) and retry shutil.rmtree.
The program should allow to run several https get requests with one aiohttp.ClientSession as the documentation suggests. It is intended to run a telegram bot.
I was not able to catch the exception with try ... except. Therefore the program hangs when exiting. During extended sessions the error is printed in the command windows (but not in the error log).
SSL error in data received
protocol: <asyncio.sslproto.SSLProtocol object at 0x0000016A581E4400>
transport: <_SelectorSocketTransport fd=644 read=polling write=<idle, bufsize=0>>
Traceback (most recent call last):
File "C:\Users\annet\Anaconda3\lib\asyncio\sslproto.py", line 526, in data_received
ssldata, appdata = self._sslpipe.feed_ssldata(data)
File "C:\Users\annet\Anaconda3\lib\asyncio\sslproto.py", line 207, in feed_ssldata
self._sslobj.unwrap()
File "C:\Users\annet\Anaconda3\lib\ssl.py", line 767, in unwrap
return self._sslobj.shutdown()
ssl.SSLError: [SSL: KRB5_S_INIT] application data after close notify (_ssl.c:2592)
^C
As the error information is very unspecific I could not really isolate the source and have a short code to reproduce the error.
A sample code is on github under https://github.com/fhag/telegram2.git
In order to run the code you will need an API token from telegram of your own bot.
This error showed up the first time when I upgraded to python 3.7.1.
Python is running on Windows 10.