Silencing broken pipe errors (`[Errno 32] Broken pipe`) in python with WSGI+Pyramid - wsgi

I have a rather simple, naive Python/WSGI/Pyramid web-server.
It's run using wsgiref.simple_server.make_server(), on a server built using pyramid.config.Configurator().make_wsgi_app(). This server works fine.
However, the application it's serving has a lot of javascript image mouseover popups. If you run the mouse across the page, it can generate 20+ image requests. This is fine as well (It's an internal thing, not a lot of users).
However, doing so causes the server to emit something like half a dozen error tracebacks:
10.1.1.4 - - [25/Apr/2014 01:56:42] "GET /*SNIP* 500 59
----------------------------------------
Exception happened during processing of request from ('10.1.1.4', 18338)
Traceback (most recent call last):
File "/usr/lib/python3.4/wsgiref/handlers.py", line 138, in run
self.finish_response()
File "/usr/lib/python3.4/wsgiref/handlers.py", line 180, in finish_response
self.write(data)
File "/usr/lib/python3.4/wsgiref/handlers.py", line 274, in write
self.send_headers()
File "/usr/lib/python3.4/wsgiref/handlers.py", line 333, in send_headers
self._write(bytes(self.headers))
File "/usr/lib/python3.4/wsgiref/handlers.py", line 453, in _write
self.stdout.write(data)
File "/usr/lib/python3.4/socket.py", line 391, in write
return self._sock.send(b)
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.4/wsgiref/handlers.py", line 141, in run
self.handle_error()
File "/usr/lib/python3.4/wsgiref/handlers.py", line 368, in handle_error
self.finish_response()
File "/usr/lib/python3.4/wsgiref/handlers.py", line 180, in finish_response
self.write(data)
File "/usr/lib/python3.4/wsgiref/handlers.py", line 274, in write
self.send_headers()
File "/usr/lib/python3.4/wsgiref/handlers.py", line 331, in send_headers
if not self.origin_server or self.client_is_modern():
File "/usr/lib/python3.4/wsgiref/handlers.py", line 344, in client_is_modern
return self.environ['SERVER_PROTOCOL'].upper() != 'HTTP/0.9'
TypeError: 'NoneType' object is not subscriptable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.4/socketserver.py", line 306, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python3.4/socketserver.py", line 332, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python3.4/socketserver.py", line 345, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python3.4/socketserver.py", line 666, in __init__
self.handle()
File "/usr/lib/python3.4/wsgiref/simple_server.py", line 126, in handle
handler.run(self.server.get_app())
File "/usr/lib/python3.4/wsgiref/handlers.py", line 144, in run
self.close()
File "/usr/lib/python3.4/wsgiref/simple_server.py", line 35, in close
self.status.split(' ',1)[0], self.bytes_sent
AttributeError: 'NoneType' object has no attribute 'split'
I understand why I'm getting broken pipe errors (the request for the image is canceled before the image has fully transfered, because the mouseover popup has closed), and it seems harmless.
However, I have no idea how to silence this traceback. There are thousands of them in my logs, and it makes debugging actual errors a nightmare. I don't care that I'm getting broken pipe errors, how can I catch them and swallow them silently?
It seems like wsgiref.simple_server.make_server() installs an internal handler that catches BrokenPipeError: [Errno 32] Broken pipe, prints the traceback, and then swallows the error. I've tried wrapping the run_server() call in a try-except clause, and it doesn't have any effect.

I wound up just switching to using the CherryPy WSGI Server. It doesn't suffer from the broken pipe log issues, and is probably far more robust as well.
It also uses a threadpool, so it's more performant as well (multiple requests aren't blocking!).

I didn't find a straightforward way for achieving this, however, you can always do some monkey patching:
from wsgiref.handlers import BaseHandler
import sys
def ignore_broken_pipes(self):
if sys.exc_info()[0] != BrokenPipeError: BaseHandler.__handle_error_original_(self)
BaseHandler.__handle_error_original_ = BaseHandler.handle_error
BaseHandler.handle_error = ignore_broken_pipes
You will not see these annoyances after running this code somewhere in the beginning anymore.
For me, it looks like a bug somewhere in wsgiref implementation of BaseHandler:
def handle_error(self):
"""Log current error, and send error output to client if possible"""
self.log_exception(sys.exc_info())
if not self.headers_sent:
self.result = self.error_output(self.environ, self.start_response)
self.finish_response()
# XXX else: attempt advanced recovery techniques for HTML or text?
if BrokenPipeError is handled by this method, finish_response crashes. Why do we ever want to finish response if pipe is broken? Where the data is sent to?

Related

I got a error while running server in django i was using git before

bibek#sweety-babe:~/Desktop/Friends$ python3 manage.py runserver
Watching for file changes with StatReloader
Performing system checks...
System check identified no issues (0 silenced).
Exception in thread django-main-thread:
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 946, in run
self._target(*self._args, **self._kwargs)
File "/home/bibek/.local/lib/python3.10/site-packages/django/utils/autoreload.py", line 53, in wrapper
fn(*args, **kwargs)
File "/home/bibek/.local/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 120, in inner_run
self.check_migrations()
File "/home/bibek/.local/lib/python3.10/site-packages/django/core/management/base.py", line 458, in check_migrations
executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS])
File "/home/bibek/.local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 18, in __init__
self.loader = MigrationLoader(self.connection)
File "/home/bibek/.local/lib/python3.10/site-packages/django/db/migrations/loader.py", line 49, in __init__
self.build_graph()
File "/home/bibek/.local/lib/python3.10/site-packages/django/db/migrations/loader.py", line 274, in build_graph
raise exc
File "/home/bibek/.local/lib/python3.10/site-packages/django/db/migrations/loader.py", line 248, in build_graph
self.graph.validate_consistency()
File "/home/bibek/.local/lib/python3.10/site-packages/django/db/migrations/graph.py", line 195, in validate_consistency
[n.raise_error() for n in self.node_map.values() if isinstance(n, DummyNode)]
File "/home/bibek/.local/lib/python3.10/site-packages/django/db/migrations/graph.py", line 195, in <listcomp>
[n.raise_error() for n in self.node_map.values() if isinstance(n, DummyNode)]
File "/home/bibek/.local/lib/python3.10/site-packages/django/db/migrations/graph.py", line 58, in raise_error
raise NodeNotFoundError(self.error_message, self.key, origin=self.origin)
django.db.migrations.exceptions.NodeNotFoundError: Migration base.0001_initial dependencies reference nonexistent parent node ('auth', '0012_alter_user_first_name_max_length')
i am getting this error suddenly after git push. I dont know what is the reason behind this please help me.
this is just dummy (The POST method is used to send data to the server to create/update a resource on the server. In this HTTP POST request example, the Content-Type request header indicates the data type in the body of the POST message, and the Content-Length request header indicates the size of the data in the body of the POST request. The sample data is listed in the Content tab, and additional headers are listed in the Headers tab. The data is sent to the server in the body of the POST message.)
I'm thinking you probably didn't track the migration file when pushing, try pulling and see. if it doesn't work then I suggest you delete the files in the migration folder apart from the init file then run make migrations again

Why Error urllib.error.ContentTooShortError:

I get this error while retrieving the program I wrote with kivy.
I have to use vpn
Downloading https://dl.google.com/android/repository/android-ndk-r19c-linux-x86_64.zip
Traceback (most recent call last):
File "/home/mm/kivyenv/bin/buildozer", line 8, in <module>
sys.exit(main())
File "/home/mm/kivyenv/lib/python3.8/site-packages/buildozer/scripts/client.py", line 13, in main
Buildozer().run_command(sys.argv[1:])
File "/home/mm/kivyenv/lib/python3.8/site-packages/buildozer/__init__.py", line 1047, in run_command
self.target.run_commands(args)
File "/home/mm/kivyenv/lib/python3.8/site-packages/buildozer/target.py", line 92, in run_commands
func(args)
File "/home/mm/kivyenv/lib/python3.8/site-packages/buildozer/target.py", line 102, in cmd_debug
self.buildozer.prepare_for_build()
File "/home/mm/kivyenv/lib/python3.8/site-packages/buildozer/__init__.py", line 169, in prepare_for_build
self.target.install_platform()
File "/home/mm/kivyenv/lib/python3.8/site-packages/buildozer/targets/android.py", line 665, in install_platform
self._install_android_ndk()
File "/home/mm/kivyenv/lib/python3.8/site-packages/buildozer/targets/android.py", line 455, in _install_android_ndk
self.buildozer.download(url,
File "/home/mm/kivyenv/lib/python3.8/site-packages/buildozer/__init__.py", line 677, in download
urlretrieve(url, filename, report_hook)
File "/usr/lib/python3.8/urllib/request.py", line 1866, in retrieve
raise ContentTooShortError(
urllib.error.ContentTooShortError: <urlopen error retrieval incomplete: got only 261037633 out of 823376982 bytes>
From the urllib documentation:
exception urllib.error.ContentTooShortError(msg, content)
This exception is raised when the urlretrieve() function detects that the amount of the downloaded data is less than the expected amount (given by the Content-Length header). The content attribute stores the downloaded (and supposedly truncated) data.
In practice, it's likely that the VPN terminated the socket. You may need to implement retry/resume capability in your program.

Having trouble catching an exception in python 3

Working with Python 3.7.3, still figuring out how exception handling works.
I'm writing an xmpp bot, using slixmpp. I'm trying to make it so that if it loses connection to the server, it will try to reconnect. There doesn't seem to be any way to do this built in to slixmpp, so I'm write something into my own code to do it.
I've imported slixmpp as xmpp, and using it's send_raw() method to test that we're still connected to the server.
while True:
time.sleep(5) # Send every 5 seconds just for testing purposes
xmpp.send_raw('aroo?')
When I sever the connection to the server, this is what it spits out:
Traceback (most recent call last):
File "C:\Program Files\Python37\lib\threading.py", line 917, in _bootstrap_inner
self.run()
File "testcom.py", line 19, in run
eval(self.thing)()
File "testcom.py", line 28, in check_conn
xmpp.send_raw('aroo?')
File "C:\Program Files\Python37\lib\site-packages\slixmpp\xmlstream\xmlstream.py", line 926, in send_raw
raise NotConnectedError
slixmpp.xmlstream.xmlstream.NotConnectedError
I'm assuming that "NotConnectedError" is the exception that I need to catch, so I put the code inside a try block, like so:
try:
while True:
time.sleep(5) # Send every 5 seconds just for testing purposes
xmpp.send_raw('aroo?')
except NotConnectedError:
# Do a thing
pass
And this is what I get:
Traceback (most recent call last):
File "testcom.py", line 28, in check_conn
xmpp.send_raw('aroo?')
File "C:\Program Files\Python37\lib\site-packages\slixmpp\xmlstream\xmlstream.py", line 926, in send_raw
raise NotConnectedError()
slixmpp.xmlstream.xmlstream.NotConnectedError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files\Python37\lib\threading.py", line 917, in _bootstrap_inner
self.run()
"testcom.py", line 19, in run
eval(self.thing)()
File "testcom.py", line 29, in check_conn
except NotConnectedError:
NameError: name 'NotConnectedError' is not defined
Can anyone tell me what I'm doing wrong here?
Thanks!
I can't see your imports but make sure you have from slixmpp.xmlstream.xmlstream import NotConnectedError otherwise it doesn't have a definition for NotConnectedError within the application. You could also change NotConnectedError to xmpp.xmlstream.xmlstream.NotConnectedError if you don't want to have it imported as well.

pexpect telnet on windows

I've used pexpect on linux successfully to telnet/ssh into Cisco switches for CLI scrapping. I'm trying to convert this code over to Windows and having some issues since it doesn't support the pxpect.spawn() command.
I read some online documentation and it suggested to use the pexpect.popen_spawn.PopenSpawn command. Can someone please point to me what I'm doing wrong here? I removed all my exception handling to simplify the code. Thanks.
import pexpect
from pexpect.popen_spawn import PopenSpawn
child = pexpect.popen_spawn.PopenSpawn('C:/Windows/System32/telnet 192.168.1.1')
child.expect('Username:')
child.sendline('cisco')
child.expect('Password:')
child.sendline('cisco')
child.expect('>')
child.close()
Error:
Traceback (most recent call last):
File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\site-packages\pexpect\expect.py", line 98, in expect_loop
incoming = spawn.read_nonblocking(spawn.maxread, timeout)
File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\site-packages\pexpect\popen_spawn.py", line 68, in read_nonblocking
raise EOF('End Of File (EOF).')
pexpect.exceptions.EOF: End Of File (EOF).
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python\windows scripts\telnet\telnet.py", line 37, in <module>
main()
File "C:\Python\windows scripts\telnet\telnet.py", line 20, in main
child.expect('Username:')
File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\site-packages\pexpect\spawnbase.py", line 327, in expect
timeout, searchwindowsize, async_)
File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\site-packages\pexpect\spawnbase.py", line 355, in expect_list
return exp.expect_loop(timeout)
File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\site-packages\pexpect\expect.py", line 104, in expect_loop
return self.eof(e)
File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\site-packages\pexpect\expect.py", line 50, in eof
raise EOF(msg)
pexpect.exceptions.EOF: End Of File (EOF).
<pexpect.popen_spawn.PopenSpawn object at 0x000000000328C550>
searcher: searcher_re:
0: re.compile("b'Username:'")
The solution is to use plink.exe, which is a part of putty installation. You can download it from https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html
Enable TelnetClient on windows system and put plink.exe in the same folder from where you are running pexpect. Then you can telnet using pexpect as mentioned in the below example.
Also, use timeout flag with PopenSpawn to wait for the connection to establish. The above error is due to a timeout flag is not set.
p = pexpect.popen_spawn.PopenSpawn('plink.exe -telnet 192.168.0.1 -P 23', timeout=1)

urllib.urlopen() gives socket error: Name or service not known on python2.7

I am just starting to use and learn Python, so this may seem vary naive to ask.
On my Linux system if i try to get a webpage using urllib.urlopen() I get an error
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/urllib.py", line 86, in urlopen
return opener.open(url)
File "/usr/lib/python2.7/urllib.py", line 207, in open
return getattr(self, name)(url)
File "/usr/lib/python2.7/urllib.py", line 344, in open_http
h.endheaders(data)
File "/usr/lib/python2.7/httplib.py", line 954, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 814, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 776, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 757, in connect
self.timeout, self.source_address)
File "/usr/lib/python2.7/socket.py", line 553, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
IOError: [Errno socket error] [Errno -2] Name or service not known
>>>
If I try to do the same in the Python 2.7 installed in my Windows 7 system, it works fine.
Since i am a novice, its difficult for me to diagnose the problem. I tried to research it but still haven't got any answers.
So my questions are:
What is different in the windows system that urlopen() works there but not on Linux.
What needs to be done to ensure that urlopen() works on the Linux system. Its necessary for me that it works since the the program I am developing has some bash command calls and the program depends extensively on the proper working of urllib.

Resources