I'm trying to upload couple of .zip files with size ranging between 20 to 30 MBs using python's sftp module 'Paramiko'. I'm able to successfully connect to the sftp server, and also able to list the contents in the destination directory. But while attempting to upload files from my ec2 linux machine, the request is taking too much time and then terminating with EOFError(). Below is the code,
>>> import paramiko
>>> import os
>>> ftp_host='*****'
>>> ftp_username='***'
>>> ftp_password='**'
>>> ssh_client=paramiko.SSHClient()
>>> ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
>>> ssh_client.connect(hostname=ftp_host,username=ftp_username,password=ftp_password, timeout=500)
>>> ftp_client=ssh_client.open_sftp()
>>> ftp_client.chdir('/inbound/')
>>> ftp_client.listdir()
[u't5-file-1.zip', u'edp_r.zip', u'T5_Transaction_Sample.gz']
>>> ftp_client.put("/path-to-zip-files-on-local/edp_revenue.zip","/inbound/edp_revenue.zip")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 669, in put
return self.putfo(fl, remotepath, file_size, callback, confirm)
File "/usr/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 626, in putfo
fr.write(data)
File "/usr/lib/python2.7/dist-packages/paramiko/file.py", line 331, in write
self._write_all(data)
File "/usr/lib/python2.7/dist-packages/paramiko/file.py", line 448, in _write_all
count = self._write(data)
File "/usr/lib/python2.7/dist-packages/paramiko/sftp_file.py", line 176, in _write
self._reqs.append(self.sftp._async_request(type(None), CMD_WRITE, self.handle, long(self._realpos), data[:chunk]))
File "/usr/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 750, in _async_request
self._send_packet(t, msg)
File "/usr/lib/python2.7/dist-packages/paramiko/sftp.py", line 170, in _send_packet
self._write_all(out)
File "/usr/lib/python2.7/dist-packages/paramiko/sftp.py", line 135, in _write_all
raise EOFError()
EOFError
When I'm trying the same thing using curl command I'm easily able to do and just in 20 seconds. What am I missing here? Is it the size which is making the request to timed out or anything else.
It seems I was using an older version of paramiko on my linux server. I upgraded it using,
pip install --upgraded paramiko
and then logged everything in a log file,
paramiko.util.log_to_file('/path_to_log_file/paramiko.log')
just to capture every detail. Now it seems to be working pretty fast and correct.
Related
I am new to Python and trying to install Airflow in my Mac, by following this tutorial
While these two commands work fine:
$ airflow initdb
$ airflow webserver -p 8080
The scheduler command (airflow scheduler) throws the following error:
[2020-02-18 13:18:09,012] {scheduler_job.py:1382} ERROR - Exception when executing execute_helper Traceback (most recent call last):
File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1380, in _execute
self._execute_helper()
File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1413, in _execute_helper
self.processor_agent.start()
File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 554, in start
self._process.start()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
return Popen(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'SchedulerJob._execute.<locals>.processor_factory'
[2020-02-18 13:18:09,035] {helpers.py:322} INFO - Sending Signals.SIGTERM to GPID None
Traceback (most recent call last): File "/Users/mac/Workspace/airflow/airflow_venv/bin/airflow", line 37, in <module>
args.func(args) File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/utils/cli.py", line 75, in wrapper
return f(*args, **kwargs) File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/bin/cli.py", line 1040, in scheduler
job.run() File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 221, in run
self._execute() File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1384, in _execute
self.processor_agent.end() File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 707, in end
reap_process_group(self._process.pid, log=self.log) File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/utils/helpers.py", line 324, in reap_process_group
signal_procs(sig) File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/utils/helpers.py", line 293, in signal_procs
os.killpg(pgid, sig)
TypeError: an integer is required (got type NoneType)
EDIT: Python 3.8 is supported now https://github.com/apache/airflow#requirements. So this answer might not be relevant now.
This due to the Python version you are using. Airflow doesn't support Python 3.8 yet https://github.com/apache/airflow#stable-version-1109.
Downgrade your Python to 3.7 and check.
Maybe there are some compatibility problems?
Using Python 3.6.10 and airflow v1.10.4, I can get airflow running. Maybe you could try some other versions?
This worked for me!
1- Make sure you are using the correct celery version that supports your other packages like RabbitMQ ( as V5 doesn't support AMQP in its usual format), my advice is to use V4.6.X
2-THIS HAS NOTHING TO DO WITH PYTHON VERSION IF YOU ARE USING AIRFLOW V2.0
3- simply make yourself happy with airflow db reset (command may differ if you are using airflow Version X<2.0 )
4- Avoid deleting any dag like you delete a file and use airflow dag ... commands to do so. (it makes up a mess in your environment that you wont like, trust me on this..)
Wish you luck bearing python stuff..
I have a short python script (called VaultTransferScript.py) that should transfer a zip file from one machine to another. The destination machine is a mapped network-attached-storage machine, which I have assigned to be the Z: drive.
The script is:
import shutil
import os
from datetime import datetime
time_stamp = datetime.now().strftime('%Y-%m-%d_%H_%M')
title_str = 'VaultBackup.zip'
name = time_stamp + title_str
shutil.move('C:\\Users\\Hawking\\Desktop\\VaultBackups\\MyBackup.zip',
os.path.join('Z:\\VaultBackups\\'+name))
I can run this script from the notepad++ run facility, using
cmd /C python "$(FULL_CURRENT_PATH)"
But running it in a batch script as:
echo off
C:\Users\Hawking\AppData\Local\Programs\Python\Python37-32\python.exe C:\Users\Hawking\Desktop\VaultBackupTransfer.py
results in this:
C:\Users\Hawking\Desktop>echo off
Traceback (most recent call last):
File "C:\Users\Hawking\AppData\Local\Programs\Python\Python37-32\lib\shutil.py", line 557, in move
os.rename(src, real_dst)
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Users\\Hawking\\Desktop\\VaultBackups\\MyBackup.zip' -> 'Z:\\VaultBackups\\2018-09-21_14_30VaultBackup.zip'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Hawking\Desktop\VaultBackupTransfer.py", line 7, in <module>
shutil.move('C:\\Users\\Hawking\\Desktop\\VaultBackups\\MyBackup.zip', os.path.join('Z:\\VaultBackups\\'+name))
File "C:\Users\Hawking\AppData\Local\Programs\Python\Python37-32\lib\shutil.py", line 571, in move
copy_function(src, real_dst)
File "C:\Users\Hawking\AppData\Local\Programs\Python\Python37-32\lib\shutil.py", line 257, in copy2
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "C:\Users\Hawking\AppData\Local\Programs\Python\Python37-32\lib\shutil.py", line 121, in copyfile
with open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: 'Z:\\VaultBackups\\2018-09-21_14_30VaultBackup.zip'
WHat is the difference in how I invoke the python script, and why does it error out from the batch script, but not notepad++?
You might be running the Python program with different user permissions in Notepad++ versus the command prompt. Alternatively, another Python VM might be used. Although, nothing in particular makes me think that the later is true.
I've used pexpect on linux successfully to telnet/ssh into Cisco switches for CLI scrapping. I'm trying to convert this code over to Windows and having some issues since it doesn't support the pxpect.spawn() command.
I read some online documentation and it suggested to use the pexpect.popen_spawn.PopenSpawn command. Can someone please point to me what I'm doing wrong here? I removed all my exception handling to simplify the code. Thanks.
import pexpect
from pexpect.popen_spawn import PopenSpawn
child = pexpect.popen_spawn.PopenSpawn('C:/Windows/System32/telnet 192.168.1.1')
child.expect('Username:')
child.sendline('cisco')
child.expect('Password:')
child.sendline('cisco')
child.expect('>')
child.close()
Error:
Traceback (most recent call last):
File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\site-packages\pexpect\expect.py", line 98, in expect_loop
incoming = spawn.read_nonblocking(spawn.maxread, timeout)
File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\site-packages\pexpect\popen_spawn.py", line 68, in read_nonblocking
raise EOF('End Of File (EOF).')
pexpect.exceptions.EOF: End Of File (EOF).
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python\windows scripts\telnet\telnet.py", line 37, in <module>
main()
File "C:\Python\windows scripts\telnet\telnet.py", line 20, in main
child.expect('Username:')
File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\site-packages\pexpect\spawnbase.py", line 327, in expect
timeout, searchwindowsize, async_)
File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\site-packages\pexpect\spawnbase.py", line 355, in expect_list
return exp.expect_loop(timeout)
File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\site-packages\pexpect\expect.py", line 104, in expect_loop
return self.eof(e)
File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\site-packages\pexpect\expect.py", line 50, in eof
raise EOF(msg)
pexpect.exceptions.EOF: End Of File (EOF).
<pexpect.popen_spawn.PopenSpawn object at 0x000000000328C550>
searcher: searcher_re:
0: re.compile("b'Username:'")
The solution is to use plink.exe, which is a part of putty installation. You can download it from https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html
Enable TelnetClient on windows system and put plink.exe in the same folder from where you are running pexpect. Then you can telnet using pexpect as mentioned in the below example.
Also, use timeout flag with PopenSpawn to wait for the connection to establish. The above error is due to a timeout flag is not set.
p = pexpect.popen_spawn.PopenSpawn('plink.exe -telnet 192.168.0.1 -P 23', timeout=1)
Recently, Dill has completely stopped working for me. It does this:
>>> import dill
>>> dill.dumps([1,2,3])
b'\x80\x03]q\x00(K\x01K\x02K\x03e.'
>>> dill.loads(_)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python34\lib\site-packages\dill\dill.py", line 260, in loads
return load(file)
File "C:\Python34\lib\site-packages\dill\dill.py", line 250, in load
obj = pik.load()
File "C:\Python34\lib\pickle.py", line 1039, in load
dispatch[key[0]](self)
File "C:\Python34\lib\pickle.py", line 1066, in load_proto
raise ValueError("unsupported pickle protocol: %d" % proto)
ValueError: unsupported pickle protocol: 93
The number is different every time. This started happening maybe a month ago; reinstalling Dill with pip didn't help.
From stepping through it with a debugger it looks like it's correctly reading in the protocol version from the beginning of the data, then reading one of the first instructions in the pickle data and interpreting it as the protocol version for some reason. I don't really know though, since I don't know much about how pickle works.
I am just starting to use and learn Python, so this may seem vary naive to ask.
On my Linux system if i try to get a webpage using urllib.urlopen() I get an error
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/urllib.py", line 86, in urlopen
return opener.open(url)
File "/usr/lib/python2.7/urllib.py", line 207, in open
return getattr(self, name)(url)
File "/usr/lib/python2.7/urllib.py", line 344, in open_http
h.endheaders(data)
File "/usr/lib/python2.7/httplib.py", line 954, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 814, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 776, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 757, in connect
self.timeout, self.source_address)
File "/usr/lib/python2.7/socket.py", line 553, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
IOError: [Errno socket error] [Errno -2] Name or service not known
>>>
If I try to do the same in the Python 2.7 installed in my Windows 7 system, it works fine.
Since i am a novice, its difficult for me to diagnose the problem. I tried to research it but still haven't got any answers.
So my questions are:
What is different in the windows system that urlopen() works there but not on Linux.
What needs to be done to ensure that urlopen() works on the Linux system. Its necessary for me that it works since the the program I am developing has some bash command calls and the program depends extensively on the proper working of urllib.