I am struggling with this one and I have come across a similar problem but I could not understand it. The problem seems to be with the paramiko sftp.put(localpath). I know I am doing it wrong can't figure out how to fix it. Please note I have heavily modified the original code. As I said earlier, I came across a similar problem but I do not understand the resolution.
import paramiko
pk = paramiko.RSAKey.from_private_key(private_key_file)
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect("hostname", username = "test", password = "password", pkey =
pk)
sftp = ssh.open_sftp()
#some files. previous_week from week number generator
PC_Rank = 'some_file_WK%s.csv' %previous_week
PC_SA = 'WK%s_some_other_file.csv' %previous_week
#Python list
Files = [PC_Rank, PC_SA]
#windows server file location
local_path = "\\10.245.102.21\Files\#GIS Files\#csv_output\delete"
#Defining remote directory
remote_path = "Images/Staging/coverage_checker"
sftp.chdir(remote_path)
for filename in Files:
sftp.put(local_path, filename)
Related
I am trying to rename the file, I can see it when I print out the shared file.filename, but it gives an error saying its not there
import os, datetime
from smb.SMBConnection import SMBConnection
from datetime import date, datetime
userID = 'username'
password = 'mypsswrd$'
client_machine_name = 'localpcname'
server_name = 'hostname'
server_ip = '10.28.7.249'
domain_name = 'domain name'
now = datetime.now()
date_time = now.strftime("%d/%m/%y")
conn = SMBConnection(userID, password, client_machine_name, server_name, domain=domain_name, use_ntlm_v2=True,
is_direct_tcp=True)
conn.connect(server_ip, 445)
shares = conn.listShares()
for share in shares:
if not share.isSpecial and share.name not in ['NETLOGON', 'SYSVOL']:
sharedfiles = conn.listPath(share.name, '/')
for sharedfile in sharedfiles:
print(sharedfile.filename)
#change name
newname = 'CS_Out_Of_Country' + date_time +'.csv'
os.rename('CS_Out_Of_Country.csv', newname)
#Copy to archive
#close the connection
conn.close()
the connection works fine and I can see my csv file but it says it doesn't exist, how come?
os.rename operates in the current working directory. It doesn't know about the smb connection you created before. To fix this, you should either
Mount the smb share into the local filesystem, then os.chdir() to said mountpoint and then run os.rename() OR
use SMBConnection.rename (disclaimer: I've never used pysmb before, but that's what I found in 5 seconds using google.)
I develop a Python two scripts to transfer lot of data (~120Go) on my vm, with Paramiko.
My vm is on OVH server.
First script transfert ~ 40Go, and the second script ~ 80Go.
Stack :
Python 3.9.1
Paramiko 2.7.2
SCP 0.13.3
On my both scripts, I use this function to setup SSH connection.
def connect():
transport = paramiko.Transport((target_host, target_port))
transport.connect(None, target_username, target_pwd)
sftp_client = paramiko.SFTPClient.from_transport(transport)
green_print("SSH connected")
return sftp_client, transport
If I create one script which do the two transfer, I'm timeout after 3 hours.
With two distincts script which run in the same time, I'm timeout after 2h30 of transfer.
I already read many many many post on Paramiko, SSH connection, timeout parameter, ClientAliveInterval, etc... But nothing works.
After this time, I have this error
Connexion fermée par l'hôte distant / Connection closed by remote host
Three functions of my script :
def connect():
transport = paramiko.Transport((target_host, target_port))
transport.connect(None, target_username, target_pwd)
sftp_client = paramiko.SFTPClient.from_transport(transport)
green_print("SSH connected")
return sftp_client, transport
def transfert(sftp, vm, object_path):
os.chdir(os.path.split(object_path)[0])
parent = os.path.split(object_path)[1]
try:
sftp.mkdir(vm)
except:
pass
for path, _, files in os.walk(parent):
try:
sftp.mkdir(os.path.join(vm, path))
except:
pass
for filename in files:
sftp.put(os.path.join(object_path, filename),
os.path.join(vm, path, filename))
def job():
green_print("\nProcess start...")
check_folder()
folder = forfiles_method()
vm, lidar, pos = name_path(folder)
sftp, transport = connect()
transfert(sftp, vm, pos)
sftp.close()
transport.close()
minimal reproducible example :
from paramiko.sftp_client import SFTPClient
import paramiko
import os
target_host = 'xx.xx.x.xxx'
target_port = 22
target_username = "xxxxxxx"
target_pwd = 'xxxxxx'
remote_path = "e:/x/" # => on your vm
target_folder = '/folder1' # => on your computer
def connect():
transport = paramiko.Transport((target_host, target_port))
transport.connect(None, target_username, target_pwd)
sftp_client = paramiko.SFTPClient.from_transport(transport)
return sftp_client, transport
def transfert(sftp, remote_path, object_path):
os.chdir(os.path.split(object_path)[0])
parent = os.path.split(object_path)[1]
try:
sftp.mkdir(remote_path)
except:
pass
for path, _, files in os.walk(parent):
try:
sftp.mkdir(os.path.join(remote_path, path))
except:
pass
for filename in files:
sftp.put(os.path.join(object_path, filename),
os.path.join(remote_path, path, filename))
def job():
sftp, transport = connect()
transfert(sftp, remote_path, target_folder)
sftp.close()
transport.close()
The tree structure of my files, and I want to transfer only the "test" folder which contains more than 120GB.
folder / test
I'm new in Python dev.
If someone have a solution, I take it !
So the solution :
subprocess.run(["winscp.com", "/script=" + cmdFile], shell=True)
If winscp.com is not found like command, insert the path like : C:/Program Files (x86)/WinSCP/winscp.com
Write your commandes line in a txt file, here cmdFile.
Links, which can help you :
Running WinSCP command from Python
From Python run WinSCP commands in console
https://winscp.net/eng/docs/commandline
I need to find out for a script I'm writing who is the true owner of a file in an smb share (mounted using mount -t cifs of course on my server and using net use through windows machines).
Turns out it is a real challenge finding this information out using python on a linux server.
I tried using many many smb libraries (such as smbprotocol, smbclient and others), nothing worked.
I find few solutions for windows, they all use pywin32 or another windows specific package.
And I also managed to do it from bash using smbcalcs but couldn't do it cleanly but using subprocess.popen('smbcacls')..
Any idea on how to solve it?
This was unbelievably not a trivial task, and unfortunately the answer isn't simple as I hoped it would be..
I'm posting this answer if someone will be stuck with this same problem in the future, but hope maybe someone would post a better solution earlier
In order to find the owner I used this library with its examples:
from smb.SMBConnection import SMBConnection
conn = SMBConnection(username='<username>', password='<password>', domain=<domain>', my_name='<some pc name>', remote_name='<server name>')
conn.connect('<server name>')
sec_att = conn.getSecurity('<share name>', r'\some\file\path')
owner_sid = sec_att.owner
The problem is that pysmb package will only give you the owner's SID and not his name.
In order to get his name you need to make an ldap query like in this answer (reposting the code):
from ldap3 import Server, Connection, ALL
from ldap3.utils.conv import escape_bytes
s = Server('my_server', get_info=ALL)
c = Connection(s, 'my_user', 'my_password')
c.bind()
binary_sid = b'....' # your sid must be in binary format
c.search('my_base', '(objectsid=' + escape_bytes(binary_sid) + ')', attributes=['objectsid', 'samaccountname'])
print(c.entries)
But of course nothing will be easy, it took me hours to find a way to convert a string SID to binary SID in python, and in the end this solved it:
# posting the needed functions and omitting the class part
def byte(strsid):
'''
Convert a SID into bytes
strdsid - SID to convert into bytes
'''
sid = str.split(strsid, '-')
ret = bytearray()
sid.remove('S')
for i in range(len(sid)):
sid[i] = int(sid[i])
sid.insert(1, len(sid)-2)
ret += longToByte(sid[0], size=1)
ret += longToByte(sid[1], size=1)
ret += longToByte(sid[2], False, 6)
for i in range(3, len(sid)):
ret += cls.longToByte(sid[i])
return ret
def byteToLong(byte, little_endian=True):
'''
Convert bytes into a Python integer
byte - bytes to convert
little_endian - True (default) or False for little or big endian
'''
if len(byte) > 8:
raise Exception('Bytes too long. Needs to be <= 8 or 64bit')
else:
if little_endian:
a = byte.ljust(8, b'\x00')
return struct.unpack('<q', a)[0]
else:
a = byte.rjust(8, b'\x00')
return struct.unpack('>q', a)[0]
... AND finally you have the full solution! enjoy :(
I'm adding this answer to let you know of the option of using smbprotocol; as well as expand in case of misunderstood terminology.
SMBProtocol Owner Info
It is possible to get the SID using the smbprotocol library as well (just like with the pysmb library).
This was brought up in the github issues section of the smbprotocol repo, along with an example of how to do it. The example provided is fantastic and works perfectly. An extremely stripped down version
However, this also just retrieves a SID and will need a secondary library to perform a lookup.
Here's a function to get the owner SID (just wrapped what's in the gist in a function. Including here in case the gist is deleted or lost for any reason).
import smbclient
from ldap3 import Server, Connection, ALL,NTLM,SUBTREE
def getFileOwner(smb: smbclient, conn: Connection, filePath: str):
from smbprotocol.file_info import InfoType
from smbprotocol.open import FilePipePrinterAccessMask,SMB2QueryInfoRequest, SMB2QueryInfoResponse
from smbprotocol.security_descriptor import SMB2CreateSDBuffer
class SecurityInfo:
# 100% just pulled from gist example
Owner = 0x00000001
Group = 0x00000002
Dacl = 0x00000004
Sacl = 0x00000008
Label = 0x00000010
Attribute = 0x00000020
Scope = 0x00000040
Backup = 0x00010000
def guid2hex(text_sid):
"""convert the text string SID to a hex encoded string"""
s = ['\\{:02X}'.format(ord(x)) for x in text_sid]
return ''.join(s)
def get_sd(fd, info):
""" Get the Security Descriptor for the opened file. """
query_req = SMB2QueryInfoRequest()
query_req['info_type'] = InfoType.SMB2_0_INFO_SECURITY
query_req['output_buffer_length'] = 65535
query_req['additional_information'] = info
query_req['file_id'] = fd.file_id
req = fd.connection.send(query_req, sid=fd.tree_connect.session.session_id, tid=fd.tree_connect.tree_connect_id)
resp = fd.connection.receive(req)
query_resp = SMB2QueryInfoResponse()
query_resp.unpack(resp['data'].get_value())
security_descriptor = SMB2CreateSDBuffer()
security_descriptor.unpack(query_resp['buffer'].get_value())
return security_descriptor
with smbclient.open_file(filePath, mode='rb', buffering=0,
desired_access=FilePipePrinterAccessMask.READ_CONTROL) as fd:
sd = get_sd(fd.fd, SecurityInfo.Owner | SecurityInfo.Dacl)
# returns SID
_sid = sd.get_owner()
try:
# Don't forget to convert the SID string-like object to a string
# or you get an error related to "0" not existing
sid = guid2hex(str(_sid))
except:
print(f"Failed to convert SID {_sid} to HEX")
raise
conn.search('DC=dell,DC=com',f"(&(objectSid={sid}))",SUBTREE)
# Will return an empty array if no results are found
return [res['dn'].split(",")[0].replace("CN=","") for res in conn.response if 'dn' in res]
to use:
# Client config is required if on linux, not if running on windows
smbclient.ClientConfig(username=username, password=password)
# Setup LDAP session
server = Server('mydomain.com',get_info=ALL,use_ssl = True)
# you can turn off raise_exceptions, or leave it out of the ldap connection
# but I prefer to know when there are issues vs. silently failing
conn = Connection(server, user="domain\username", password=password, raise_exceptions=True,authentication=NTLM)
conn.start_tls()
conn.open()
conn.bind()
# Run the check
fileCheck = r"\\shareserver.server.com\someNetworkShare\someFile.txt"
owner = getFileOwner(smbclient, conn, fileCheck)
# Unbind ldap session
# I'm not clear if this is 100% required, I don't THINK so
# but better safe than sorry
conn.unbind()
# Print results
print(owner)
Now, this isn't super efficient. It takes 6 seconds for me to run this one a SINGLE file. So if you wanted to run some kind of ownership scan, then you probably want to just write the program in C++ or some other low-level language instead of trying to use python. But for something quick and dirty this does work. You could also setup a threading pool and run batches. The piece that takes longest is connecting to the file itself, not running the ldap query, so if you can find a more efficient way to do that you'll be golden.
Terminology Warning, Owner != Creator/Author
Last note on this. Owner != File Author. Many domain environments, and in particular SMB shares, automatically alter ownership from the creator to a group. In my case the results of the above is:
What I was actually looking for was the creator of the file. File creator and modifier aren't attributes which windows keeps track of by default. An administrator can enable policies to audit file changes in a share, or auditing can be enabled on a file-by-file basis using the Security->Advanced->Auditing functionality for an individual file (which does nothing to help you determine the creator).
That being said, some applications store that information for themselves. For example, if you're looking for Excel this answer provides a method for which to get the creator of any xls or xlsx files (doesn't work for xlsb due to the binary nature of the files). Unfortunately few files store this kind of information. In my case I was hoping to get that info for tblu, pbix, and other reporting type files. However, they don't contain this information (which is good from a privacy perspective).
So in case anyone finds this answer trying to solve the same kind of thing I did - Your best bet (to get actual authorship information) is to work with your domain IT administrators to get auditing setup.
I need to edit the remote file.
Right now, I'm logging into the machine via SSH. I can execute commands and get back response etc.
I am facing difficulty to modify the remote file.
Source Machine : Windows
Destination : Linux
Is anything like, get the file to Windows machine and edit it and then again transfer the file to Linux ? or any other better way ?
import SSHLibrary
s = SSHLibrary.SSHLibrary()
s.open_connection("10.10.10.10",username, password)
#s.write("sudo vi file_name_along_with_path") it has to force edit the file
# any ftp mechanism would be better
Can you please help me ?
try python paramiko and linux cat and echo not vi.
import paramiko
host = 'test.example.com'
username='host_user_name'
password='host_password'
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname=host, username=username, password=password)
stdin, stdout, stderr = ssh.exec_command("cat path_to_file_for_read")
all_lines = ''
for line in stdout.readlines():
all_lines += line
new_line = all_lines + 'add more or edit'
print new_line
stdin, stdout, stderr = ssh.exec_command("echo '{}' >> path_of_file_to_write".format(new_line))
ssh.close()
Use SFTP (SSH file transfer protocol). That's a protocol designed for file transfers.
import paramiko
host = "sftp.example.com"
transport = paramiko.Transport(host)
transport.connect(username = "username", password = "password")
sftp = paramiko.SFTPClient.from_transport(transport)
# Download
filepath = '/remote/path/file.txt'
localpath = '/tmp/file.txt'
sftp.get(filepath, localpath)
# Open in your favorite editor here
# Upload back
sftp.put(localpath, filepath)
# Close
sftp.close()
transport.close()
(base on paramiko's sshclient with sftp)
I am trying to change the directory of the couch database. I am using a python script to import a csv file to the CouchDB. Script is running ok. Here it is just in case:
from couchdbkit import Server, Database
from couchdbkit.loaders import FileSystemDocsLoader
from csv import DictReader
import sys, subprocess, math, os
def parseDoc(doc):
for k,v in doc.items():
if (isinstance(v,str)):
#print k, v, v.isdigit()
# #see if this string is really an int or a float
if v.isdigit()==True: #int
doc[k] = int(v)
else: #try a float
try:
if math.isnan(float(v))==False:
doc[k] = float(v)
except:
pass
return doc
def upload(db, docs):
db.bulk_save(docs)
del docs
return list()
def uploadFile(fname, dbname):
#connect to the db
theServer = Server()
db = theServer.get_or_create_db(dbname)
#loop on file for upload
reader = DictReader(open(fname, 'rU'), dialect = 'excel')
docs = list()
checkpoint = 100
i = 0
for doc in reader:
newdoc = parseDoc(doc)
docs.append(newdoc)
if len(docs)%checkpoint==0:
docs = upload(db,docs)
i += 1
print 'Number : %d' %i
#don't forget the last batch
docs = upload(db,docs)
if __name__=='__main__':
x = '/media/volume1/Crimes_-_2001_to_present.csv'
filename = x
dbname = 'test'
uploadFile(filename, dbname)
I saw plenty posts on how to change the directory for appending the database. If I leave the /etc/couchdb/local.ini as it is (original after installation) the script is appending data to the default directory /var/lib/couchdb/1.0.1/. When I modify the local.ini to store the database to another disk:
database_dir = /media/volume1
view_index_dir = /media/volume1
and after the reboot of the CouchDB service I get this error :
restkit.errors.RequestError: socket.error: [Errno 111] Connection refused
I have checked the open sockets (couchdb uses 5984 as default) and it is not opened. But I get no errors when I start CouchDB service.
Any ideas how to fix it ?
I think the error may be due to you have changed the directory location in Local.ini but when you are trying to make new connection to existing database, it cannot find it there.
So move the database_name.couch file to new location which you can put in local.ini and then try to make a connection. I think this should work.