I use impyla and ibis to connect hive server, but I got the error.
I tried the following code:
from impala.dbapi import connect
impcur = connect(host="kudu3", port=10000, database="yingda_test", password=None, user='admin', kerberos_service_name='None').cursor()
The new error came out:
Traceback (most recent call last):
File "/Users/edy/src/PythonProjects/dt-center-algorithm/test/1.py", line 4, in <module>
impcur = connect(host="kudu3", port=10000, database="yingda_test", password=None, user='admin', kerberos_service_name='None').cursor()
File "/usr/local/conda3/envs/py37/lib/python3.7/site-packages/impala/hiveserver2.py", line 129, in cursor
session = self.service.open_session(user, configuration)
File "/usr/local/conda3/envs/py37/lib/python3.7/site-packages/impala/hiveserver2.py", line 1187, in open_session
resp = self._rpc('OpenSession', req, True)
File "/usr/local/conda3/envs/py37/lib/python3.7/site-packages/impala/hiveserver2.py", line 1080, in _rpc
response = self._execute(func_name, request, retry_on_http_error)
File "/usr/local/conda3/envs/py37/lib/python3.7/site-packages/impala/hiveserver2.py", line 1142, in _execute
.format(self.retries))
impala.error.HiveServer2Error: Failed after retrying 3 times
thrift 0.15.0
thrift-sasl 0.4.3
thriftpy2 0.4.14
pure-sasl 0.6.2
sasl 0.2.1
thrift-sasl 0.4.3
ibis-framework 2.0.0
impyla 0.17.0
python version: 3.7.12 with anaconda
And I have tried ibis-1.3.0 and 2.0 version. Can u guys give some advices? tks a lot
I also met this problem.
My code is:
from impala.dbapi import connect
import psycopg2
conn_hive = connect(host="xxx.xxx.xxx.xxx", port=xxx, user='admin',
password='password', database='xxx', auth_mechanism="PLAIN", timeout=6000)
hive_cursor = conn_hive.cursor()
hive_cursor.execute(query_sql)
data_list = hive_cursor.fetchall()
...get data...
hive_cursor.close()
conn_hive.close()
After my colleagues and I tried, we find it will be successful when we reconnect hive manually.
It means if you want to get different data from the same hive database, you had better close the connection and reconnect hive manually by the following codes:
conn_hive = connect(host="xxx.xxx.xxx.xxx", port=xxx, user='admin',
password='password', database='xxx', auth_mechanism="PLAIN", timeout=6000)
hive_cursor = conn_hive.cursor()
hive_cursor.execute(query_sql)
data_list = hive_cursor.fetchall()
...get data1...
hive_cursor.close()
conn_hive.close()
conn_hive = connect(host="xxx.xxx.xxx.xxx", port=xxx, user='admin',
password='password', database='xxx', auth_mechanism="PLAIN", timeout=6000)
hive_cursor = conn_hive.cursor()
hive_cursor.execute(query_sql)
data_list = hive_cursor.fetchall()
...get data2...
hive_cursor.close()
conn_hive.close()
Finally, my colleagues told me that there was a problem with impala recently.
Related
I keep getting an error with Pandas' 0.25.0 read_sql(). The code below is supposed to establish a couple connections, check if tables need to be created, and then begin extracting the result set from Oracle.
Stack Trace
RESTART: C:\xxxx\xxxx\AppData\Local\Programs\Python\Python36\xxxx\main.py
Creating connections
Attempting to connect
Connection to box_db Successful
Attempting to connect
Connection to JDE Successful
Box table already existed
BoxStaging table already existed
LotStartSiteMap table already existed
Extracting from JDE
Traceback (most recent call last):
File "C:\xxxx\xxxx\AppData\Local\Programs\Python\Python36\xxxx\main.py", line 42, in <module>
main()
File "C:\xxxx\xxxx\AppData\Local\Programs\Python\Python36\xxxx\main.py", line 26, in main
df = data.box_data_extract(queries.BOX_QUERY, jde_conn)
File "C:\xxxx\xxxx\AppData\Local\Programs\Python\Python36\xxxx\data.py", line 46, in box_data_extract
pd.read_sql_query(query, connection)
File "C:\xxxx\xxxx\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\io\sql.py", line 332, in read_sql_query
chunksize=chunksize,
File "C:\xxxx\xxxx\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\io\sql.py", line 1645, in read_query
cursor = self.execute(*args)
File "C:\xxxx\xxxx\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\io\sql.py", line 1590, in execute
cur = self.con.cursor()
AttributeError: module 'connection' has no attribute 'cursor'
main.py
import connection
import credential
import data
import emailer
import queries
import tables
def main():
# Create connection to box_db file and JDE
#
print("Creating connections")
sqlite_conn = connection.box_db_connect(tables.DB_PATH)
jde_conn = connection.mrap0680_connect(credential.USERNAME, credential.MRAP0680_PASSWORD)
# If new path has been chosen create sqlite db and tables accordingly
#
tables.create_box_table(sqlite_conn)
tables.create_box_staging_table(sqlite_conn)
tables.create_lot_startsite_map_table(sqlite_conn)
# Extract & insert JDE Box Query results into BoxStaging table
#
print("Extracting from JDE")
df = data.box_data_extract(queries.BOX_QUERY, jde_conn) #############LINE CAUSING ERROR
print("Storing in box_db")
data.box_data_insert(df, sqlite_conn)
print("BoxStaging updated")
data.py
# Extract the box data from JDE
#
def box_data_extract(query, connnection):
try:
df = pd.read_sql(query, con=connection)
print("JDE Box Data extracted")
return df
except Exception as e:
print(repr(e))
connection.py
def xxxx_connect(username, password):
# Connect to JDE database
try:
print("Attempting to connect")
conn = cx_Oracle.connect(user=username, password=password, dsn=DSN_TNS)
print("Connection to JDE Successful")
return conn
except Exception as e:
print(repr(e))
I am able to extract the data if running manually.
Please help. Thanks.
I found the error! The issue was in data.py. In the parameter list of data_box_extract, connection is spelled as "connnection". Inside the function, "connection" was being used so a cursor couldn't be instantiated. Not sure why it didn't throw an error about "connection" not being instantiated. Oh well, I'm writing the program in IDLE, maybe it's time to switch to an IDE that will catch those mistakes...
Thanks to all who viewed!
I am unable to create a database file using python. I am facing this issue only in windows. On my Mac machine i was able to get it working. Here is a test code i tried -
import os, sys, shutil, glob, pandas, sqlite3
db_path = 'E:\\Archive.db'
conn = sqlite3.connect(db_path)
Here are other versions of variable db_path i have tried
db_path = 'E:\Archive.db'
db_path = 'E:/Archive.db'
db_path = 'E://Archive.db'
But i always get following error -
Traceback (most recent call last):
File "test.py", line 4, in <module>
conn = sqlite3.connect(db_path)
sqlite3.OperationalError: unable to open database file
I do have write permissions on E:
I was able to create csv files using python script. Not sure why its stuck with database. It should create the database in E:
Following example taken from here works for me. Try this out. If this doesn't work, try to create a file in your documents folder by changing the path
import sqlite3
from sqlite3 import Error
def create_connection(db_file):
""" create a database connection to a SQLite database """
conn = None
try:
conn = sqlite3.connect(db_file)
print(sqlite3.version)
except Error as e:
print(e)
finally:
if conn:
conn.close()
if __name__ == '__main__':
create_connection(r"E:\Archive.db")
i wrote a script to connect to sftp server in python but it showing this error below and i do not understand it. please help me fix the bug
import pysftp
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
with pysftp.Connection(host="127.0.0.1", username="new34",password="password",cnopts=cnopts) as srv:
print("connection successful")
# Get the directory and file listing
data = srv.listdir()
srv.put("testfile.txt")
# Closes the connection
srv.close()
# Prints out the directories and files, line by line
for i in data:
print(i)
it shows the following error; please help to fix the bug
C:\Users\Rohan\PycharmProjects\untitled1\venv\Scripts\python.exe C:/Users/Rohan/PycharmProjects/untitled1/yu.py
C:\Users\Rohan\PycharmProjects\untitled1\venv\lib\site-packages\pysftp\__init__.py:61: UserWarning: Failed to load HostKeys from C:\Users\Rohan\.ssh\known_hosts. You will need to explicitly load HostKeys (cnopts.hostkeys.load(filename)) or disableHostKey checking (cnopts.hostkeys = None).
warnings.warn(wmsg, UserWarning)
Traceback (most recent call last):
connection successful
File "C:/Users/Rohan/PycharmProjects/untitled1/yu.py", line 10, in <module>
data = srv.listdir()
File "C:\Users\Rohan\PycharmProjects\untitled1\venv\lib\site-packages\pysftp\__init__.py", line 591, in listdir
self._sftp_connect()
File "C:\Users\Rohan\PycharmProjects\untitled1\venv\lib\site-packages\pysftp\__init__.py", line 205, in _sftp_connect
self._sftp = paramiko.SFTPClient.from_transport(self._transport)
File "C:\Users\Rohan\PycharmProjects\untitled1\venv\lib\site-packages\paramiko\sftp_client.py", line 164, in from_transport
chan = t.open_session(
AttributeError: 'NoneType' object has no attribute 'open_session'
Process finished with exit code 1
Your code has indentation issue.
Try this,
with pysftp.Connection(host="127.0.0.1", username="new34",password="password",cnopts=cnopts) as srv:
print("connection successful")
# Get the directory and file listing
data = srv.listdir()
srv.put("testfile.txt")
with automatically closes the connection. No need to close explicitly.
I learning python recent, but I have some error.
environment python3 , chrome , webdriver(chrome)
from selenium import webdriver
import time
import random
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
driver = webdriver.Chrome("./chromedriver.exe")
mobile_emulation = { "deviceName": 'Nexus 5' }
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("mobileEmulation", mobile_emulation)
driver = webdriver.Remote(command_executor='https:xxx.com',desired_capabilities = chrome_options.to_capabilities())
driver.get("https:/xxx.com")
num = random.randint(11111111 , 99999999)
red = driver.find_element_by_class_name("***")
red.click()
numBox = driver.find_element_by_name("***")
numBox.send_keys(int(num))
reader = driver.find_element_by_id("***")
reader.send_keys("***")
comment = driver.find_element_by_css_selector(" ***")
comment.click()
and result error is here
Traceback (most recent call last):
File "C:\python\pad\pad.py", line 16, in <module>
driver = webdriver.Remote(command_executor='https:xxx.com',desired_capabilities = chrome_options.to_capabilities())
File "C:\Users\***\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 156, in __init__
self.start_session(capabilities, browser_profile)
File "C:\Users\***\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 254, in start_session
self.session_id = response['sessionId']
TypeError: string indices must be integers
I think the error because number of this code includes　Decimal. but I cant find such number .
please give me advise
This error message...
Traceback (most recent call last):
File "C:\python\pad\pad.py", line 16, in <module>
driver = webdriver.Remote(command_executor='https:xxx.com',desired_capabilities = chrome_options.to_capabilities())
.
TypeError: string indices must be integers
...implies that there was a TypeError while invoking webdriver.Remote() method.
As per your code trials as you are using webdriver.Remote() with the argument command_executor perhaps you were trying to execute your tests in Selenium Grid Configuration.
As per the documentation documentation:
command_executor : remote_connection.RemoteConnection object used to execute commands.
Example:
command_executor='http://127.0.0.1:4444/wd/hub'
The complete implementation:
driver = webdriver.Remote(command_executor='http://127.0.0.1:4444/wd/hub', desired_capabilities = chrome_options.to_capabilities())
Note: Here we have considered that the Selenium Grid Hub and Selenium Grid Node are configured, up and running successfully with default configuration on the localhost.
Solution (Python 3.6)
Your effective code block will be:
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("start-maximized")
chrome_options.add_argument('disable-infobars')
#driver = webdriver.Remote(command_executor='https:xxx.com', desired_capabilities = chrome_options.to_capabilities())
driver = webdriver.Remote(command_executor='http://127.0.0.1:4444/wd/hub', desired_capabilities = chrome_options.to_capabilities())
driver.get('https://www.google.co.in')
print("Page Title is : %s" %driver.title)
driver.quit()
I'm trying to use PyMySQL 0.79 with python 3.5.0 under Windows 10 but get a connect error when running a simple program I called 'test.py'...
import pymysql
db = pymysql.connect( host = 'sqlserver.example.com', passwd 'SECRET12345', user = 'dbuser', db='myDatabase')
cursor = db.cursor()
sql = "INSERT INTO people (name, email, age) VALUES (%s, %s, %s)"
cursor.execute( sql, ( "Jay", "jay#example.com", '77') )
cursor.close()
db.commit() #Makes sure the DB saves your changes!
db.close()
But the above code gives me the error message:
C:\Users\Avtar\AppData\Local\Programs\Python\Python35-32\Lib\site-packages\pymysql>test.py
Traceback (most recent call last):
File "C:\Users\Avtar\AppData\Local\Programs\Python\Python35-32\lib\site-packages\pymysql\connections.py", line 890, in connect
(self.host, self.port), self.connect_timeout)
File "C:\Users\Avtar\AppData\Local\Programs\Python\Python35-32\lib\socket.py", line 689, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
File "C:\Users\Avtar\AppData\Local\Programs\Python\Python35-32\lib\socket.py", line 728, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 11001] getaddrinfo failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Avtar\AppData\Local\Programs\Python\Python35-32\Lib\site-packages\pymysql\test.py", line 2, in <module>
db = pymysql.connect( host = 'sqlserver.example.com', passwd = 'SECRET12345', user = 'dbuser', db='myDatabase')
File "C:\Users\Avtar\AppData\Local\Programs\Python\Python35-32\lib\site-packages\pymysql\__init__.py", line 90, in Connect
return Connection(*args, **kwargs)
File "C:\Users\Avtar\AppData\Local\Programs\Python\Python35-32\lib\site-packages\pymysql\connections.py", line 688, in __init__
self.connect()
File "C:\Users\Avtar\AppData\Local\Programs\Python\Python35-32\lib\site-packages\pymysql\connections.py", line 937, in connect
raise exc
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'sqlserver.example.com' ([Errno 11001] getaddrinfo failed)")
I installed PyMySQL using the command 'pip install pymysql' and to check the installation I typed:
'pip list' which gives me:
pip (9.0.1)
PyMySQL (0.7.9)
setuptools (18.2)
yolk (0.4.3)
'pip show pymysql' which gives me:
Name: PyMySQL
Version: 0.7.9
Summary: Pure Python MySQL Driver
Home-page: https://github.com/PyMySQL/PyMySQL/
Author: INADA Naoki
Author-email: songofacandy#gmail.com
License: MIT
Location: c:\users\avtar\appdata\local\programs\python\python35-32\lib\site-packages
Requires:
I cannot tell from this what I am doing wrong, so would really appreciate if anyone can help me sort this. Thanks in advance.
First and foremost, hosts of type: "sqlserver.example.com", is usually Microsoft SQL Server , and not MySQL. If that's the case (ie you are using SQL Server), then instead of using the pymysql library, you should use a more generic library; such as pyodbc.
Based on a another stackoverflow question, for Python2.7, the way to use that library is as follows:
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=sqlserver.example.com;DATABASE=testdb;UID=me;PWD=pass')
cursor = cnxn.cursor()
However, if infact you are using MySQL, then your configuration is incorrect...
Your code, notice it does not have "=" between passwd and your password; nor does it have the port number.
db = pymysql.connect( host = 'sqlserver.example.com', passwd 'SECRET12345', user = 'dbuser', db='myDatabase')
As such I would suggest:
db = pymysql.connect( host = 'sqlserver.example.com', port=3306, passwd='SECRET12345', user = 'dbuser', db='myDatabase')
If your port is not 3306, and you are not sure which port your mysql is in, then to find the port, do the following:
mysql> select * from INFORMATION_SCHEMA.GLOBAL_VARIABLES WHERE VARIABLE_NAME LIKE 'PORT';
+---------------+----------------+
| VARIABLE_NAME | VARIABLE_VALUE |
+---------------+----------------+
| PORT | 2127 |
+---------------+----------------+
1 row in set (0.00 sec)