I am trying to connect to memsql (running as docker container - cluster-in-a-box). I am using Python3.9. Tried with Python 3.8 as well.
Here is the code snippet:
from memsql.common import database
conn = database.connect(host="127.0.0.1", port=3306, user="root")
print(conn.query("show databases"))
When i run this, I am getting the following error:
Traceback (most recent call last):
File "/Users/ngarg/PycharmProjects/memsqlKafka/startup_try.py", line 3, in <module>
conn = database.connect(host="127.0.0.1", port=3306, user="root")
File "/Users/ngarg/Library/Python/3.9/lib/python/site-packages/memsql/common/database.py", line 19, in connect
return Connection(*args, **kwargs)
File "/Users/ngarg/Library/Python/3.9/lib/python/site-packages/memsql/common/database.py", line 62, in __init__
self.reconnect()
File "/Users/ngarg/Library/Python/3.9/lib/python/site-packages/memsql/common/database.py", line 93, in reconnect
conn = _mysql.connect(**self._db_args)
TypeError: 'db' is an invalid keyword argument for connect()
Try to google this, but didn’t find anything. I am blocked on this step.Any help is appreciated.
When you connect from the latest version of Django to SingleStore DB, you might receive the following error message:
django.db.utils.OperationalError: (2012, 'Error in server handshake')
To connect to SingleStore DB, you will need to configure the auth_plugin in the OPTIONS field.
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'HOST': '{HOST}',
'NAME': '{DBNAME}',
'USER': '{USERNAME}',
'PASSWORD' : '{PASSWORD}',
'PORT': '3306',
'OPTIONS': {
'auth_plugin': 'mysql_native_password'
}
}
}
https://support.singlestore.com/hc/en-us/articles/360057857552-Connecting-Django-to-SingleStore-DB
Related
python 3.11 / asyncssh 2.13 / psycopg2-binary 2.9.5
After using the sshtunnel library i wanted to switch to the asyncssh library because its more maintained with newer Python Versions and brings async benefits.
After reading the asyncssh Doc i wrote a little test script and could connect via SSH from my local computer to my Server. I execute some ls terminal statement on my server and could print the outcome with my python script. After that small success i wanted to connect to my postgreSQL Database via the asyncssh Tunnel, so i can push my local panda data to my server database.
In the past it worked well with the sshtunnel library. Unfortunately i fail with asyncssh library to establish a connection to my database on my server.
Problem
My main Problem is to wrap psycopg2 into the tunnel like in sshtunnel with remote_bind_address. I tried with forward_local_port or forward_remote_port and it seems the connection is established but how to funnel psycopg2 into it? Instead of connecting to my 'Server-Database Port 5432' i should connect to the tunnel port (see example 2)?
Example: How sshtunnel worked.
(this is an from geo.rocks but i used that structure also).
import psycopg2
from sshtunnel import SSHTunnelForwarder
try:
with SSHTunnelForwarder(
('some.linktodb.com', 22), # port 22 as standard SSH port
ssh_username="username",
ssh_pkey="your/private/key", # your private key file
ssh_private_key_password="****",
remote_bind_address=('localhost', 5432)) as server: # mirroring to local port 5432
server.start()
params = { # database params
'database': 'test',
'user': 'dome',
'password': '*****',
'host': 'localhost',
'port': server.local_bind_port
}
conn = psycopg2.connect(**params)
curs = conn.cursor() # if this works, you are connected
print("DB connected")
except:
print("Connection failed")
Example: My current asyncssh approach.
async def run_client() -> None:
async with asyncssh.connect(
host=os.environ.get('SSH_HOST'),
known_hosts=None,
username=os.environ.get('SSH_USER'),
passphrase=os.environ.get('SSH_PW'),
client_keys=os.environ.get('SSH_PRIVAT_KEY')) as tunnel:
listener = await tunnel.forward_local_port(
listen_host='',
listen_port=8084,
dest_host='127.0.0.1',
dest_port=5432)
conn = psycopg2.connect(
host=os.environ.get('DB_HOST'),
port=os.environ.get('DB_PORT'), # <- `local_bind_port` like in sshtunnel?
database=os.environ.get('DB_NAME'),
user=os.environ.get('DB_USER'),
password=os.environ.get('DB_PW'),
)
OUTPUT:
...
[ 2023-01-23,12:17:17.+0100 ] [INFO] logging.py - log - [conn=0] Creating local TCP forwarder from port 8084 to 127.0.0.1, port 5432
[ 2023-01-23,12:17:17.+0100 ] [INFO] logging.py - log - [conn=0] Closing connection
[ 2023-01-23,12:17:17.+0100 ] [INFO] logging.py - log - [conn=0] Sending disconnect: Disconnected by application (11)
[ 2023-01-23,12:17:17.+0100 ] [INFO] logging.py - log - [conn=0] Connection closed
Traceback (most recent call last):
File "/home/user/atlas/user_atlas/mono/tutorials/python/db_ssh_activity.py", line 135, in <module>
loop.run_until_complete(coroutine)
File "/usr/lib64/python3.11/asyncio/base_events.py", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/home/user/atlas/user_atlas/mono/tutorials/python/db_ssh_activity.py", line 125, in main
await run_client()
File "/home/user/atlas/user_atlas/mono/tutorials/python/db_ssh_activity.py.py", line 82, in run_client
conn = psycopg2.connect(
^^^^^^^^^^^^^^^^^
File "/home/user/atlas/mono/mono_env/lib64/python3.11/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.OperationalError: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?
EXAMPLE 2:
Instead of connecting to my 'Server-Database Port 5432' i should connect to the tunnel port
async def run_client() -> None:
async with asyncssh.connect(
host=os.environ.get('SSH_HOST'),
known_hosts=None,
username=os.environ.get('SSH_USER'),
passphrase=os.environ.get('SSH_PW'),
client_keys=os.environ.get('SSH_PRIVAT_KEY')) as tunnel:
tunnel.forward_remote_port
listener = await tunnel.forward_local_port(
listen_host='localhost',
listen_port=1,
dest_host='127.0.0.1',
dest_port=5432)
conn = psycopg2.connect(
host='localhost',
port=listener.get_port(),
database=os.environ.get('DB_NAME'),
user=os.environ.get('DB_USER'),
password=os.environ.get('DB_PW'),
)
listener.wait_closed()
OUTPUT:
[ 2023-01-23,12:40:46.+0100 ] [INFO] logging.py - log - [conn=0] Auth for user hendrix succeeded
[ 2023-01-23,12:40:46.+0100 ] [INFO] logging.py - log - [conn=0] Creating local TCP forwarder from localhost, port 1 to 127.0.0.1, port 5432
[ 2023-01-23,12:40:46.+0100 ] [DEBUG] logging.py - log - [conn=0] Failed to create local TCP listener: [Errno 13] error while attempting to bind on address ('::1', 1, 0, 0): Permission denied
[ 2023-01-23,12:40:46.+0100 ] [INFO] logging.py - log - [conn=0] Closing connection
[ 2023-01-23,12:40:46.+0100 ] [INFO] logging.py - log - [conn=0] Sending disconnect: Disconnected by application (11)
[ 2023-01-23,12:40:46.+0100 ] [INFO] logging.py - log - [conn=0] Connection closed
Traceback (most recent call last):
File "/home/user/atlas/mono/monokapi_jupyter/tutorials/python/db_ssh_activity.py", line 136, in <module>
loop.run_until_complete(coroutine)
File "/usr/lib64/python3.11/asyncio/base_events.py", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/home/user/atlas/user_atlas/mono/tutorials/python/db_ssh_activity.py", line 126, in main
await run_client()
File "/home/user/atlas/user_atlas/mono/tutorials/python/db_ssh_activity.py.py", line 76, in run_client
listener = await tunnel.forward_local_port(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/atlas/user_atlas/mono_env/lib64/python3.11/site-packages/asyncssh/connection.py", line 2944, in forward_local_port
listener = await create_tcp_forward_listener(self, self._loop,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/atlas/user_atlas/mono_env/lib64/python3.11/site-packages/asyncssh/listener.py", line 341, in create_tcp_forward_listener
return await create_tcp_local_listener(conn, loop, protocol_factory,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/atlas/user_atlas/mono_env/lib64/python3.11/site-packages/asyncssh/listener.py", line 314, in create_tcp_local_listener
raise OSError(exc.errno, 'error while attempting ' # type: ignore
PermissionError: [Errno 13] error while attempting to bind on address ('::1', 1, 0, 0): Permission denied
why my script run on local phycharm but when i upload my script on VPS ( host )i get errors ?
my python code
`
import json
import socketio
TOKEN = "my-super-token" #You donation alert token
sio = socketio.Client()
#sio.on('connect')
def on_connect():
sio.emit('add-user', {"token": TOKEN, "type": "alert_widget"})
#sio.on('donation')
def on_message(data):
y = json.loads(data)
print(y['username'])
print(y['message'])
print(y['amount'])
print(y['currency'])
sio.connect('wss://socket.donationalerts.ru:443',transports='websocket')
`
from localhost , pycharmall work but from host i get this
error screenshot here
error :
Traceback (most recent call last): File "test.py", line 22, in
sio.connect('wss://socket.donationalerts.ru:443',transports='websocket')
File "/usr/local/lib/python3.8/site-packages/socketio/client.py", line
338, in connect
raise exceptions.ConnectionError(exc.args[0]) from None socketio.exceptions.ConnectionError: Connection error
i try install ,reinstall python , packages etc
I'm trying to connect MongoDB with Django.
settings.py
DATABASES = {
'default': {
'ENGINE': 'djongo',
'NAME': '<name>',
'ENFORCE_SCHEMA': False,
'CLIENT': {
'host': f'mongodb+srv://{mongodb_username}:{mongodb_password}#{mongodb_cluster}/?retryWrites=true',
'uuidRepresentation': 'standard',
'waitQueueTimeoutMS': 30000
}
}
}
models.py
import uuid
from django.db import models
# Create your models here.
class ModernConnectUsers(models.Model):
user_id = models.UUIDField(primary_key=True, default=uuid.uuid4())
user_name = models.CharField(max_length=30, null=False)
The model has not been used anywhere for now.
python manage.py makemigrations && python manage.py migrate
outputs:
Migrations for 'modern_connect':
modern_connect/migrations/0003_alter_modernconnectusers_user_id.py
- Alter field user_id on modernconnectusers
Operations to perform:
Apply all migrations: admin, auth, contenttypes, modern_connect, sessions
Running migrations:
No migrations to apply.
Your models in app(s): 'modern_connect' have changes that are not yet reflected in a migration, and so won't be applied.
Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
Traceback (most recent call last):
File "manage.py", line 22, in <module>
main()
File "manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/home/sarvesh/PycharmProjects/modernConnect/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
utility.execute()
File "/home/sarvesh/PycharmProjects/modernConnect/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 440, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/sarvesh/PycharmProjects/modernConnect/venv/lib/python3.8/site-packages/django/core/management/base.py", line 414, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/sarvesh/PycharmProjects/modernConnect/venv/lib/python3.8/site-packages/django/core/management/base.py", line 460, in execute
output = self.handle(*args, **options)
File "/home/sarvesh/PycharmProjects/modernConnect/venv/lib/python3.8/site-packages/django/core/management/base.py", line 98, in wrapped
res = handle_func(*args, **kwargs)
File "/home/sarvesh/PycharmProjects/modernConnect/venv/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 317, in handle
emit_post_migrate_signal(
File "/home/sarvesh/PycharmProjects/modernConnect/venv/lib/python3.8/site-packages/django/core/management/sql.py", line 52, in emit_post_migrate_signal
models.signals.post_migrate.send(
File "/home/sarvesh/PycharmProjects/modernConnect/venv/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 176, in send
return [
File "/home/sarvesh/PycharmProjects/modernConnect/venv/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 177, in <listcomp>
(receiver, receiver(signal=self, sender=sender, **named))
File "/home/sarvesh/PycharmProjects/modernConnect/venv/lib/python3.8/site-packages/django/contrib/auth/management/__init__.py", line 83, in create_permissions
ctypes.add(ctype)
File "/home/sarvesh/PycharmProjects/modernConnect/venv/lib/python3.8/site-packages/django/db/models/base.py", line 597, in __hash__
raise TypeError("Model instances without primary key value are unhashable")
TypeError: Model instances without primary key value are unhashable
The only collection I'm currently using has a Primary Key, but for some reason - The program is falling for self.pk == None in Base.py which is raising this error. Solutions under other questions didn't help, so starting another question.
In your models.py, try importing the models from djongo.
from djongo import models
And for primary key I use
class User(models.Model):
_id = models.ObjectIdField()
But you can play around with uuid() and see if it works or not.
I use that guide to retrieve data from my Fingerbot: https://www.home-assistant.io/integrations/tuya/
I also use that repo to extract the data:
https://github.com/redphx/tuya-local-key-extractor
When I use the extract.py file I get that error:
Exception in thread Thread-1:
Traceback (most recent call last):
openapi <tuya_iot.openapi.TuyaOpenAPI object at 0x7f52fff924c0>
File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
openmq <TuyaOpenMQ(Thread-1, started 139994399807232)>
self.run()
File "/home/xavier/.local/lib/python3.8/site-packages/tuya_iot/openmq.py", line 158, in run
self.__run_mqtt()
File "/home/xavier/.local/lib/python3.8/site-packages/tuya_iot/openmq.py", line 164, in __run_mqtt
mq_config = self._get_mqtt_config()
File "/home/xavier/.local/lib/python3.8/site-packages/tuya_iot/openmq.py", line 67, in _get_mqtt_config
"uid": self.api.token_info.uid,
AttributeError: 'NoneType' object has no attribute 'uid'
Traceback (most recent call last):
File "./extract.py", line 41, in <module>
device_manager.update_device_list_in_smart_home()
File "/home/xavier/.local/lib/python3.8/site-packages/tuya_iot/device.py", line 239, in update_device_list_in_smart_home
response = self.api.get(f"/v1.0/users/{self.api.token_info.uid}/devices")
AttributeError: 'NoneType' object has no attribute 'uid'
Here is the code I use:
import json
import logging
import os
from config import (
ENDPOINT,
APP,
EMAIL,
PASSWORD,
ACCESS_ID,
ACCESS_KEY,
)
from tuya_iot import (
TuyaOpenAPI,
AuthType,
TuyaOpenMQ,
TuyaDeviceManager,
)
openapi = TuyaOpenAPI(ENDPOINT, ACCESS_ID, ACCESS_KEY, AuthType.SMART_HOME)
openapi.connect(EMAIL, PASSWORD, country_code=84, schema=APP.value)
openmq = TuyaOpenMQ(openapi)
openmq.start()
print('openapi {}'.format(openapi))
print('openmq {}'.format(openmq))
device_manager = TuyaDeviceManager(openapi, openmq)
device_manager.update_device_list_in_smart_home()
devices = []
for tuya_device in device_manager.device_map.values():
device = {
'device_id': tuya_device.id,
'device_name': tuya_device.name,
'product_id': tuya_device.product_id,
'product_name': tuya_device.product_name,
'category': tuya_device.category,
'uuid': tuya_device.uuid,
'local_key': tuya_device.local_key,
}
try:
resp = openapi.get('/v1.0/iot-03/devices/factory-infos?device_ids={}'.format(tuya_device.id))
factory_info = resp['result'][0]
if 'mac' in factory_info:
mac = ':'.join(factory_info['mac'][i:i + 2] for i in range(0, 12, 2))
device['mac_address'] = mac
except Exception as e:
print(e)
devices.append(device)
print(json.dumps(devices, indent=2))
os._exit(0)
I am on Ubuntu 20.04 and I install the python sdk with tha t command:
pip3 install tuya-iot-py-sdk
Overview:
Selenium scraper works perfectly in headless mode. Spawning a virtual display shows no errors via XVFB:
from xvfbwrapper import Xvfb
vdisplay = Xvfb()
vdisplay.start()
vdisplay.stop()
But when I try to run them together, it errors out with:
[ERROR] SessionNotCreatedException: Message: session not created
from disconnected: Unable to receive message from renderer (Session info: chrome=96.0.4664.0)
Traceback:
Traceback (most recent call last):
File "/var/task/slack_main.py", line 34, in handler
scrape_price(asin_list)
File "/var/task/slack_main.py", line 58, in scrape_price
driver = webdriver.Chrome("/opt/chromedriver",options=options)
File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/chrome/webdriver.py", line 70, in __init__
super(WebDriver, self).__init__(DesiredCapabilities.CHROME['browserName'], "goog",
File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/chromium/webdriver.py", line 92, in __init__
RemoteWebDriver.__init__(
File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 275, in __init__
self.start_session(capabilities, browser_profile)
File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 365, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 430, in execute
self.error_handler.check_response(response)
File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response
raise exception_class(message, screen, stacktrace)
Configuration:
Below is my complete Selenium and XVFB configuration:
from selenium import webdriver
from selenium_stealth import stealth
from xvfbwrapper import Xvfb
vdisplay = Xvfb()
vdisplay.start()
options = webdriver.ChromeOptions()
prefs = {"browser.downloads.dir": "//tmp//", "download.default_directory": "//tmp//", "directory_upgrade": True}
options.add_experimental_option("prefs", prefs)
options.binary_location = '/opt/chrome/chrome'
#options.add_argument('--headless') #toggled on and off when running with or without XVFB
options.add_argument('--no-sandbox')
options.add_argument("--disable-gpu")
options.add_argument("--window-size=1280x1696")
options.add_argument("--single-process")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--disable-dev-tools")
options.add_argument("--no-zygote")
options.set_capability('unhandledPromptBehavior', 'ignore')
options.add_argument("download.default_directory=/tmp")
driver = webdriver.Chrome("/opt/chromedriver",options=options)
stealth(driver,
languages=["en-US", "en"],
vendor="Google Inc.",
platform="Win32",
fix_hairline=True,
)
vdisplay.stop()
driver.close()
Why is it not connecting to the display? My guess is it has something to do with the '--headless' toggle?
Versions and Tools:
Selenium version 3.141.0
xvfbwrapper version 0.2.9
Docker is used to compile and push to AWS Lambda, base image used (no changes are made in docker file with or without XVFB)
Edit:
Found a pull request for XVFB configuration in the Github Repo of my base image. Even used the exact same code from the pull request and I still recieve the exact same error. Maybe this has something to do with AWS offboard?