Airflow SFTP sensor doesn't exist - python-3.x

I am building a DAG that starts with an SFTPSensor Operator. As we can see, the SFTPSensor class takes an sftp_conn_id parameter. (Strangely, the SFTP Operator uses an SSH connection, which is what I would have expected from the sensor as well).
So I go to the UI to create an SFTP connection like the example sftp connection that ships with airlfow:
However this connection type does not exist among the selectable connection types. When I 'edit' the existing example, the 'type' shows up as Amazon Web Services:
Do I have to create this type of connection with the CLI as described here. So something like:
airflow connections -a \
--conn_id my_sftp_connection \
--conn_type sftp \
--conn_login **ux**\
--conn_port 22 \
--conn_host **host** \
--conn_extra "{"key_file": "**keypath**"}"

Both https://airflow.apache.org/docs/apache-airflow-providers-sftp/stable/index.html
and https://airflow.apache.org/docs/apache-airflow-providers-ssh/stable/index.html use Paramiko implementation of the SSHv2 protocol.
As described here http://docs.paramiko.org/en/stable/api/sftp.html paramiko sftp client uses an ssh transport to perform remote file operations.
You need to install both extra provider package's to see each connection type listed in the UI.
pip install apache-airflow-providers-sftp[ssh]

Related

FortiClient headless linux cli - how to install and configure to handle certain IP ranges only or permit SSH

I am trying to configure the headless VPN only FortiClient on an AWS ubuntu 20.04 ec2 instance, and though I am able to connect to the target, I am then disconnected from the instance and cannot progress.
Setup:
wget http://cdn.software-mirrors.com/forticlientsslvpn_linux_4.4.2328.tar.gz
tar -xzvf forticlientsslvpn_linux_4.4.2328.tar.gz
cd ./forticlientsslvpn/64bit/helper
sudo ./setup.linux.sh
# Accept license
cd ..
./forticlientsslvpn_cli --server serveraddress:port --vpnuser username
# Enter password
##Connected!
At this stage, I am booted out of the instance and cannot reconnect (requiring a soft restart of the instance to gain access again)
I can see that there is a configuration file at forticlientsslvpn/64bit/helper/config but I cannot find any documentation describing what can be configured there or whether it is something I should be concerned with.
The CLI itself doesn't take any other options other than:
forticlientsslvpn_cli [--proxy proxyaddress:proxyport] --server vpnserveraddress:vpnport [--proxyuser proxyuser] [--vpnuser vpnuser] [--pkcs12 pkcs12path] [--keepalive]
I would like to either:
Preserve my original SSH connection (and any future connections) so I can develop within the VPN or;
Limit the VPN to only package traffic that is going to a specific IP range (CIDR block)
I have found three different methods for installing the client (sudo apt install forticlient, sudo apt install -y openfortivpn, see above) and cannot navigate through them. I have looked into FortiClientLinuxGuide and installed that tool but couldn't find out how to configure it as a VPN instead (or where to add the configuration). Similar experience with the second one.
This seems to be the only documentation about how to configure the CLI and its just the bear minimum How to setup and install SSLVPN.
This post seems to be having the same problem ssh-telnet-disconnects and the solution looks like it would work if only I knew how to set that configuration.
alternatively, I have looked up split tunnel configuration which looks like it would be ideal but cannot work out how I would set that up. The documentation is only via the GUI Enable-split-tunnel-feature

Get all supported KexAlgorithms of an sftp server

I try to connect with an sftp server using the nodejs package https://github.com/mscdex/ssh2.
In the section "serverHostKey" they list all available host key formats.
I try to connect with a new server, but no matter which format I choose, I cant establish a connection because the format/algorithm is not supported.
Is there a way - a command for the command line - , to see which formats/algorithms a certain server supports?
I already tried WINSCP (shows me only the format WINSCP uses), Putty (same as WINSCP)-
https://superuser.com/questions/868998/how-can-i-find-a-list-of-macs-ciphers-and-kexalgorithms-that-my-openssh-client
The second answer of this question, is the answer of my problem.
ssh USER#HOST -p PORT

AWS EMR: Spark - SparkException java IOException: Failed to create local dir in /tmp/blockmgr*

I have a AWS EMR cluster with Spark. I can connect to it (spark):
from master node after SSHing into it
from another AWS EMR cluster
But NOT able to connect to it:
from my local machine (macOS Mojave)
from non-emr machines like Metabase and Redash
I have read answers of this question. I have checked that folder permissions and disk space are fine on all the nodes. My assumption is I'm facing similar problem what James Wierzba is asking in the comments. However, I do not have enough reputation to add a comment there. Also, this might be a different problem considering it is specific to AWS EMR.
Connection works fine after SSHing to master node.
# SSHed to master node
$ ssh -i ~/identityfile hadoop#ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com
# on master node
$ /usr/lib/spark/bin/beeline -u 'jdbc:hive2://localhost:10001/default'
# it connects fine and I can run commands, for e.g., 'show databases;'
# Beeline version 1.2.1-spark2-amzn-0 by Apache Hive
Connection to this node works fine from master node of another EMR cluster as well.
However, connection does not work from my local machine (macOS Mojave), Metabase and Redash.
My local machine:
# installed hive (for beeline)
$ brew install hive
# Beeline version 3.1.1 by Apache Hive
# connect directly
# I have checked that all ports are open for my IP
$ beeline -u 'jdbc:hive2://ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com:10001/default'
# ERROR: ConnectException: Operation timed out
#
# this connection timeout probably has something to do with spark accepting only localhost connections
# I have allowed all the ports in AWS security group for my IP
# connect via port forwarding
# open a port
$ ssh -i ~/identityfile -Nf -L 10001:localhost:10001 hadoop#ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com
$ beeline -u 'jdbc:hive2://localhost:10001/default'
# Failed to connect to localhost:10001
# Required field 'client_protocol' is unset!
$ beeline -u 'jdbc:hive2://localhost:10001/;transportMode=http'
# org.apache.http.ProtocolException: The server failed to respond with a valid HTTP response
I have setup Metabase and Redash in ec2.
Metabase → connect using data source Spark SQL → results in
java.sql.SQLException: org.apache.spark.SparkException: java.io.IOException: Failed to create local dir in /mnt/tmp/blockmgr*
Redash → connect using data source Hive → results in same error.
You need to update the inbound rules of the security group attached to Master node of EMR. You will need to add the public IP address of your network provider. You can find your public IP address on the following website :
What is my IP
For more details on how to update the inbound rules with your IP address refer following AWS documentation :
Authorizing Inbound Traffic for Your Linux Instances
You should also check the outbound rules of your own network in case you are working in a restricted network environment.
So make sure you have outbound access in your network and inbound access in your EMR's master node security group for all the ports you want to access.

Unable to use Guacamole Reverse VNC connection

I am using Guacamole v0.9.9 and want to connect to my Win 10 laptop which is behind a NAT of my ISP.
I figured I might have to use Reverse VNC for this. The instructions are given here:
https://guacamole.incubator.apache.org/doc/gug/configuring-guacamole.html#vnc-reverse-connections
But I am using MYSQL Auth as described here:
https://guacamole.incubator.apache.org/doc/0.9.0/gug/mysql-auth.html
The problem is that I am not able to see any options for Reverse Connection in VNC settings and there is no XML file to put the parameters in.
Also there is no instruction what to do after that. In a conventional VNC connection you would run the client in the destination and run the server in listen/reverse mode after giving destination ip. In this case there is no client running. So I am clueless what to do next.
Any help will be much appreciated.
There are a few things you'll need to do in order to setup the reverse-connect functionality:
So in a typical authorization scenario you have something like this in the user-mapping.xml with the necessary information for the reverse-connect:
<authorize username="user" password="password">
<connection name="reverse">
<protocol>vnc</protocol>
<param name="hostname">localhost</param>
<param name="port">9999</param>
<param name="reverse-connect">true</param>
<param name="listen-timeout">30000</param>
<param name="autoretry">true</param>
</connection>
</authorize>
Since you are doing this through MySQL it's the same principle:
Connections and parameters
Each connection has an entry in the guacamole_connection table, with a
one-to-many relationship to parameters, stored as name/value pairs in
the guacamole_connection_parameter table.
The guacamole_connection table is simply a pairing of a unique and
descriptive name with the protocol to be used for the connection. Adding a connection and corresponding parameters is relatively easy compared to adding a user as there is no salt to generate nor password to hash:
-- Create connection
INSERT INTO guacamole_connection (connection_name, protocol) VALUES ('reverse', 'vnc');
SET #id = LAST_INSERT_ID();
-- Add parameters
INSERT INTO guacamole_connection_parameter VALUES (#id, 'hostname', 'localhost');
INSERT INTO guacamole_connection_parameter VALUES (#id, 'port', '9999');
INSERT INTO guacamole_connection_parameter VALUES (#id, 'reverse-connect', 'true');
...
Connecting:
Open the connection within Guacamole, then connect to the port on the Guacamole Server with the VNC client (eg. :9999 as shown in the example above). If you don't open the connection within Guacamole first, guacd won't be listening on the given port.
If you cannot establish a connection after setting up the user-mapping.xml or MySQL authorization that includes the reverse-connect parameter, it's suggested to install the latest version of libvncserver, which has ENABLED_VNC_LISTEN defined. You should notice when executing Guacamole's ./configure a warning if it's not defined:
--------------------------------------------
No listening support found in libvncclient.
Support for listen-mode connections will not be built.
--------------------------------------------
To save yourself some hassle you could use a vnc repeater, it will listen for connection from the vnc servers and viewers, and connect the servers and viewers that use the same id
You can get one from here
Get build packages
For Debian use
apt-get install linux-headers-`uname -r` libx11-6 libx11-dev x-window-system-core x-window-system xspecs libxtst6 psmisc build-essential
For CentOS use:
yum install linux-headers-`uname -r` libx11-6 libx11-dev x-window-system-core x-window-system xspecs libxtst6 psmisc build-essential
Get source into /usr/local/src
cd /usr/local/src
wget http://www.wisdomsoftware.gr/download/uvncrep017-ws.tar.gz
Unzip source file
gunzip uvncrep017-ws.tar.gz
tar -xvf uvncrep017-ws.tar
Install startup script
cd uvncrep017-ws
make; make install;
Add a user for the service
useradd uvncrep
Edit /etc/uvnc/uvncrepeater.ini according to your needs.
Check the following parameters:
viewerport = 5901
maxsessions = 10
runasuser = uvncrep
logginglevel = 2
srvListAllow1 = 192.168.0.0 ;Allow network 192.168.x.x
srvListDeny0 = 127.0.0.1 ;Deny loopback
requirelistedserver=1
Start the service
/etc/init.d/uvncrepeater start
Original link: here
Discussion on a board about this: here

pg_upgrade on AWS EC2 linux - pg_hba.conf settings

I am running an Amazon EC2 CentOS 6.6 server instance with pre-installed PostgreSQL 8.4.20 server which I want to upgrade to 9.4.1 using pg_upgrade via SSH.
What I've done so far: Downloaded and installed PostgreSQL 9.4.1 with yum, configured it. Configured the postgres user to have the same password on the UNIX server and for both database instances. Both database instances are functioning correctly - old one on port 5432, new on 5433.
What I am trying to do:
su - postgres
/usr/pgsql-9.4/bin/pg_upgrade
-b /usr/bin/
-B /usr/pgsql-9.4/bin/
-d /var/lib/pgsql/data/
-D /var/lib/pgsql/9.4/data/
Here is my issue with pg_hba.conf. Using
TYPE DATABASE USER METHOD
local all all trust
or
TYPE DATABASE USER METHOD
local all all peer
I can't start the old server, getting:
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
connection to database failed: fe_sendauth: no password supplied
Failure, exiting
Using the default setting
TYPE DATABASE USER METHOD
local all all ident
is the only method that allows me to start the server, but then I get the following error:
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
*failure*
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.
connection to database failed: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.50432"?
could not connect to old postmaster started with the command:
"/usr/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data/" -o "-p 50432 -c autovacuum=off -c autovacuum_freeze_max_age=2000000000 -c listen_addresses='' -c unix_socket_permissions=0700" start
Failure, exiting
I have been reading more than 10 hours straight everything related, before I posted this, but can't seem to find the solution. Will be very grateful if you can give me any hints.

Resources