pgadmin 4 container : How to import servers and make them visible to every LDAP user? - amazon-rds

I have a pgadmin container running in ECS. The container is using a specific LDAP_BIND_USER user to bind to the LDAP server but then every user is connecting to pgadmin with their own credentials.
I have modified the entrypoint.sh to fetch the list of RDS instances and generate the servers.json file when the container is first started, this is done by this command defined in the entrypoint.sh:
/venv/bin/python3 /pgadmin4/setup.py --load-servers "${PGADMIN_SERVER_JSON_FILE}" --user ${PGADMIN_DEFAULT_EMAIL}
The problem
As the container is running in SERVER mode, I need to specify a user (PGADMIN_DEFAULT_EMAIL) to whom these servers are going to be visible when doing the import. This means that only this user will see the server list.
Is there a way to import the servers for every user that connects?

So, sadly it is not possible! You can only import servers to a particular user and as the users are coming from LDAP, they are created in the sqlite db just once the users connects for the first time.
There is a workaround which it is not nice but it does the job:
Create and import the db tables (servergroup and server) manually!
This is how the user table looks like:
CREATE TABLE user (
id INTEGER NOT NULL,
username VARCHAR(256) NOT NULL,
email VARCHAR(256),
password VARCHAR(256),
active BOOLEAN NOT NULL,
confirmed_at DATETIME,
masterpass_check VARCHAR(256),
auth_source VARCHAR(256) NOT NULL DEFAULT 'internal',
PRIMARY KEY (id),
UNIQUE (username, auth_source),
CHECK (active IN (0, 1))
);
This is how the server table looks like:
CREATE TABLE server (
id INTEGER NOT NULL,
user_id INTEGER NOT NULL,
servergroup_id INTEGER NOT NULL,
name VARCHAR(128) NOT NULL,
host VARCHAR(128),
port INTEGER,
maintenance_db VARCHAR(64) NOT NULL,
username VARCHAR(64),
password VARCHAR(64),
role VARCHAR(64),
ssl_mode VARCHAR(16) NOT NULL CHECK(ssl_mode IN
( 'allow' , 'prefer' , 'require' , 'disable' ,
'verify-ca' , 'verify-full' )
),
comment VARCHAR(1024),
discovery_id VARCHAR(128),
hostaddr TEXT(1024),
db_res TEXT,
passfile TEXT,
sslcert TEXT,
sslkey TEXT,
sslrootcert TEXT,
sslcrl TEXT,
sslcompression INTEGER DEFAULT 0,
bgcolor TEXT(10),
fgcolor TEXT(10),
service TEXT,
use_ssh_tunnel INTEGER DEFAULT 0,
tunnel_host TEXT,
tunnel_port TEXT,
tunnel_username TEXT,
tunnel_authentication INTEGER DEFAULT 0,
tunnel_identity_file TEXT, connect_timeout INTEGER DEFAULT 0, tunnel_password TEXT(64), save_password INTEGER DEFAULT 0, shared BOOLEAN,
PRIMARY KEY(id),
FOREIGN KEY(user_id) REFERENCES "user_old"(id),
FOREIGN KEY(servergroup_id) REFERENCES servergroup(id)
);
And this is how the servergroup table looks like:
CREATE TABLE servergroup (
id INTEGER NOT NULL,
user_id INTEGER NOT NULL,
name VARCHAR(128) NOT NULL,
PRIMARY KEY (id),
FOREIGN KEY(user_id) REFERENCES "user_old" (id),
UNIQUE (user_id, name)
);
Looking at the server table closer we can see that for each id there is a user_id and servergroup_id associated, so we can create these tables manually and import the values before the server is started.
The catch: There is a server line for each user, so just iterate over the lenght of users and everyone will see the servers in their console

Related

How do I insert records in a node-pg-migrate migration?

I'm trying to use node-pg-migrate to handle migrations for an ExpressJS app. I can translate most of the SQL dump into pgm.func() type calls, but I can't see any method for handling actual INSERT statements for initial data in my solution's lookup tables.
It is possible using the pgm.sql catch all:
pgm.sql(`INSERT INTO users (username, password, created, forname, surname, department, reviewer, approver, active) VALUES
('rd#example.com', 'salty', '2019-12-31 11:00:00', 'Richard', 'Dyce', 'MDM', 'No', 'No', 'Yes');`)
Note the use of backtick (`) to allow breaking the SQL statement across multiple lines.
You can use raw sql if you needed.
Create a migration file with the extension .sql and write usual requests.
This article has a great example.
My example:
-- Up Migration
CREATE TABLE users
(
id BIGSERIAL NOT NULL PRIMARY KEY,
username VARCHAR(50) NOT NULL,
email VARCHAR(50) NOT NULL,
password VARCHAR(50) NOT NULL,
class_id INTEGER NOT NULL,
created_at DATE NOT NULL,
updated_at DATE NOT NULL
);
CREATE TABLE classes
(
id INTEGER NOT NULL PRIMARY KEY,
name VARCHAR(50) NOT NULL,
health INTEGER NOT NULL,
damage INTEGER NOT NULL,
attack_type VARCHAR(50) NOT NULL,
ability VARCHAR(50) NOT NULL,
created_at DATE NOT NULL,
updated_at DATE NOT NULL
);
INSERT INTO classes (id,
name,
health,
damage,
attack_type,
ability,
created_at,
updated_at)
VALUES (0,
'Thief',
100,
25,
'Archery Shot',
'Run Away',
NOW(),
NOW());
-- Down Migration
DROP TABLE users;
DROP TABLE classes;

How can I get data from two different database tables

I am creating an endpoint (GET /feed) where
user can view all articles or gifs, showing the most recently posted articles
or gifs first. It is to return the following as JSON
id​: Integer​
createdOn: DateTime
title: String
article/url: String (use url for gif post and article for articles)
userId: ​Integer​
Here is my article and gif table,
CREATE TABLE article(
article_id SERIAL PRIMARY KEY,
title VARCHAR(300),
article text ,
user_id INTEGER REFERENCES user(id) ON DELETE CASCADE,
created_on TIMESTAMP DEFAULT Now()
);
CREATE TABLE gif(
gif_id SERIAL PRIMARY KEY,
title text NOT NULL ,
cloudinary_id VARCHAR (3000),
url VARCHAR(3000) ,
user_id INTEGER REFERENCES user(id) ON DELETE CASCADE,
created_on TIMESTAMP DEFAULT Now()
)
How do I go about it?

Internal network application data model with Cassandra

I'm working on designing an application which will enable users to send requests to connect with each other, see their sent or received requests, make notes during their interactions for later reference if connected, and remove users from their contact lists.
In a RDBMS, the schema would be:
table User with column
uid (a unique string for each user)
table Request with columns:
from - user id
to - user id Primary Key (from, to)
created - timestamp
message - string
expiry - timestamp
table Connection with columns:
from - user id
to - user id
Primary Key (from, to)
notes - String
created - timestamp
modified - timestamp
isFavourite - to is a favourite of from user, value 0 or 1
isActive - soft delete, value 0 or 1
pairedConnection - shows whether the connection between to and from was deactivated (the to user removed the from user from its contact list), value 0 or 1
The queries I anticipate to be needed are:
find the sent requests for a user
find the received requests for a user
find all the active contacts of a given user
find all the favourites of a user
find all the users who deleted the given from user from their lists
update the notes taken by a user when meeting another user he is connected with
update user as favourite
mark connection for soft deletion
I'm trying to model this in Cassandra, but feel confused about the keys to choose for max efficiency.
So far, I have the following ideas, and would welcome feedback from more experienced Cassandra users:
create table users(
uid text PRIMARY KEY
);
create table requestsByFrom(
from text,
to text,
message text,
created timestamp,
expiry timestamp,
PRIMARY KEY (from,to)
create table requestsByTo(
from text,
to text,
message text,
created timestamp,
expiry timestamp,
PRIMARY KEY (to,from)
);
create table connections(
from text,
to text,
notes text,
created timestamp,
modified timestamp,
isFavourite boolean,
isActive boolean,
pairedConnection boolean,
PRIMARY KEY (from,to)
);
create table activeConnections(
from text,
to text,
isActive boolean,
PRIMARY KEY (from,isActive)
);
create table favouriteConnections(
from text,
to text,
isFavourite boolean,
PRIMARY KEY (from, isFavourite)
);
create table pairedConnection(
from text,
to text,
pairedConnection boolean,
PRIMARY KEY ((from,to), pairedConnection)
);
Cassandra has a different paradigm to RDBMS, and this is more evident with the way that the data modeling has to be done. You need to keep in mind that denormalization is preferred, and that you'll have repeated data.
The tables definition should be based on the queries to retrieve the data, this is partially stated in the definition of the problem, for instance:
find the sent requests for a user
Taking the initial design of the table requestsByFrom, an alternative will be
CREATE TABLE IF NOT EXISTS requests_sent_by_user(
requester_email TEXT,
recipient_email TEXT,
recipient_name TEXT,
message TEXT,
created TIMESTAMP
PRIMARY KEY (requester_email, recipient_email)
) WITH default_time_to_live = 864000;
Note that from is a restricted keyword, the expiry information can be set with the definition of the default_time_to_live clause (TTL) which will remove the record after the time defined; this value is the amount of seconds after the record is inserted, and the example is 10 days (864,000 seconds).
The primary key is suggested to be the email address, but it can also be an UUID, name is not recommended as there can be multiple persons sharing the same name (like James Smith) or the same person can have multiple ways to write the name (following the example Jim Smith, J. Smith and j smith may refer to the same person).
The name recipient_name is also added as it is most likely that you'll want to display it; any other information that will be displayed/used with the query should be added.
find the received requests for a user
CREATE TABLE IF NOT EXISTS requests_received_by_user(
recipient_email TEXT,
requester_email TEXT,
requester_name TEXT,
message TEXT,
created TIMESTAMP
PRIMARY KEY (recipient_email, requester_email)
) WITH default_time_to_live = 864000;
It will be preferred to add records to requests_sent_by_user and requests_received_by_user at the same time using a batch, which will ensure consistency in the information between both tables, also the TTL (expiration of the data) will be the same.
storing contacts
In the question there are 4 tables of connections: connections, active_connections, favourite_connections, paired_connections, what will be the difference between them? are they going to have different rules/use cases? if that is the case, it makes sense to have them as different tables:
CREATE TABLE IF NOT EXISTS connections(
requester_email TEXT,
recipient_email TEXT,
recipient_name TEXT,
notes TEXT,
created TIMESTAMP,
last_update TIMESTAMP,
is_favourite BOOLEAN,
is_active BOOLEAN,
is_paired BOOLEAN,
PRIMARY KEY (requester_email, recipient_email)
);
CREATE TABLE IF NOT EXISTS active_connections(
requester_email TEXT,
recipient_email TEXT,
recipient_name TEXT,
last_update TIMESTAMP,
PRIMARY KEY (requester_email, recipient_email)
);
CREATE TABLE IF NOT EXISTS favourite_connections(
requester_email TEXT,
recipient_email TEXT,
recipient_name TEXT,
last_update TIMESTAMP,
PRIMARY KEY (requester_email, recipient_email)
);
CREATE TABLE IF NOT EXISTS paired_connections(
requester_email TEXT,
recipient_email TEXT,
recipient_name TEXT,
last_update TIMESTAMP,
PRIMARY KEY (requester_email, recipient_email)
);
Note that the boolean flag is removed, the logic is that if the record exists in active_connections, it will be assumed that it is an active connection.
When a new connection is created, it may have several records in different tables; to bundle all those inserts or updates, it is preferred to use batch
find all the active contacts of a given user
Based on the proposed tables, if the requester's email is test#email.com:
SELECT * FROM active_connections WHERE requester_email = 'test#email.com'
update user as favourite
It will be a batch updating the record in connections and adding the new record to favourite_connections:
BEGIN BATCH
UPDATE connections
SET is_favourite = true, last_update = dateof(now())
WHERE requester_email ='test#email.com'
AND recipient_email = 'john.smith#test.com';
INSERT INTO favourite_connections (
requester_email, recipient_email, recipient_name, last_update
) VALUES (
'test#email.com', 'john.smith#test.com', 'John Smith', dateof(now())
);
APPLY BATCH;
mark connection for soft deletion
The information of the connection can be kept in connections with all the flags disabled, as well as the records removed from active_connections, favourite_connections and paired_connections
BEGIN BATCH
UPDATE connections
SET is_active = false, is_favourite = false,
is_paired = false, last_update = dateof(now())
WHERE requester_email ='test#email.com'
AND recipient_email = 'john.smith#test.com';
DELETE FROM active_connections
WHERE requester_email = 'test#email.com'
AND recipient_email = 'john.smith#test.com';
DELETE FROM favourite_connections
WHERE requester_email = 'test#email.com'
AND recipient_email = 'john.smith#test.com';
DELETE FROM paired_connections
WHERE requester_email = 'test#email.com'
AND recipient_email = 'john.smith#test.com';
APPLY BATCH;

Cassandra - how to update a record with a compound key

In the process of learning Cassandra and using it on a small pilot project at work. I've got one table that is filtered by 3 fields:
CREATE TABLE webhook (
event_id text,
entity_type text,
entity_operation text,
callback_url text,
create_timestamp timestamp,
webhook_id text,
last_mod_timestamp timestamp,
app_key text,
status_flag int,
PRIMARY KEY ((event_id, entity_type, entity_operation))
);
Then I can pull records like so, which is exactly the query I need for this:
select * from webhook
where event_id = '11E7DEB1B162E780AD3894B2C0AB197A'
and entity_type = 'user'
and entity_operation = 'insert';
However, I have an update query to set the record inactive (soft delete), which would be most convenient by partition key in the same table. Of course, this isn't possible:
update webhook
set status_flag = 0
where webhook_id = '11e8765068f50730ac964b31be21d64e'
An example of why I'd want to do this, is a simple DELETE from an API endpoint:
http://myapi.com/webhooks/11e8765068f50730ac964b31be21d64e
Naturally, if I update based on the composite key, I'd potentially inactivate more records than I intend to.
Seems like my only choice, doing it the "Cassandra Way", is to use two tables; the one I already have and one to track status_flag by webhook_id, so I can update based on that id. I'd then have to select by webhook_id in the first table and disable it there as well? Otherwise, I'd have to force users to pass all the compound key values in the URL of the API's DELETE request.
Simple things you take for granted in relational data, seem to get complex very quickly in Cassandraland. Is this the case or am I making it more complicated than it really is?
You can add webhook to your primary key.
So your table defination becomes somethign like this.
CREATE TABLE webhook (
event_id text,
entity_type text,
entity_operation text,
callback_url text,
create_timestamp timestamp,
webhook_id text,
last_mod_timestamp timestamp,
app_key text,
status_flag int,
PRIMARY KEY ((event_id, entity_type, entity_operation),webhook_id)
Now lets say you insert 2 records.
INSERT INTO dev_cybs_rtd_search.webhook(event_id,entity_type,entity_operation,status_flag,webhook_id) VALUES('11E7DEB1B162E780AD3894B2C0AB197A','user','insert',1,'web_id');
INSERT INTO dev_cybs_rtd_search.webhook(event_id,entity_type,entity_operation,status_flag,webhook_id) VALUES('12313131312313','user','insert',1,'web_id_1');
And you can update like following
update webhook
set status_flag = 0
where webhook_id = 'web_id' AND event_id = '11E7DEB1B162E780AD3894B2C0AB197A' AND entity_type = 'user'
AND entity_operation = 'insert';
It will only update 1 record.
However you have to send all the things defined in your primary key.

Cassandra Schema for a Chat Application

I have gone though this article and here is the schema I have got from it. This is helpful for my application for maintaining statuses of a user, but how can I extend this to maintain one to one chat archive and relations between users, relations mean people belong to specific group for me. I am new to this and need an approach for this.
Requirements :
I want to store messages between user-user in a table.
Whenever a user want to load messages by a user. I want to retrieve them back and send it to user.
I want to retrieve all the messages from different users to the user when user has requested.
And also want to store class of users. I mean for example user1 and user2 belong to "family" user3, user4, user1 belong to friends etc... This group can be custom name given by the user.
This is what I have tried so far:
CREATE TABLE chatarchive (
chat_id uuid PRIMARY KEY,
username text,
body text
)
CREATE TABLE chatseries (
username text,
time timeuuid,
chat_id uuid,
PRIMARY KEY (username, time)
) WITH CLUSTERING ORDER BY (time ASC)
CREATE TABLE chattimeline (
to text,
username text,
time timeuuid,
chat_id uuid,
PRIMARY KEY (username, time)
) WITH CLUSTERING ORDER BY (time ASC)
Below is the schema that I currently have:
CREATE TABLE users (
username text PRIMARY KEY,
password text
)
CREATE TABLE friends (
username text,
friend text,
since timestamp,
PRIMARY KEY (username, friend)
)
CREATE TABLE followers (
username text,
follower text,
since timestamp,
PRIMARY KEY (username, follower)
)
CREATE TABLE tweets (
tweet_id uuid PRIMARY KEY,
username text,
body text
)
CREATE TABLE userline (
username text,
time timeuuid,
tweet_id uuid,
PRIMARY KEY (username, time)
) WITH CLUSTERING ORDER BY (time DESC)
CREATE TABLE timeline (
username text,
time timeuuid,
tweet_id uuid,
PRIMARY KEY (username, time)
) WITH CLUSTERING ORDER BY (time DESC)
With C* you need to store data in the way you'll use it.
So let's see how this would look like for this case:
I want to store messages between user-user in a table.
Whenever a user want to load messages by a user. I want to retrieve them back and send it to user.
CREATE TABLE chat_messages (
message_id uuid,
from_user text,
to_user text,
body text,
class text,
time timeuuid,
PRIMARY KEY ((from_user, to_user), time)
) WITH CLUSTERING ORDER BY (time ASC);
This will allow you to retrieve a timeline of messages between two users. Note that a composite primary key is used so that wide rows are created for each pair of users.
SELECT * FROM chat_messages WHERE from_user = 'mike' AND to_user = 'john' ORDER BY time DESC ;
I want to retrieve all the messages from different users to the user when user has requested.
CREATE INDEX chat_messages_to_user ON chat_messages (to_user);
This allows you to do:
SELECT * FROM chat_messages WHERE to_user = 'john';
And also want to store class of users. I mean for example user1 and user2 belong to "family" user3, user4, user1 belong to friends etc... This group can be custom name given by the user.
CREATE INDEX chat_messages_class ON chat_messages (class);
This will allow you to do:
SELECT * FROM chat_messages WHERE class = 'family';
Note that in this kind of database, DENORMALIZED DATA IS A GOOD PRACTICE. This means that using the name of the class again and again is not a bad practice.
Also note that I haven't used a 'chat_id' nor a 'chats' table. We could easily add this but I feel that your use case didn't require it as it has been put forward. In general, you cannot do joins in C*. So, using a chat id would imply two queries.
EDIT: Secondary indexes are inefficient. A materialised view will be a better implementation with C* 3.0
There is a chat application created by Alan Chandler on github that has the features you request:
MBchat
It uses a 2-phase authentication. First the user is validated in the forums and then, the user is validated on the chat database.
Here's the first validation part of the schema (schema located in inc/user.sql):
BEGIN;
CREATE TABLE users (
uid integer primary key autoincrement NOT NULL,
time bigint DEFAULT (strftime('%s','now')) NOT NULL,
name character varying NOT NULL,
role text NOT NULL DEFAULT 'R', -- A (CEO), L (DIRECTOR), G (DEPT HEAD), H (SPONSOR) R(REGULAR)
cap integer DEFAULT 0 NOT NULL, -- 1 = blind, 2 = committee secretary, 4 = admin, 8 = mod, 16 = speaker 32 = can't whisper( OR of capabilities).
password character varying NOT NULL, -- raw password
rooms character varying, -- a ":" separated list of rooms nos which define which rooms the user can go in
isguest boolean DEFAULT 0 NOT NULL
);
CREATE INDEX userindex ON users(name);
-- Below here you can add the specific users for your set up in the form of INSERT Statements
-- This list is test users to cover the complete range of functions. Note names are converted to lowercase, so only put lowercase names in here
INSERT INTO users(uid,name,role,cap,password,rooms,isguest) VALUES
(1,'alice','A',4,'password','7',0), -- CEO class user alice
(2,'bob','L',3,'password','8',0), -- DIRECTOR class user bob
(3,'carol','G',2,'password','7:8:9',0), -- DEPT HEAD class user carol
And here's the second validation part of the schema (schema located in data/chat.sql):
CREATE TABLE users (
uid integer primary key NOT NULL,
time bigint DEFAULT (strftime('%s','now')) NOT NULL,
name character varying NOT NULL,
role char(1) NOT NULL default 'R',
rid integer NOT NULL default 0,
mod char(1) NOT NULL default 'N',
question character varying,
private integer NOT NULL default 0,
cap integer NOT NULL default 0,
rooms character_varying
);
The following is the schema of the chat rooms you can see the user classes and the examples of it:
CREATE TABLE rooms (
rid integer primary key NOT NULL,
name varchar(30) NOT NULL,
type integer NOT NULL -- 0 = Open, 1 = meeting, 2 = guests can't speak, 3 moderated, 4 members(adult) only, 5 guests(child) only, 6 creaky door
) ;
INSERT INTO rooms (rid, name, type) VALUES
(1, 'The Forum', 0),
(2, 'Operations Gallery', 2), -- Guests Can't Speak
(3, 'Dungeon Club', 6), -- creaky door
(4, 'Auditorium', 3), -- Moderated Room
(5, 'Blue Room', 4), -- Members Only (in Melinda's Backups this is Adults)
(6, 'Green Room', 5), -- Guest Only (in Melinda's Backups this is Juveniles AKA Baby Backups)
(7, 'The Board Room', 1), -- Various meeting rooms - need to be on users room list
The users have another table to indicate the participation of the conversation:
CREATE table wid_sequence ( value integer);
INSERT INTO wid_sequence (value) VALUES (1);
CREATE TABLE participant (
uid integer NOT NULL REFERENCES users (uid) ON DELETE CASCADE ON UPDATE CASCADE,
wid integer NOT NULL,
primary key (uid,wid)
);
And the archives are recorded as follows:
CREATE TABLE chat_log (
lid integer primary key,
time bigint DEFAULT (strftime('%s','now')) NOT NULL,
uid integer NOT NULL REFERENCES user (uid) ON DELETE CASCADE ON UPDATE CASCADE,
name character varying NOT NULL,
role char(1) NOT NULL,
rid integer NOT NULL,
type char(2) NOT NULL,
text character varying
);
Edit: However this type of data modeling is not very suitable for Cassandra. Because, in Cassandra your data does not fit on one machine so joins are not available. So, in Cassandra denormalizing data is the practical choice. Check below for the denormalized version of chat_log table:
CREATE TABLE chat_log (
lid uuid,
time timestamp,
sender text NOT NULL,
receiver text NOT NULL,
room text NOT NULL,
sender_role varchar NOT NULL,
receiver_role varchar NOT NULL,
rid decimal NOT NULL,
status varchar NOT NULL,
message text,
PRIMARY KEY (sender, receiver, room)
-- PRIMARY KEY (sender, receiver) if you don't want the messages to be separated by the rooms
) WITH CLUSTERING ORDER BY (time ASC);
Now in order to retrieve data you'd use the following queries:
Whenever a user want to load messages by a user. I want to retrieve them back and send it to user.
SELECT * FROM chat_log WHERE sender = 'bob' ORDER BY time ASC
I want to retrieve all the messages from different users to the user when user has requested.
SELECT * FROM chat_log WHERE receiver = 'alice' ORDER BY time ASC
I want to store and retrieve class of users.
SELECT * FROM chat_log WHERE sender_role = 'A' ORDER BY time ASC -- messages sent by CEOs
SELECT * FROM chat_log WHERE receiver_role = 'A' ORDER BY time ASC -- messages received by CEOs
After modeling the data. You'd need to create indexes for quick and efficient querying as follows:
For retrieving all messages from different users to the user efficiently
CREATE INDEX chat_log_uid ON chat_log (sender);
CREATE INDEX chat_log_uid ON chat_log (receiver);
For retrieving all messages from user classes efficiently
CREATE INDEX chat_log_class ON chat_log (sender_role);
CREATE INDEX chat_log_class ON chat_log (receiver_role);
I believe these examples will give you the approach you need.
If you'd like to learn more about Cassandra data modeling you can check down below:
Cassandra Data Modeling Best Practices, Part 1
Cassandra Data Modeling Best Practices, Part 2
Cassandra Data Modeling Best Practices Slide
Data Modeling Example

Resources