I'm developing a shell script that loops through a series of Postgres database table names and dumps the table data. For example:
# dump data
psql -h $SRC_IP_ADDRESS -p 5432 -U postgres -c "BEGIN;" AWARE
do
:
pg_dump -U postgres -h $IP_ADDRESS -p 5432 -t $i -a --inserts MYDB >> \
out.sql
done
psql -h $IP_ADDRESS -p 5432 -U postgres -c "COMMIT;" MYDB
I'm worried about concurrent access to the database, however. Since there is no database lock for Postgres, I tried to wrap a BEGIN and COMMIT around the loop (using psql, as shown above). This resulted in an error message from the psql command, saying that:
WARNING: there is no transaction in progress
Is there any way to achieve this? If not, what are the alternatives?
Thanks!
Your script has two main problems. The first problem is practical: a transaction is part of a specific session, so your first psql command, which just starts a transaction and then exits, has no real effect: the transaction ends when the command completes, and later commands do not share it. The second problem is conceptual: changes made in transaction X aren't seen by transaction Y until transaction X is committed, but as soon as transaction X is committed, they're immediately seen by transaction Y, even if transaction Y is still in-progress. This means that, even if your script did successfully wrap the entire dump in a single transaction, this wouldn't make any difference, because your dump could still see inconsistent results from one query to the next. (That is: it's meaningless to wrap a series of SELECTs in a transaction. A transaction is only meaningful if it contains one or more DML statements, UPDATEs or INSERTs or DELETEs.)
However, since you don't really need your shell-script to loop over your list of tables; rather, you can just give pg_dump all the table-names at once, by passing multiple -t flags:
pg_dump -U postgres -h $IP_ADDRESS -p 5432 \
-t table1 -t table2 -t table3 -a --inserts MYDB >> out.sql
and according to the documentation, pg_dump "makes consistent backups even if the database is being used concurrently", so you wouldn't need to worry about setting up a transaction even if that did help.
(By the way, the -t flag also supports a glob notation; for example, -t table* would match all tables whose names begin with table.)
Related
I am a beginner in bash scripting and I have created a bash script to add users and remove users on Linux.
But since I am facing some issues with the script not really major issues but would be helpful if anyone could point me how to improve the script and the worst practice I am doing the script would be helpful
however the problem I have noticed is that the script takes -a to add a user -d to remove user and -h to get help the -a flag as 2 optional arguments -p for password and -s for shell so the command would be
./useradd.sh -a user -p password -s shell
this works as expected and the user is added to the system but the problem I am facing is that if I do not enter -a flag and specify the -s and -p flag the script is just exited I want to show a clear idea to the user why it exited and there is so many such errors I am assuming but I have not tested it out so much any help would be appreciated, so here is my script
https://github.com/abhijith7025/bash_scripts/blob/master/useradd.sh
... I have created a bash script to add users and remove users on Linux.
You might want to reconsider this, given that Linux has separate commands for these two operations.
There's a lot more to creating a user than there is is getting rid of one, so you may be asking for trouble trying to conjoin the two.
Anyway:
Your case statement has no "other" branch ("* )").
This may be why you're getting no errors when "misusing" your script.
case $opt in
a )
;;
d )
;;
h )
;;
* )
usage
exit 1
;;
esac
Other things to look out for:
useradd is, perhaps, a poor name for a script that can delete users.
You allow a shell to be specified as an argument, but you don't check to see if that exists.
It's conventional for Linux commands to operate on a "thing" or list of "things" and for that operation to be qualified by options that [usually] precede the thing(s). Thus, your script might be better invoked like this:
./manage_user -a -p password -s shell_exe user1
| | | |
| | | User name
| | Shell for User
| Password for User
Add a User
./manage_user -d user2
| |
| User name
Delete a User
I need to request certain commands via su including password in one line.
I found a solution and it is working in a standard environment (Ubuntu) (more about solution here):
{ sleep 1; echo password; } | script -qc 'su -l user -c id' /dev/null | tail -n +2
But I am faced with the problem that this solution is not suitable in a Docker container environment
Script terminates the command without waiting for echo and as a result i get:
su: Authentication failure
Any help is much appreciated.
Passing the password for su via stdin is problematic for various reasons: the biggest one is probably that your password will end up in the history.
You could instead:
Call the entire script as the specific user and thus enter the password manually
Use sudo with the appropriate NOPASSWD sudoers configuration
In your case you are using docker, so you could just set the USER in your Dockerfile
Is there a way for pgbouncer to force to re-read /etc/hosts file without restart? I have added a new server to /etc/hosts and I want pgbouncer to connect to the new server with a minimum of hassle.
I know issuing RELOAD; command will force to re-read configuration file, but it seems that this does not apply to /etc/hosts. Also running command SHOW DNS_HOSTS (after changes in configuration and /etc/hosts) new hostname value would appear, but addrs value is left blank.
pgbouncer version: 1.7.2 running on Ubuntu 14.04
Please don't read this as instruction for usage. This is more academical interest - what you need to do to make pgbouncer 1.7 to reread /etc/hosts without restart:
first demo:
pgbouncer=# show dns_hosts;
hostname | ttl | addrs
----------+-----+-------------
one | 6 | 127.0.0.3:0
(1 row)
pgbouncer=# \! sudo sed -i 's/127.0.0.3/127.0.0.2/' /etc/hosts
pgbouncer=# pause test;
PAUSE
pgbouncer=# kill test;
KILL
pgbouncer=# resume test;
RESUME
pgbouncer=# \! psql -p 6432 -h 127.0.0.1 -U vao -d test -c "\! tail -1 /etc/hosts"
Password for user vao:
127.0.0.2 one
pgbouncer=# show dns_hosts;
hostname | ttl | addrs
----------+-----+-------------
one | 7 | 127.0.0.2:0
(1 row)
pgbouncer=# \! sudo sed -i 's/127.0.0.2/127.0.0.12/' /etc/hosts
pgbouncer=# pause test;
PAUSE
pgbouncer=# kill test;
KILL
pgbouncer=# resume test;
RESUME
pgbouncer=# \! psql -p 6432 -h 127.0.0.1 -U vao -d test -c "\! tail -1 /etc/hosts"
Password for user vao:
127.0.0.12 one
pgbouncer=# show dns_hosts;
hostname | ttl | addrs
----------+-----+--------------
one | 10 | 127.0.0.12:0
(1 row)
Now why:
RELOAD rereads config, thus won't help here. dns_max_ttl controls roundroubin between several reselves returned by dns, thus won't play here. Recalling
Hostnames are resolved on connect time
I make an assumption that in order to reinitiate connection I need to drop existing connections (so connection would not be taken from the pool) - two way to do it - either restart pgbouncer or KILL db - isolating the impact to only one db from pgbouncer.ini [databases] section. So I added
test = host=one port=5432 dbname=t
to it and
127.0.0.3 one
to /etc/hosts. The rest is in demo.
I would interpret this answer as a cheat - I don't restart pgbouncer, but yet all existing connections to wanted db need to be dropped. (of course we don't affect other databases clients connected, but still). So the answer would be - yes, you can do it without restart, yet all connections to that db will be dropped, thus you can't do it without dropping existing connections to the host that has changed. just PAUSE + RESUME combination won't help here as
Hostnames are resolved on connect time
I would like for my psql command to fail if zero rows are found:
psql -U postgres -d db -c "select * from user where id=1 and name='Joe';"
I want to be able to check the return value. Return 0 from the process(!) if at least one row exists and return non-zero from the psql process if no such row exists. How can I set a return code if no rows are found?
I don't think psql can do it by itself, but if you just want to see if there are any rows or not with the exit status you could combine it like
psql -U postgres -d db -t -c "select * from user where id=1 and name='Joe'" | egrep .
That will cause egrep to exit with non-zero if it cannot match anything. The -t will make it not print the column headers and summary information, so you may need to tweak this command line if you need that stuff.
I am trying to execute the CQL commands from shell script.
I am able to connect to the cqlsh (CQL version i'm using is 1.1.18) but unable to send the queries to cql.
Any ideas or suggestion how to proceed on this?
Do I need to connect to Cassandra and execute few commands (select/update ) with shell script ??
cqlsh -e "select * from ks.table limit 1;" > ~/output
I'm not sure about Cassandra 1.1.18, but you should be able to accomplish this with the -f flag of cqlsh. Let's say have a file of CQL commands called "commands.cql". I can invoke those commands against my local Cassandra instance like this:
$ cqlsh -f commands.cql -u myusername -p mypassword localhost
If I wanted to invoke that from within a Bash script, the script's code would look something like this:
#!/bin/bash
cqlsh -f commands.cql -u myusername -p mypassword localhost
Save that as an executable file, and run it like any other.
Need to connect to cassandra and execute few commands (select / update ) with shell script
You can execute your commands with shell script in next way:
echo "some QUERY; exit" | cqlsh CASSANDRA_HOST -u 'USER' -p 'PASS'
The "exit" command in the last suggestion is a bit hacky.
I would propose using xargs with cqlsh -e.
echo "some QUERY;" | xargs cqlsh CASSANDRA_HOST -u 'USER' -p 'PASS' -e
I recently had to use this approach when working with docker, because clqsh -f was not an option (too complex to configure access to the file needed).
echo "some QUERY;" | xargs cqlsh CASSANDRA_HOST -u 'USER' -p 'PASS' -e
But what if you Cassandra instance is on a different server to where the shell script is being executed? (Specifically in StreamSets - wouldn't the above require Cassandra installed on the same server such that it has access to the cqlsh lib?)