I'm planning on running a .sh script that will run periodically through cron on linux. I'm running postgres 8.4 on centos.
My script will have something like this in it:
psql -U username -d db_name -c "COPY orders TO stdout DELIMITER ',' CSV HEADER;" > orders.csv
I know there are other ways to dump tables into csv files but this is the only one I could use without admin rights.
My problem is naming the files. I want to specifically name the file something along the lines of:
yyyymmdd-hhmm-orders.csv
I'm not the best scripting guru out there (as you can tell) so how can I get the dumps to dynamically do this?
Thanks
`date '+%Y%m%d-%H%M'`-orders.csv
I personally also add the seconds %S to the file name
man date
Will show the other formatting options
Use below code and its worked fine
Assigned date format with one variable and used the same
Code:
I=`date +%Y%m%d-%H%M%S -d`
psql -U username -d db_name -c "COPY orders TO stdout DELIMITER ',' CSV HEADER;" > $i-orders.csv
Related
Currently, the search path in .sh file is set as:
export PSQL_PREAMBLE='SET search_path TO public,mimiciii'
I am running:
{ echo "${PSQL_PREAMBLE}; DROP TABLE IF EXISTS ventilation_durations; CREATE TABLE ventilation_durations AS "; cat durations/ventilation_durations.sql; } | sed -r -e "${REGEX_DATETIME_DIFF}" | sed -r -e "${REGEX_SCHEMA}" | psql ${CONNSTR}
Question: After running the code above in the .sh file, it seems like the code I am running above did not catch the search path set in PSQL_PREAMBLE. It just does not catch the functions I created in the mimiciii schema
(The full script is here: https://github.com/MIT-LCP/mimic-code/blob/main/mimic-iii/concepts/postgres_make_concepts.sh)
That should work just fine, but it seems unnecessarily complicated.
Just run
export PGOPTIONS=-csearch_path=public,mimiciii
at the beginning of your script, and search_path will automatically be set like that for all future psql invocations.
See the documentation of PGOPTIONS, the options connection parameter and the -c option of postgres.
I've been using the psql Postgres terminal to import CSV files into tables using the following
COPY tbname FROM
'/tmp/the_file.csv'
delimiter '|' csv;
which works fine except that I have to be logged into the psql terminal to run it.
I would like to know if anyone knows of a way to do a command similar to this from the Linux shell command line similar to how Postgres allows a shell command like bellow
/opt/postgresql/bin/pg_dump dbname > /tmp/dbname.sql
This allows the dumping of a database from the Linux shell without being logged into psql terminal.
The solution in the accepted answer will only work on the server and when the user executing the query will have permissions to read the file as explained in this SO answer.
Otherwise, a more flexible approach is to replace the SQL's COPY command with the psql's "meta-command" called \copy which which takes all the same options as the "real" COPY, but is run inside the client (with no need for ; at the end):
psql -c "\copy tbname FROM '/tmp/the_file.csv' delimiter '|' csv"
As per docs, the \copy command:
Performs a frontend (client) copy. This is an operation that runs an SQL COPY command, but instead of the server reading or writing the specified file, psql reads or writes the file and routes the data between the server and the local file system. This means that file accessibility and privileges are those of the local user, not the server, and no SQL superuser privileges are required.
In addition, if the the_file.csv contains the header in the first line, it can be recognized by adding header at the end of the above command:
psql -c "\copy tbname FROM '/tmp/the_file.csv' delimiter '|' csv header"
As stated in The PostgreSQL Documentation (II. PostgreSQL Client Applications - psql) you can pass a command to psql (PostgreSQL interactive terminal) with the switch -c. Your options are:
1, Client-side CSV: \copy meta-command
perform the SQL COPY command but the file is read on the client and the content routed to the server.
psql -c "\copy tbname FROM '/tmp/the_file.csv' delimiter '|' csv"
(client-side option originally mentioned in this answer)
2. Server-side CSV: SQL COPY command
reads the file on the server (current user needs to have the necessary permissions):
psql -c "COPY tbname FROM '/tmp/the_file.csv' delimiter '|' csv;"
the DB roles needed for reading the file on the server:
COPY naming a file or command is only allowed to database superusers
or users who are granted one of the default roles
pg_read_server_files, pg_write_server_files, or
pg_execute_server_program
also the PostgreSQL server process needs to have access to the file.
To complete the previous answer, I would suggest:
psql -d your_dbname --user=db_username -c "COPY tbname FROM '/tmp/the_file.csv' delimiter '|' csv;"
The most flexible way is to use a shell HERE document, which allows you to use shell variables inside your query, even inside (double or single) quotes:
#!/bin/sh
THE_USER=moi
THE_DB=stuff
THE_TABLE=personnel
PSQL=/opt/postgresql/bin/psql
THE_DIR=/tmp
THE_FILE=the_file.csv
${PSQL} -U ${THE_USER} ${THE_DB} <<OMG
COPY ${THE_TABLE} FROM '${THE_DIR}/${THE_FILE}' delimiter '|' csv;
OMG
The problem that I am having is that I want to run the following command (and I can't):
cqlsh < cql_directory/cql_create_stuff.cql
Because I have not logged in to cqlsh.
So I logged in:
cqlsh -u 'my_username' -p 'my_super_secret_password'
and now I tried doing the command in cqlsh shell but It just responds with a syntax error.
Basically, how do I login into cqlsh and run an external CQL script in my file system?
Use the SOURCE
http://www.datastax.com/documentation/cql/3.1/cql/cql_reference/source_r.html
You can use -f option as well to execute commands from file
http://www.datastax.com/documentation/cql/3.1/cql/cql_reference/cqlsh.html
Assuming that the path of the file with the CQL commands is /mydir/myfile.cql, there are two ways:
If you are not logged in to cqlsh:
cqlsh -u 'my_username' -p 'my_password' -f /mydir/myfile.cql
If you are logged in to cqlsh:
SOURCE '/mydir/myfile.cql'
Notice the single quotation marks. The shorthand notation for $HOME (for example, '~/mydir/myfile.cql') is also supported.
Both ways also work with relative paths (to the current directory).
Assuming your filename is "tables.cql" and it is placed as: /files/tables.cql;
A - Locally
cqlsh -f /files/tables.cql
B - Connecting To A Docker Container Running Cassandra
Assuming the name of the Docker container that which running Cassandra is "cas" (keep in mind that you can also use the hash id of the docker container if there is no name assigned to it);
docker exec -it cas cqlsh -f /files/tables.cql
As stated on other answers, -u and -p options can be added in order to use the username/password combinations.
This is for Window system
suppose you cassandra dir is
C:\Program Files\DataStax-DDC\apache-cassandra\bin
Suppose directory where your .cql file OR cql query file is
D:\ril\s\developement\new one\excel after parse\Women catalogue template.cql
Now follow below steps for importing cql file
Go on command prompt (cmd)
Go on the directory where cql file is there (cd "..\ril\sizeguide\developement\new one\excel after parse")
Run below command
"c:\Program Files\DataStax-DDC\apache-cassandra\bin\cqlsh.bat" <"Women catalogue template.cql"
And its Done.
Important Note:
Please make sure column value should not have single quote ' character like ('If you don't find a exact match, go for the next large size') other wise it will fail.
If you want single quote to be inserted, please use it two times like below and Cassandra will treat it as one time
('If you don''t find a exact match, go for the next large size')
All text column should be enclosed by single quote '' like 'Sale category'. For empty value, please use two single quote ''.
how can I be able to convert my cygwin bash into a c Shell.
I have tried by changing the .bat file in installation directory like:
#echo off
#echo ----Welcome NAME----
#echo 'have a nice day'
cygdrive\
chdir \
set HOME=\cygwin\home\
tcsh -i
please help.
Is it right process?
though it is working for me a bit...
any ideas...?
To change your Cygwin shell, you can alter the /etc/passwd file. Each line is a delimited list of user accounts, where the last entry is the shell for that user. Simply change the line that reads, for example:
abhisek:[some other stuff]:/usr/bin/bash
to:
abhisek:[some other stuff]:/usr/bin/tcsh
The current version of Cygwin doesn't have an /etc/passwd file, and the system I'm working on has Windows account information in a domain database out of my control. Consequently, chsh is no longer supported.
I also found that bash is not hard-coded into the startxwin script, nor is it hard-coded in any .bat file. Turns out you don't need to fiddle with .bat files at all.
Searching for how to change my shell, I found some advice about mkpasswd
I added it to the mix.
The man-page said:
SYNOPSIS
mkpasswd [OPTIONS]...
OPTIONS
Don't use this command to generate a local /etc/passwd file, unless you
really need one. See the Cygwin User's Guide for more information.
-c,--current
Print current user.
DESCRIPTION
The mkpasswd program can be used to create a /etc/passwd
file. Cygwin doesn't need this file, because it reads user
information from the Windows account databases, but you can add
an /etc/passwd file, for instance if your machine is often dis‐
connected from its domain controller.
Note that this information is static, in contrast to the informa‐
tion automatically gathered by Cygwin from the Windows account
databases. If you change the user information on your system,
you'll need to regenerate the passwd file for it to have the new
information.
For very simple needs, an entry for the current user can be cre‐
ated by using the option -c.
(I don't know why the spacing is so "off"...)
I then used the following command:
mkpasswd -c | sed -e 'sX/bashX/tcshX' | tee -a /etc/passwd
and voila! the next time I opened a Cygwin Terminal, it went straight to tcsh
And that's the way (Uh-huh, uh-huh!) I like it.
I have a crontab set up that errors out every time I attempt to do it. It works fine in the shell. It's the format I'm using when I attempt to automatically insert the date into the filename of the database backup. Does anyone know the syntax I need to use to get cron to let me insert the date into the filename?
mysqldump -hServer -uUser -pPassword Table | gzip >
/home/directory/backups/table.$(date +"%Y-%m-%d").gz
Thanks in advance!
What about something like this for the "command" part of the crontab :
mysqldump --host=HOST --user=USER --password=PASSWORD DATABASE TABLE | gzip > /tmp/table.`date +"\%Y-\%m-\%d"`.gz
What has changed from OP is the escaping of the date format :
date +"\%Y-\%m-\%d"
(And I used backticks -- but that should do much of a difference)
(Other solution would be to put your original command in a shell-script, and execute this one from the crontab, instead of the command -- would probably be easier to read/write ^^)
The most typical reason for "works in shell but not in cron" is that commands you try to execute are not in PATH.. Reason is that shell invoked from cron aint loading same files as your login shell.
Fix: add absolute path to each command you try to execute.
Second thing i notice in your command. Syntax for running your date command looks like its not very portable. Change that to be in backticks, or run put your whole command to shellscript (also, you can use it to set your path too) and execute that script from cron..
EDIT:
During the writing my original reply my keyboard layout didnt have backticks so check what Pascal wrote.
And example of what you could do with a shellscript:
Copy following to /usr/local/bin/dumptable.sh
#!/bin/sh
/usr/bin/mysqldump --host=HOST --user=USER --password=PASSWORD DATABASE TABLE | /bin/gzip > /tmp/table.`/bin/date +"\%Y-\%m-\%d"`.gz
and then put the the /usr/local/bin/dumptable.sh into cron..