Hi my database has started to go over 2GB in backed up size, so I'm looking at options for splitting the file and then reassembling it to restore the database.
I've got a series of files from doing the following backup shell file:
DATE_STRING=`date +%u%a`
BACKUP_DIR=/home/myhome/backups
/usr/local/mysql_versions/mysql-5.0.27/bin/mysqldump --defaults-file=/usr/local/mysql_versions/mysql-5.0.27/my.cnf
--user=myuser
--password=mypw
--add-drop-table
--single-transaction
mydb |
split -b 100000000 - rank-$DATE_STRING.sql-;
this prodes a sequence of files like:
mydb-3Wed.sql-aa
mydb-3Wed.sql-ab
mydb-3Wed.sql-ac
...
my question is what is the corresponding sequence of commands that I need to use for linux to do the restore?
Previously I was using this command:
/usr/local/mysql_versions/mysql-5.0.27/bin/mysql
--defaults-file=/usr/local/mysql_versions/mysql-5.0.27/my.cnf
--user=myuser
--password=mypw
-D mydb < the_old_big_dbdump.sql
Any suggestions even if they don't involve split / cat would be greatly appreciated
I don't see why you can't just do:
cat mydb-3Wed.sql-* | /usr/local/mysql_versions/mysql-5.0.27/bin/mysql --defaults-file=/usr/local/mysql_versions/mysql-5.0.27/my.cnf --user=myuser --password=mypw -D mydb
The * globbing should provide the files in the sorted order, check with ls mydb-3Wed.sql-* that they actually are though.
Related
I teach students how to fix wordpress sites and I would like to write a script that looks at what information they have in their wp-config.php and add a single letter or number to the database name.
For example the line is such
define('DB_NAME', 'cpanelUser_NameofDB');
I would like to add a number or a line to the end of NameofDB
I can use this to isolate the cpanelUser_NameofDB
grep -i 'DB_NAME' wp-config.php | cut -d"'" -f4
but I'm not sure how to add information, nor if this is the correct script I should run to get there. I would also like it to not matter what the name of the database is since it will be ran on multiple sites. I'm sure I could use regex but I'm not too versed in that and would not know where to start. Help please!
You can use WP CLI on the server:
Go to the website root:
cd /var/www/mysite.com/htdocs
...and list all the wp-config values:
wp config list
...or set the value you need to change:
sudo wp config set DB_NAME put_my_custom_db_name_here --allow-root
See more WP CONFIG features here:
https://developer.wordpress.org/cli/commands/config/
You can use sed for this purpose to fix this line:
#!/bin/bash
id=34
sed -i "s/^.*DB_NAME.*$/define('DB_NAME', 'cpanelUser_NameofDB${id}');/" wp-config.php
flag -i means that we do change in file directly.
Alternatively you can make template file and generate new file while sed works on stdin and stdout
#!/bin/bash
id=34
cat wp-config.php-template| sed "s/^.*DB_NAME.*$/define('DB_NAME', 'cpanelUser_NameofDB${id}');/" > wp-config.php.$id
Case: I wanna insert some data from a file to specific position of the string.
Example:
cat users.log | mysql -h localhost -u mysql -e 'SELECT id FROM users WHERE ${USERS_IDS}'
I want to replace ${USERS_IDS} string to the data from the file
I'm sure this case is very popular, but I did not find a suitable solution
To insert the contents of a file in a specific position of a string you use the facilities of your shell.
For example, Bash has the $(< filename) construction:
mysql -h localhost -u mysql -e 'SELECT id FROM users WHERE '"$(< users.log)"
If the data to be inserted needs to be editted a little, you have the $(command) construction:
mysql -h localhost -u mysql -e 'SELECT id FROM users WHERE '"$(< users.log sed -e 's/.../.../')"
Whether the file substitution or the command substitution are to be enclosed in doublequotes or not depends on the specific use case.
And, if the intended use is really to feed values in a MySQL command, beware of Bobby Tables.
I need to get the DB name and use it as a variable from part of a file name. Currently I have it pulling in the name of the file from a ls command. This is what it returns:
Test-20150311-1200.sql
I need the first part of the filename up to the "-" so for this example it would be Test. How can I get just part of the file name for my variable? Below is the full script I am working on. Any help would be great. Thanks.
#!/bin/bash
DB_USER=x
DB_PASS=x
DB=
for DB_GZFILE in $( ls *.sql.gz ); do
gunzip $DB_GZFILE
echo item: $DB_FILE unzipped
done
for DB_FILE in $( ls *.sql ); do
#Use this statement to insert dump into a new server
mysql $DB <$DB_FILE
done
#Use this command to insert into a server already in use
# mysql -u$DB_USER -p$DB_PASS <$DB_FILE
echo $DB_FILE inserted into database
done
#Remove sql files used to insert into this server
# rm $DB_FILE
# echo $DB_FILE removed
done
echo restarting the mysql process .....
# /etc/init.d/mysql restart
echo mysql restarted
done
Don't parse ls
for DB_GZFILE in *.sql.gz; do
for DB_FILE in *.sql; do
To get just the first part, use parameter substitution to remove the first - and all following characters:
first_part=${DB_FILE%%-*}
You should quote all your "$vars", especially any whose value you get from the user or from the filesystem: you never know when you'll get a filename with a space in it. Example:
gunzip "$DB_GZFILE"
I'd recommend you do not use ALL_CAPS_VARNAMES: one day you'll accidentally use PATH=... and then wonder why your script is broken. Leave ALL_CAPS for system environment variables and shell special vars.
Checking if anybody else had the similar issue.
Code in the shell script:
## Convert file into Unix format first.
## THIS is IMPORTANT.
#####################
dos2unix "${file}" "${file}";
#####################
## Actual DB Change
db_change_run_op="$(ssh -qn ${db_ssh_user}#${dbserver} "sqlplus $dbuser/${pswd}#${dbname} <<ENDSQL
#${file}
ENDSQL
")";
Summary:
1. From a shell script (on a SunOS source server) I'm running a sqlplus session via ssh on a target machine to run a .sql script.
2. Output of this target ssh session (running sqlplus) is getting stored in a variable within the shell script. Variable name: db_change_run_op (as shown above in the code snapshot).
3. Most of the .sql scripts (that the variable "${file}" stores) that I'm running, shell script runs it fine and returns me the output of the .sql file (ran on target server via ssh from source server) provided, if the .sql file contains something which doesn't take much time to complete -or generates reasonable amount of output log/lines.
for ex: Let's assume if .sql I want to run does the following, then it runs fine.
select * from database123;
udpate table....
alter table..
insert ....
...some procedure .... which doesn't take much time to create....
...some more sql commands which complete..within few minutes to an hour....
4. Now, the issue I'm facing is:
Let's assume I have a .sql file where a single select command from a table have couple of hundred thousands - upto 1-5millions of lines i.e.
select * from database321;
assume the above generates the above bullet 4 condition.
In this case, I'm getting the following error message thrown by the shell script (running on the source server).
Error:
*./db_change_load.sh: xrealloc: subst.c:4072: cannot reallocate 1073741824 bytes (0 bytes allocated)*
My questions:
1. Did the .sql script complete - I assume yes. But, how can I get the output LOG file of the .sql file generated on the target server directly. If this can be done, then I won't need the variable to hold the output of whole ssh session sqlplus command and then create a log file on source server by doing [ echo "${db_change_run_op}" > sql.${file}.log ] way.
I assume the error is coming as the output or no. of lines generated by the ssh session i.e. by the sqlplus is so big that it can't fit Unix/Linux BASH variable's limit and thus, xrealloc error.
Please advise if on the above 2 questions if you have any experience or how can i solve this.
I assume, I'll try using " | tee /path/on.target.ssh.server/sql.${file}.log" soon after << ENDSQL or final close of ENDSQL (here doc keyword), wondering if that would work or not..
OK. got it working. No more store stuff in a var and then echo $var to a file.
Luckily, I had a same mount point on both source and target server i.e. if I go to /scm on source and on target, the mount (df -kvh .) shows same output for Share/NAS mount value.
Filesystem size used avail capacity Mounted on
ServerNAS02:/vol/vol1/scm 700G 560G 140G 81% /scm
Now, instead of using the variable to store the whole output of ssh session calling sqlplus session, all I did is was to create a file on the remote server using the following code.
## Actual DB Change
#db_change_run_op="$(ssh -qn ${pdt_usshu_dbs}#${dbs} "sqlplus $dbu/${pswd}#$dbn <<ENDSQL | tee "${sql_run_output_file}".ssh.log
#set echo off
#set echo on
#set timing on
#set time on
#set serveroutput on size unlimited
##${file}
#ENDSQL
#")";
ssh -qn ${pdt_usshu_dbs}#${dbs} "sqlplus $dbu/${pswd}#$dbn <<ENDSQL | tee "${sql_run_output_file}".ssh.log
set echo off
set echo on
set timing on
set time on
set serveroutput on size 1000000
#${file}
ENDSQL
"
seems like unlimited doesn't work in 11g so I had to use the 1000000 value (these small sql cmds help to show command with its output, show clock time for each output line etc).
But basically, in the above code, I'm calling the ssh command directly without using a variable="$(.....)" way.. and after the <
Even if I wouldn't have the same mount, I could have tee'd the output to a file on the remote server path (which is not available from source server) but atleast I can see upto what level the .sql command completed or generated output as now output is going directly to a file on remote server and Unix/Linux doesn't care much about the file size until there's no space left.
In PostgreSQL and bash(linux), are there ways to directly import a file from the filesystem like
[execute.sh]
pgsql .... -f insert.txt
[insert.txt]
insert into table(id,file) values(1,import('/path/to/file'))
There seems to be no import function, bytea_import as well, lo_import would save int and I don't know how to get the file back (these files are in small sizes so using lo_import seems not appropriate)
And can how do I move the insert.txt statement to PostgreSQL?
I'm not sure what you're after, but if you have script with SQL-statements, for example the insert statements that you mention, you can run psql and then run the script from within psql. For example:
postgres#server:~$ psql dbname
psql (8.4.1)
Type "help" for help.
dbname=# \i /tmp/queries.sql
This will run the statements in /tmp/queries.sql.
Hope this was what you asked for.
In case of more detailed parameters:
$ psql -h [host] -p [port] -d [databaseName] -U [user] -f [/absolute/path/to/file]
The manual has some examples:
testdb=> \set content '''' `cat my_file.txt` ''''
testdb=> INSERT INTO my_table VALUES (:content);
See http://www.postgresql.org/docs/8.4/interactive/app-psql.html