On my windows machine I have CQLSH working and using a .cert file
Now I am starting to use DSBulk, but can't get the command line to know where to find my certificate.
I have a cert file here: C:\myfolder\mycert.cer
Here is a sample of my command line:
dsbulk count --ssl -u "myusername" -p "mypassword" -h "123.12.123.12" -k "mykeyspace" -query "select count(*) from mytable;"
the error message:
Operation failed: Expecting long or short option, got: 'myusername'
I suspect that I need to modify my command parameters to reference the cert file.
Any advice would be greatly appreciated!
#John O'Sullivan
According to the documentation idea shared above by Alex, you need to feed a file to the dsbulk command:
dsbulk {
connector.name = "csv"
connector.csv.delimiter = "|"
schema.keyspace = "myKeyspace"
schema.table = "myTable"
schema.mapping = "0=name, 1=age, 2=email"
}
datastax-java-driver {
advanced {
ssl-engine-factory {
keystore-password = "cassandra"
keystore-path = "/Users/myaccount/tmp/ssl/keystore.node0"
class = DefaultSslEngineFactory
truststore-password = "dse#r0cks!"
truststore-path = "/Users/myaccount/tmp/ssl/truststore.node0"
}
}
}
Then the command line references the file:
dsbulk load -f my-application.conf -url file1.csv -k ks1 -t table1
The specific page you need to reference is:
https://docs.datastax.com/en/dsbulk/doc/dsbulk/dsbulkUseSsl.html
Related
I am trying to get values from postgresql DB using the bash command line : There is an issue trying to get run a select statement from the table.
For example if I execute this select statement, it return successful and gives the value
psql -U postgres -d postgres -p 5432 -t -c "select count(*) from sampledata.sif_work where servicerequesttype='CreatepostgresCase'"
However when I tried to add more where statement either hardcoded or variables to the WHERE statement, I got this error :
ERROR: operator does not exist: character varying <> integer
LINE 1: ...questtype='CreatepostgresCase' and applicationerrorcode!=25 and a...
^
HINT: No operator matches the given name and argument types. You might need to add explicit type casts.
The script :
#!/bin/bash
errorCodeSuccess=0
errorCodeFailure=30
sampleDbUser=postgres
sampleDBPort=5432
appErrorCodeFailure=25
#hardcoded
psql -U postgres -d postgres -p 5432 -t -c "select count(*) from sampledata.sif_work where servicerequesttype='CreatepostgresCase' and applicationerrorcode=25 and pxcreatedatetime>current_date"
#variables used
psql -U "${sampleDbUser}" -d postgres -p "${sampleDBPort}" -t -c "select count(*) from sampledata.sif_work where servicerequesttype='CreatepostgresCase' and applicationerrorcode!="${appErrorCodeFailure}" and applicationerrorcode!="${errorCodeSuccess}" and pxcreatedatetime>current_date"
Any reason why even though I hardcoded the value, it is still throwing error. Any reason ?
PostgreSQL understands 25 as an integer literal but '25' will be interpreted as a text literal/string constant, which would work with your character varying type column.
You could add the single quote ' before you close and after you open the double quotes ", but you also don't need to close that double-quoted string at all - bash evaluates $ expressions in double quotes:
errorCodeSuccess=0
errorCodeFailure=30
sampleDbUser=postgres
sampleDBPort=5432
appErrorCodeFailure=25
#hardcoded
psql -U postgres -d postgres -p 5432 -t \
-c "select count(*)
from sampledata.sif_work
where servicerequesttype='CreatepostgresCase'
and applicationerrorcode='25'--single quotes indicate a text literal
and pxcreatedatetime>current_date"
#variables used
psql -U "${sampleDbUser}" -d postgres -p "${sampleDBPort}" -t \
-c "select count(*)
from sampledata.sif_work
where servicerequesttype='CreatepostgresCase'
and applicationerrorcode!='${appErrorCodeFailure}'
and applicationerrorcode!='${errorCodeSuccess}'
and pxcreatedatetime>current_date; "
You already knew you can safely use single quotes within a double-quoted string, looking at servicerequesttype='CreatepostgresCase'.
You can also make the single quotes a part of the value:
#already doesn't work:
errorCodeSuccess=0
#same effect:
errorCodeSuccess='0'
#this will be interpreted as a column named "0":
errorCodeSuccess='"0"'
#"0" would be a valid name, but I doubt you have one or want one
#this will work:
errorCodeSuccess="'0'"
errorCodeFailure="'30'"
sampleDbUser=postgres
sampleDBPort=5432
psql -U "${sampleDbUser}" -d postgres -p "${sampleDBPort}" -t \
-c "select count(*)
from sampledata.sif_work
where servicerequesttype='CreatepostgresCase'
and applicationerrorcode != ${appErrorCodeFailure}
and applicationerrorcode != ${errorCodeSuccess}
and pxcreatedatetime>current_date; "
Keep in mind that it's always unsafe to construct queries this way - both in terms of security and convenience. You could start improving this with psql -v.
I have a table named "myTable" in my PostgreSQL database which has 4 columns - id, non_keyword_columnA, non_keyword_columnB and group.
Structure of the table is as follows:
Column | Type | Modifiers
--------------------+------------+------------------------------------------
id | integer | not null default nextval('myTable_id_seq'::regclass)
non_keyword_columnA | integer |
non_keyword_columnB | integer |
group | integer | not null
Foreign-key constraints:
"tablename_group_fkey" FOREIGN KEY ("group") REFERENCES groups(id)
I want to insert data into this table using shell and i am using following code to do it:
sudo /usr/sbin/chroot environment_path_here su - username -c "psql -A -z -c \"INSERT INTO myTable (non_keyword_columnA ,non_keyword_columnB ,"group") VALUES (1,2,(SELECT id from groups WHERE name='someGroupName'));\""
I am not an expert in Databases but I understand that group is a keyword which can be used in psql queries if used with double quotes which i did in above script.
But receive error as
ERROR: syntax error at or near "group"
LINE 1: ...RT INTO myTable(entity,machine,group) VAL...
^`
If I enter the environment manually and then execute the psql query then query executes successfully and row gets inserted but nothing is working through shell script.
I have tried various permutations and combinations to try as escape group keyword by using these combinations:
sudo /usr/sbin/chroot environment_path_here su - username -c "psql -A -z -c \"INSERT INTO myTable (non_keyword_columnA ,non_keyword_columnB ,\""group"\") VALUES (231,3355,(SELECT id from groups WHERE name='releventGroupName'));\""
sudo /usr/sbin/chroot environment_path_here su - username -c "psql -A -z -c \"INSERT INTO myTable (non_keyword_columnA ,non_keyword_columnB ,"\"group\"") VALUES (231,3355,(SELECT id from groups WHERE name='releventGroupName'));\""
But none of them have worked till now. I am not an expert in shell either so it may be possible that I might be making some really silly mistake here. Any help would be appreciated.
Use a here-document as standard input for your psql.
(Untested:)
#!/bin/sh
(sudo /usr/sbin/chroot environment_path_here su - username -c 'psql -A -z' ) <<OMG
INSERT INTO myTable (non_keyword_columnA ,non_keyword_columnB ,"group")
SELECT 1,2,id
FROM groups
WHERE name = 'someGroupName'
;
OMG
see also...
i need to load data from flat file into mariaDB on linux environment.
i've plan to put mariaDB script on shell file. then call shell from cron.
mariadb script shown as follow:
set #path = (select path_file from param);
set #tbl = (select table_name from param);
set #x = concat(
'LOAD DATA LOCAL INFILE ',
#path,
' INTO TABLE ', #tbl,
' (#row) set id = trim(substr(#row,1,2)), name = trim(substr(#row,3,19)), address= trim(substr(#row,22,20))'
);
prepare y from #x;
execute y;
deallocate prepare y;
when i execute the script directly on heidisql,
error shown:
this command is not supported in the prepared statement protocol yet
does any one have better way to load data from flat file into MariaDB on linux environment regularly (scheduled) without using any ETL tools?
Thanks.
One option you can try is (adjust as needed):
File: load_data.sh
path=$(mysql -u ${mysql_user} -p${mysql_password} -s -N <<GET_PATH
SELECT '/path/to/file/data.csv';
GET_PATH
)
tbl=$(mysql -u ${mysql_user} -p${mysql_password} -s -N <<GET_TABLE
SELECT 'table';
GET_TABLE
)
# mysql -u ${mysql_user} -p${mysql_password} -s -N <<LOAD_DATA
# LOAD DATA LOCAL INFILE '${path}'
# INTO TABLE \`${tbl}\` ...
# LOAD_DATA
# TEST
cat <<LOAD_DATA
LOAD DATA LOCAL INFILE '${path}'
INTO TABLE \`${tbl}\` ...
LOAD_DATA
Command line:
$ ls -l
-r-x------ load_data.sh
$ ./load_data.sh
LOAD DATA LOCAL INFILE '/path/to/file/data.csv'
INTO TABLE `table` ...
For clarity, write as much of the SQL into a STORED PROCEDURE. Then use bash to call that SP.
I have something like this on a Jenkinsfile (Groovy) and I want to record the stdout and the exit code in a variable in order to use the information later.
sh "ls -l"
How can I do this, especially as it seems that you cannot really run any kind of groovy code inside the Jenkinsfile?
The latest version of the pipeline sh step allows you to do the following;
// Git committer email
GIT_COMMIT_EMAIL = sh (
script: 'git --no-pager show -s --format=\'%ae\'',
returnStdout: true
).trim()
echo "Git committer email: ${GIT_COMMIT_EMAIL}"
Another feature is the returnStatus option.
// Test commit message for flags
BUILD_FULL = sh (
script: "git log -1 --pretty=%B | grep '\\[jenkins-full]'",
returnStatus: true
) == 0
echo "Build full flag: ${BUILD_FULL}"
These options where added based on this issue.
See official documentation for the sh command.
For declarative pipelines (see comments), you need to wrap code into script step:
script {
GIT_COMMIT_EMAIL = sh (
script: 'git --no-pager show -s --format=\'%ae\'',
returnStdout: true
).trim()
echo "Git committer email: ${GIT_COMMIT_EMAIL}"
}
Current Pipeline version natively supports returnStdout and returnStatus, which make it possible to get output or status from sh/bat steps.
An example:
def ret = sh(script: 'uname', returnStdout: true)
println ret
An official documentation.
quick answer is this:
sh "ls -l > commandResult"
result = readFile('commandResult').trim()
I think there exist a feature request to be able to get the result of sh step, but as far as I know, currently there is no other option.
EDIT: JENKINS-26133
EDIT2: Not quite sure since what version, but sh/bat steps now can return the std output, simply:
def output = sh returnStdout: true, script: 'ls -l'
If you want to get the stdout AND know whether the command succeeded or not, just use returnStdout and wrap it in an exception handler:
scripted pipeline
try {
// Fails with non-zero exit if dir1 does not exist
def dir1 = sh(script:'ls -la dir1', returnStdout:true).trim()
} catch (Exception ex) {
println("Unable to read dir1: ${ex}")
}
output:
[Pipeline] sh
[Test-Pipeline] Running shell script
+ ls -la dir1
ls: cannot access dir1: No such file or directory
[Pipeline] echo
unable to read dir1: hudson.AbortException: script returned exit code 2
Unfortunately hudson.AbortException is missing any useful method to obtain that exit status, so if the actual value is required you'd need to parse it out of the message (ugh!)
Contrary to the Javadoc https://javadoc.jenkins-ci.org/hudson/AbortException.html the build is not failed when this exception is caught. It fails when it's not caught!
Update:
If you also want the STDERR output from the shell command, Jenkins unfortunately fails to properly support that common use-case. A 2017 ticket JENKINS-44930 is stuck in a state of opinionated ping-pong whilst making no progress towards a solution - please consider adding your upvote to it.
As to a solution now, there could be a couple of possible approaches:
a) Redirect STDERR to STDOUT 2>&1
- but it's then up to you to parse that out of the main output though, and you won't get the output if the command failed - because you're in the exception handler.
b) redirect STDERR to a temporary file (the name of which you prepare earlier) 2>filename (but remember to clean up the file afterwards) - ie. main code becomes:
def stderrfile = 'stderr.out'
try {
def dir1 = sh(script:"ls -la dir1 2>${stderrfile}", returnStdout:true).trim()
} catch (Exception ex) {
def errmsg = readFile(stderrfile)
println("Unable to read dir1: ${ex} - ${errmsg}")
}
c) Go the other way, set returnStatus=true instead, dispense with the exception handler and always capture output to a file, ie:
def outfile = 'stdout.out'
def status = sh(script:"ls -la dir1 >${outfile} 2>&1", returnStatus:true)
def output = readFile(outfile).trim()
if (status == 0) {
// output is directory listing from stdout
} else {
// output is error message from stderr
}
Caveat: the above code is Unix/Linux-specific - Windows requires completely different shell commands.
this is a sample case, which will make sense I believe!
node('master'){
stage('stage1'){
def commit = sh (returnStdout: true, script: '''echo hi
echo bye | grep -o "e"
date
echo lol''').split()
echo "${commit[-1]} "
}
}
For those who need to use the output in subsequent shell commands, rather than groovy, something like this example could be done:
stage('Show Files') {
environment {
MY_FILES = sh(script: 'cd mydir && ls -l', returnStdout: true)
}
steps {
sh '''
echo "$MY_FILES"
'''
}
}
I found the examples on code maven to be quite useful.
All the above method will work. but to use the var as env variable inside your code you need to export the var first.
script{
sh " 'shell command here' > command"
command_var = readFile('command').trim()
sh "export command_var=$command_var"
}
replace the shell command with the command of your choice. Now if you are using python code you can just specify os.getenv("command_var") that will return the output of the shell command executed previously.
How to read the shell variable in groovy / how to assign shell return value to groovy variable.
Requirement : Open a text file read the lines using shell and store the value in groovy and get the parameter for each line .
Here , is delimiter
Ex: releaseModule.txt
./APP_TSBASE/app/team/i-home/deployments/ip-cc.war/cs_workflowReport.jar,configurable-wf-report,94,23crb1,artifact
./APP_TSBASE/app/team/i-home/deployments/ip.war/cs_workflowReport.jar,configurable-temppweb-report,394,rvu3crb1,artifact
========================
Here want to get module name 2nd Parameter (configurable-wf-report) , build no 3rd Parameter (94), commit id 4th (23crb1)
def module = sh(script: """awk -F',' '{ print \$2 "," \$3 "," \$4 }' releaseModules.txt | sort -u """, returnStdout: true).trim()
echo module
List lines = module.split( '\n' ).findAll { !it.startsWith( ',' ) }
def buildid
def Modname
lines.each {
List det1 = it.split(',')
buildid=det1[1].trim()
Modname = det1[0].trim()
tag= det1[2].trim()
echo Modname
echo buildid
echo tag
}
If you don't have a single sh command but a block of sh commands, returnstdout wont work then.
I had a similar issue where I applied something which is not a clean way of doing this but eventually it worked and served the purpose.
Solution -
In the shell block , echo the value and add it into some file.
Outside the shell block and inside the script block , read this file ,trim it and assign it to any local/params/environment variable.
example -
steps {
script {
sh '''
echo $PATH>path.txt
// I am using '>' because I want to create a new file every time to get the newest value of PATH
'''
path = readFile(file: 'path.txt')
path = path.trim() //local groovy variable assignment
//One can assign these values to env and params as below -
env.PATH = path //if you want to assign it to env var
params.PATH = path //if you want to assign it to params var
}
}
Easiest way is use this way
my_var=`echo 2`
echo $my_var
output
: 2
note that is not simple single quote is back quote ( ` ).
I'm trying to load my Azure SQL Data Warehouse using the bcp utility but have been running into issue after issue...I finally got a .txt file with one record to import successfully, but now when I put two or more records into the file, it bombs out with the error (via an error output file):
Row 1, Column 5: Numeric value out of range
The data looks like this:
2014-06-01,11111,test,used,1
2014-06-01,22222,test,used,1
and the table I'm importing to looks like this:
[Date] (date, not null)
[Code] (varchar(50), not null)
[Model] (varchar(100), not null)
[Type] (varchar(20), not null)
[Quantity] (int, not null)
I think it has something to do with the new line character but I haven't been able to work around it. I have tried changing the encoding in Notepad++ to ANSI, ISO-8859-1, UTF-8 w/o BOM, as well as UTF-16 LE & BE with Visual Studio CODE. When 'ANSI' was specified, the one-line file imported successfully. The end-of-line sequence is set to LF, and my bcp command is as follows:
bcp Schema.Table in C:\BcpFiles\sourceData.txt -S serverName -d databaseName -U userName -P password -q -c -t "," -r/n -e C:\BcpFiles\Errors.txt
The -r parameter requires a back slash rather than forward: try -r \n instead. This article explains the various combinations: https://msdn.microsoft.com/en-gb/library/ms191485.aspx
UPDATE:
create table tst (
[Date] date not null,
[Code] varchar(50) not null,
[Model] varchar(100) not null,
[Type] varchar(20) not null,
[Quantity] int not null
)
And then using this:
bcp dbo.tst in so.txt -S TONYMSI -d AdventureWorks2012 -T -q -c -t "," -r \n
Worked fine.