perl - print sql row - linux

I want to print all sql result of my perl script, I've connected to the database and I want to show the result of my sql query :
MySQL("SELECT * FROM test");
# define subroutine to submit MySQL command
sub MySQL
{
#Connect to the database.
my $dbh = DBI->connect("DBI:mysql:database=database;host=ip",
"login", 'password',
{'RaiseError' => 1});
my $query = $_[0]; #assign argument to string
my $sth = $prepare($query); #prepare query
$sth->execute(); #execute query
while (my #row = $sth->fetchrow_array)
{
print "#row\n";
}
}
I have this errors :
Global symbol "$prepare" requires explicit package name at test3.pl line 34.
syntax error at test3.pl line 34, near "$prepare("
Global symbol "$sth" requires explicit package name at test3.pl line 36.
Execution of test3.pl aborted due to compilation errors.

Change this line:
my $sth = $prepare($query);
to
my $sth = $dbh->prepare($query);
and don't forget to close the $sth and $dbh handles after the while loop with:
$sth->finish;
$dbh->disconnect;

In your last post there is only one reference to e in the following comment
#INNER JOIN events e ON (e.objectid = t.triggerid)
probably it is better to use syntax INNER JOIN events as e ON (e.objectid = t.triggerid) -- more readable

I do not have a database at hand available right at this moment. The following example should work in theory. I have replaced fetch_array on fetch_hashref the output will provide information about each row
ddddd column=value\tcolumn=value\t...
use strict;
use warnings;
my $query = qq(SELECT * FROM tb_test);
db_query($query);
sub db_query {
my $query = shift;
my $count = 1;
my $dbh = DBI->connect(
"DBI:mysql:database=database;host=ip",
"login", 'password',
{'RaiseError' => 1}
);
my $sth = $dbh->prepare($query);
$sth->execute();
while ( my $row = $sth->fetchrow_hashref() ) {
print "%5d ", $count++; # rows count
while( my($k,$v) = each %$row ) { print "%s=%s\t",$k,$v; }
print "\n"; # we done with current row
}
$sth->finish;
$dbh->disconnect;
}
You need look into DBI documentation to get full understanding how it works.

Related

Add eight thousand line in one commit [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 4 years ago.
Improve this question
I have a log file, perl script who took the log file and transcript the log file, i want to send this all line (eight thousand line) in one commit
my script:
# Connect to the database.
my $dbh = DBI->connect(
"DBI:mysql:database=DB;host=>IP",
"hostname", 'password',
{'RaiseError' => 1,'AutoCommit'=> 0}
);
open (FILE, 'file.log');
while (<FILE>) {
($word1, $word2, $word3, $word4, $word5, $word6, $word7, $word8, $word9, $word10, $word11, $word12, $word13, $word14) = split(" ");
$word13 =~ s/[^\d.]//g;
if ($word2 eq "Feb") {
$word2 = "02"
}
print "'$word5-$word2-$word3 $word4', $word11, $word13 \n";
eval {
#on peut utiliser insert mais il y aura des doublons et ici on est dans une table unique
my $sth = $dbh->prepare("INSERT INTO `test_query` (time, cstep, time_in_seconde) VALUES('$word5-$word2-$word3 $word4', $word11, $word13);");
#print $sth->rows . " rows found.\n";
#$sth->finish;
# do inserts, updates, deletes, queries here
#$sth->execute() or die "execution failed: $dbh->errstr()";
$sth->execute() or die "execution failed: $dbh->errstr()";
$dbh->commit();
};
### If something went wrong...
}
}
$dbh->disconnect();
thanks
For better performance, you want to simplify your code and move as many code as possible out of the loop :
prepare the statement out of the loop, using bind parameters : the statement is always the same, only bind parameters get to change
commit out of the loop : this will increase performance and also has the advantage of making your process atomic. As all changes occur within the same database transaction, either all lines will be processed (and commited), or, if a failure occurs on any line, no line at all will be commited. While implementing this optimization you need to watch for resource usage on your database (this will typically require more space in the UNDO tablespace) ; if resources are not enough, either increase them or commit every Nth record (with N being as high as possible)
avoid printing inside the loop unless your really need it (I commented that line)
you are building a connection with RaiseError attribute enabled, but then you ignore errors that can occur at execute. If this is really what you want, then just disable the RaiseError attribute on the statement handler, and remove the eval around execute
Other considerations in terms of coding practices :
always use strict and use warnings
use an array to store parsed data instead of a list of scalars : could make your code faster and will make it more readable
Code :
use strict;
use warnings;
# Connect to the database.
my $dbh = DBI->connect(
"DBI:mysql:database=DB;host=>IP",
"hostname", 'password',
{'RaiseError' => 1,'AutoCommit'=> 0}
);
# prepare the insert statement
my $sth = $dbh->prepare("INSERT INTO `test_query` (time, cstep, time_in_seconde) VALUES(?, ?, ?)");
$sth->{RaiseError} = 0;
open (my $file, 'file.log') or die "could not open : $!";
while (<$file>) {
my #words = split / /;
$words[12] =~ s/[^\d.]//g;
if ($words[1] eq "Feb") {
$words[1] = "02" ;
}
# print "'$words[4]-$words[1]-$words[2] $words[3]', $words[10], $words[12] \n";
$sth->execute( "$words[4]-$words[1]-$words[2] $words[3]", $words[10], $words[12] );
}
$dbh->commit;
$dbh->disconnect;
The last solution, that would probably perform even faster than this one, is to use DBI method execute_array to perform bulk database inserts. Attribute ArrayTupleFetch can be used to provide a code reference that DBI will invoke everytime it is ready to perform the next INSERT : this code reference should read the next file line and provide an array reference of values suitable for INSERT. When the file is exhausted, the sub should return undef, which will indicate DBI that the bulk process is completed.
Code :
#!/usr/local/bin/perl
use strict;
use warnings;
use DBI;
# open the file
open (my $file, 'log.file') or die "could not open : $!";
# connect the database
my $dbh = DBI->connect("DBI:mysql:database=DB;host=ip", "hostname", 'password', {'RaiseError' => 1,'AutoCommit'=> 0});
# prepare the INSERT statement
my $sth = $dbh->prepare("INSERT INTO `test_query` (time, cstep, time_in_seconde) VALUES(?, ?, ?)");
# run bulk INSERTS
my $tuples = $sth->execute_array({
ArrayTupleStatus => \my #tuple_status,
ArrayTupleFetch => sub {
my $line = <$file>;
return unless $line;
my #words = split / /;
# ... do anything you like with the array, then ...
return [ "$words[4]-$words[1]-$words[2] $words[3]", $words[10], $words[12] ];
}
});
if ($tuples) {
print "Successfully inserted $tuples records\n";
} else {
# do something usefull with #tuple_status, that contains the detailed results
}
$dbh->commit;
$dbh->disconnect;

format the following result file into a tabular format using Perl

I have a sort of a problem, and I am still novice with Perl.
I just want to ask how can I format the following results file into an Excel readable format (let's say CSV).
Result file example. llq1_dly.mt0
$MEAS COMMANDS SOURCE='llq1_dly.meas' DB FILE='clk_top_45h_lpe_sim.fsdb'
.TITLE '**-------------'
tdrll10_0 tdfll10_0 tdrll10_1 tdfll10_1 tdrll10_2 tdfll10_2 tdrll10_3
2.106560e-10 1.990381e-10 2.102583e-10 1.986280e-10 2.095036e-10 1.978480e-10 2.083813e-10
into the following file with a result like this one below
llq1_dly,tdr,tdf,
ll10_0,2.106560e-10,1.990381e-10,
ll10_1,2.102583e-10,1.986280e-10,
ll10_2,2.095036e-10,1.978480e-10,
ll10_3,2.083813e-10,1.967019e-10,
...
or more likely this one (to be compatible with engineering scientific notations):
llq1_dly,tdr,tdf,
ll10_0,210.6560e-12,199.0381e-12,
ll10_1,210.2583e-12,198.6280e-12,
ll10_2,209.5036e-12,197.8480e-12,
ll10_3,208.3813e-12,196.7019e-12,
...
Here's a program that produces the output you ask for. I don't generally approve of offering answers to questions where the OP hasn't made any attempt to write a solution themselves, but this question interested me.
It may well be that this could be written more simply, but you don't say what parts of the input are invariant. For instance, I have written it so that there can be any number of different columns with any names, rather than just tdr and tdf every time. As it is I have had to guess that the trailing part of each header ends in ll, so for instance tdrll10_0 is tdr and ll10_0. If that is wrong then you will need a different way of splitting the string.
I have written the program so that it reads from the DATA file handle. I trust you are able to write an open statement to read from the correct input file?
I hope this helps
use strict;
use warnings;
use 5.010;
use Number::FormatEng 'format_eng';
Number::FormatEng::use_e_zero();
my $fh = \*DATA;
my ($source, #headers, #values);
while ( <$fh> ) {
if ( /SOURCE=(?|'([^']+)'|"([^"]+)")/ ) { #' code highlighting fix
($source = $1) =~ s/\.[^.]*\z//;
}
elsif ( /^\.TITLE/ ) {
#headers = split ' ', <$fh>;
#values = split ' ', <$fh>;
last;
}
}
my #title = ( $source );
my (%headers, #table, #line);
for my $i ( 0 .. $#headers) {
my #fields = split /(?=ll)/, $headers[$i];
if ( $headers{$fields[0]} ) {
push #table, [ #line ];
#line = ();
%headers = ();
}
++$headers{$fields[0]};
push #line, $fields[1] if #line == 0;
push #line, format_eng($values[$i]);
push #title, $fields[0] unless #table;
}
print "$_," for #title;
print "\n";
for ( #table ) {
print "$_," for #$_;
print "\n";
}
__DATA__
$MEAS COMMANDS SOURCE='llq1_dly.meas' DB FILE='clk_top_45h_lpe_sim.fsdb'
.TITLE '**-------------'
tdrll10_0 tdfll10_0 tdrll10_1 tdfll10_1 tdrll10_2 tdfll10_2 tdrll10_3
2.106560e-10 1.990381e-10 2.102583e-10 1.986280e-10 2.095036e-10 1.978480e-10 2.083813e-10
output
llq1_dly,tdr,tdf,
ll10_0,210.656e-12,199.0381e-12,
ll10_1,210.2583e-12,198.628e-12,
ll10_2,209.5036e-12,197.848e-12,

Perl - Pseudo-hashes are deprecated

I am trying to get d request through perl. But i m getting below error :
#!/usr/bin/perl
use lib "/usr/packages/perl/perl-5.16.3/lib/5.16.3";
use strict;
use warnings;
use LWP::UserAgent;
use JSON;
use MIME::Base64;
my $url = 'https://example.com:8443/cli/agentCLI';
my $credentials = encode_base64('username:password');
my $ua = LWP::UserAgent->new(ssl_opts =>{ verify_hostname => 0});
my $response = $ua->get($url, 'Authorization' =>" Basic $credentials");
die 'http status: ' . $response->code . ' ' . $response->message
unless ($response->is_success);
my $json_obj = JSON->new->utf8->decode($response->content);
# the number of rows returned will be in the 'rowCount' propery
print $json_obj->{rowCount} . " rows:n";
# and the rows array will be in the 'rows' property.
foreach my $row(#{$json_obj->{rows}}){
#Results from this particular query have a "Key" and a "Value"
print $row->{Key} . ":" . $row->{Value} . "n";
}
output ( error ) :
Pseudo-hashes are deprecated at agent.pl line 21.
No such pseudo-hash field "rowCount" at agent.pl line 21.
Thanks,
Kalaiyarasan
See:
http://perldoc.perl.org/5.8.8/perlref.html#Pseudo-hashes%3A-Using-an-array-as-a-hash
In the recent version: http://perldoc.perl.org/perlref.html#Pseudo-hashes%3a-Using-an-array-as-a-hash
This is deprecated. I would imagine ( but can't tell without your JSON) that your JSON top level is an array.
Data::Dumper can help tell you what your actual data structure is.

Perl: String to anonymous array?

SOLVED ALREADY --> See edit 7
At this moment I'm fairly new on Perl, and trying to modify part of an existing page (in Wonderdesk).
The way the page works, is that it gets the information from the GET url and parses it to an SQL query.
Since this is part of a much larger system, I'm not able to modify the coding around it, and have to solve it in this script.
A working test I performed:
$input->{help_id} = ['33450','31976'];
When running this, the query that is being build returns something as
select * from table where help_id in(33450,31976)
The part of my code that does not work as expected:
my $callIDs = '33450,31450';
my #callIDs = split(/,/,$callIDs);
my $callIDsearch = \#callIDs;
$input->{help_id} = $callIDsearch;
When running this, the query that is being build returns something as
select * from table where help_id = '33450,31976'
I've tried to debug it, and used Data::Dumper to get the result of $callIDsearch, which appears as [33450, 31450] in my browser.
Can someone give me a hint on how to transform from '123,456' into ['123', '456']?
With kind regards,
Marcel
--===--
Edit:
As requested, minimal code piece that works:
$input->{help_id} = ['123','456']
Code that does not work:
$str = '123,456';
#ids = split(/,/,$str);
$input->{help_id} = \#ids;
--===--
Edit 2:
Source of the question:
The following part of the code is responsible for getting the correct information from the database:
my $input = $IN->get_hash;
my $db = $DB->table('help_desk');
foreach (keys %$input){
if (/^corr/ and !/-opt$/ and $input->{$_} or $input->{keyword}){
$db = $DB->table('help_desk','correspondence');
$input->{rs} = 'DISTINCT help_id,help_name,help_email,help_datetime,help_subject,help_website,help_category,
help_priority,help_status,help_emergency_flag,help_cus_id_fk,help_tech,help_attach';
$input->{left_join} = 1;
last;
}
}
# Do the search
my $sth = $db->query_sth($input);
my $hits = $db->hits;
Now instead of being able to provide a single parameter help_id, I want to be able to provide multiple parameters.
--===--
Edit 3:
query_sth is either of the following two, have not been able to find it out yet:
$COMPILE{query} = __LINE__ . <<'END_OF_SUB';
sub query {
# -----------------------------------------------------------
# $obj->query($HASH or $CGI);
# ----------------------------
# Performs a query based on the options in the hash.
# $HASH can be a hash ref, hash or CGI object.
#
# Returns the result of a query as fetchall_arrayref.
#
my $self = shift;
my $sth = $self->_query(#_) or return;
return $sth->fetchall_arrayref;
}
END_OF_SUB
$COMPILE{query_sth} = __LINE__ . <<'END_OF_SUB';
sub query_sth {
# -----------------------------------------------------------
# $obj->query_sth($HASH or $CGI);
# --------------------------------
# Same as query but returns the sth object.
#
shift->_query(#_)
}
END_OF_SUB
Or
$COMPILE{query} = __LINE__ . <<'END_OF_SUB';
sub query {
# -------------------------------------------------------------------
# Just performs the query and returns a fetchall.
#
return shift->_query(#_)->fetchall_arrayref;
}
END_OF_SUB
$COMPILE{query_sth} = __LINE__ . <<'END_OF_SUB';
sub query_sth {
# -------------------------------------------------------------------
# Just performs the query and returns an active sth.
#
return shift->_query(#_);
}
END_OF_SUB
--===--
Edit 4: _query
$COMPILE{_query} = __LINE__ . <<'END_OF_SUB';
sub _query {
# -------------------------------------------------------------------
# Parses the input, and runs a select based on input.
#
my $self = shift;
my $opts = $self->common_param(#_) or return $self->fatal(BADARGS => 'Usage: $obj->insert(HASH or HASH_REF or CGI) only.');
$self->name or return $self->fatal('NOTABLE');
# Clear errors.
$self->{_error} = [];
# Strip out values that are empty or blank (as query is generally derived from
# cgi input).
my %input = map { $_ => $opts->{$_} } grep { defined $opts->{$_} and $opts->{$_} !~ /^\s*$/ } keys %$opts;
$opts = \%input;
# If build_query_cond returns a GT::SQL::Search object, then we are done.
my $cond = $self->build_query_cond($opts, $self->{schema}->{cols});
if ( ( ref $cond ) =~ /(?:DBI::st|::STH)$/i ) {
return $cond;
}
# If we have a callback, then we get all the results as a hash, send them
# to the callback, and then do the regular query on the remaining set.
if (defined $opts->{callback} and (ref $opts->{callback} eq 'CODE')) {
my $pk = $self->{schema}->{pk}->[0];
my $sth = $self->select($pk, $cond) or return;
my %res = map { $_ => 1 } $sth->fetchall_list;
my $new_results = $opts->{callback}->($self, \%res);
$cond = GT::SQL::Condition->new($pk, 'IN', [keys %$new_results]);
}
# Set the limit clause, defaults to 25, set to -1 for none.
my $in = $self->_get_search_opts($opts);
my $offset = ($in->{nh} - 1) * $in->{mh};
$self->select_options("ORDER BY $in->{sb} $in->{so}") if ($in->{sb});
$self->select_options("LIMIT $in->{mh} OFFSET $offset") unless $in->{mh} == -1;
# Now do the select.
my #sel = ();
if ($cond) { push #sel, $cond }
if ($opts->{rs} and $cond) { push #sel, $opts->{rs} }
my $sth = $self->select(#sel) or return;
return $sth;
}
END_OF_SUB
--===--
Edit 5: I've uploaded the SQL module that is used:
https://www.dropbox.com/s/yz0bq8ch8kdgyl6/SQL.zip
--===--
Edit 6:
On request, the dumps (trimmed to only include the sections for help_id):
The result of the modification in Base.pm for the non-working code:
$VAR1 = [
33450,
31450
];
The result of the modification in Condition.pm for the non-working code:
$VAR1 = [
"help_id",
"IN",
[
33450,
31450
]
];
$VAR1 = [
"cus_username",
"=",
"Someone"
];
$VAR1 = [
"help_id",
"=",
"33450,31450"
];
The result for the modification in Base.pm for the working code:
$VAR1 = [
33450,
31976
];
The result for the modification in Condition.pm for the working code:
$VAR1 = [
"help_id",
"IN",
[
33450,
31976
]
];
It looks as if the value gets changed afterwards somehow :S
All I changed for the working/non-working code was to replace:
$input->{help_id} = ['33450','31976'];
With:
$input->{help_id} = [ split(/,/,'33450,31450') ];
--===--
Edit 7:
After reading all the tips, I decided to start over and found that by writing some logs to files, I could break down into the issue with more details.
I'm still not sure why, but it now works, using the same methods as before. I think it's a typo/glitch/bug in my code somewhere..
Sorry to have bothered you all, but I still recommend the points to go to amon due to his tips providing the breakthrough.
I don't have an answer, but I have found a few critical points where we need to know what is going on.
In build_query_cond (Base.pm line 528), an array argument will be transformed into an key in (...) relation:
if (ref($opts->{$field}) eq 'ARRAY' ) {
my $add = [];
for ( #{$opts->{$field}} ) {
next if !defined( $_ ) or !length( $_ ) or !/\S/;
push #$add, $_;
}
if ( #$add ) {
push #ins, [$field, 'IN', $add];
}
}
Interesting bit in sql (Condition.pm line 181). Even if there is an arrayref, an IN test will be simplified to an = test if it contains only a single element.
if (uc $op eq 'IN' || $op eq '=' and ref $val eq 'ARRAY') {
if (#$val > 1) {
$op = 'IN';
$val = '('
. join(',' => map !length || /\D/ ? quote($_) : $_, #$val)
. ')';
}
elsif (#$val == 0) {
($col, $op, $val) = (qw(1 = 0));
}
else {
$op = '=';
$val = quote($val->[0]);
}
push #output, "$col $op $val";
}
Before these two conditions, it would be interesting to insert the following code:
Carp::cluck(Data::Dumper::Dump(...));
where ... is $opts->{$field} in the first snippet or $cond in the second snippet. The resulting stack trace would allow us to find all subroutines which could have modified the value. For this to work, the following code has to be placed in your main script before starting the query:
use Carp ();
use Data::Dumper;
$Data::Dumper::Useqq = 1; # escape special characters
Once the code has been modified like this, run both the working and not-working code, and print out the resulting query with
print Dumper($result);
So for each of your code snippets, we should get two stack traces and one resulting SQL query.
A shot in the dark... there's a temporary array #callIDs created by this code:
my #callIDs = split(/,/,$callIDs);
my $callIDsearch = \#callIDs;
$input->{help_id} = $callIDsearch;
If some other part of your code modifies #callIDs, even after it's been assigned to $input->{help_id}, that could cause problems. Of course the fact that it's a lexical (my) variable means that any such changes to #callIDs are probably "nearby".
You could eliminate the named temporary array by doing the split like this:
$input->{help_id} = [ split(/,/,$callIDs) ];
I'm not sure I understand exactly why this is happening. It seems that your query builder needs an arrayref of strings. You can use map to do that
my $callIDs = '33450,31450';
my #callIDs = map {$_*1} split(/,/,$callIDs);
$input->{help_id} = \#callIDs;
This code should work
my $callIDs = '33450,31450';
$input->{help_id} = [split ",", $callIDs];
If your code somehow detect your data is number you can use
my $callIDs = '33450,31450';
$input->{help_id} = [map 0+$_, split ',', $callIDs];
If it somehow become number and you need string instead which should not in this case but advice for future work:
my $callIDs = '33450,31450';
$input->{help_id} = [map ''.$_, split ',', $callIDs];

words count example in Scala?

I see a lot of Scala tutorials with examples doing things like recrursive traversals or solving math problems. In my daily programming life I have the feeling most of my coding time is spent on mundane tasks like string manipulation, database queries and date manipulations. Is anyone interested to give an example of the a Scala version of the following perl script?
#!/usr/bin/perl
use strict;
#opens a file with on each line one word and counts the number of occurrences
# of each word, case insensitive
print "Enter the name of your file, ie myfile.txt:\n";
my $val = <STDIN>;
chomp ($val);
open (HNDL, "$val") || die "wrong filename";
my %count = ();
while ($val = <HNDL>)
{
chomp($val);
$count{lc $val}++;
}
close (HNDL);
print "Number of instances found of:\n";
foreach my $word (sort keys %count) {
print "$word\t: " . $count{$word} . " \n";
}
In summary:
ask for a filename
read the file (contains 1 word per line)
do away with line ends ( cr, lf or crlf)
lowercase the word
increment count of the word
print out each word, sorted alphabetically, and its count
TIA
A simple word count like that could be written as follows:
import io.Source
import java.io.FileNotFoundException
object WC {
def main(args: Array[String]) {
println("Enter the name of your file, ie myfile.txt:")
val fileName = readLine
val words = try {
Source.fromFile(fileName).getLines.toSeq.map(_.toLowerCase.trim)
} catch {
case e: FileNotFoundException =>
sys.error("No file named %s found".format(fileName))
}
val counts = words.groupBy(identity).mapValues(_.size)
println("Number of instances found of:")
for((word, count) <- counts) println("%s\t%d".format(word, count))
}
}
If you're going for concise/compact, you can in 2.10:
// Opens a file with one word on each line and counts
// the number of occurrences of each word (case-insensitive)
object WordCount extends App {
println("Enter the name of your file, e.g. myfile.txt: ")
val lines = util.Try{ io.Source.fromFile(readLine).getLines().toSeq } getOrElse
{ sys.error("Wrong filename.") }
println("Number of instances found of:")
lines.map(_.trim.toLowerCase).toSeq.groupBy(identity).toSeq.
map{ case (w,ws) => s"$w\t: ${ws.size}" }.sorted.foreach(println)
}
val lines : List[String] = List("this is line one" , "this is line 2", "this is line three")
val linesConcat : String = lines.foldRight("")( (a , b) => a + " "+ b)
linesConcat.split(" ").groupBy(identity).toList.foreach(p => println(p._1+","+p._2.size))
prints :
this,3
is,3
three,1
line,3
2,1
one,1

Resources