I recently used loadimpact.com for performance testing , written a lua script and able to run in the loadimpact.com. Now how to run my lua script on different environments.I need to parameterize the environment URL , i can use csv files.But keeping csv files for different environment is not preferred.Can any one help me on this please.
Thanks
Two suggestions. You can use util.dns_remap() in version 3.0 (Lua scripting)
-- Remap loadimpact.com to IP 195.178.177.179
util.dns_remap("example.com", "195.178.177.179")
-- All requests to mysite.com will now be sent to 195.178.177.179
local response = http.request_batch({
{"GET", "http://example.com/"}
})
You should consider utilizing v4.0 and k6 (k6.io) with JavaScript scripting. When triggering your test from the command line you can pass environment variables to the test/script to test different environments. This would require rewriting your existing scripts though. e.g.
k6 cloud -e MY_HOSTNAME=dev.example.com script.js
Where the following script is script.js:
import { check, sleep } from "k6";
import http from "k6/http";
export default function() {
var r = http.get(`http://${__ENV.MY_HOSTNAME}/`);
check(r, {
"status is 200": (r) => r.status === 200
});
sleep(5);
}
This would allow you to utilize the Load Impact cloud service and solving the original problem a little more elegantly. You can either pass a new ENV variable for your tests or if you are automating tests in a CI pipeline you can pass a new ENV. If each build was creating it's own unique environment, this would be very handy.
I need to write a script, that takes an array of values and multithreaded way it (forks?) runs another script with a value from array as a param, but so max running forks would be set, so it would wait for script to finish if there are more than n running already. How do I do that?
There is a plugin named child_process, but not sure how to get it done, as it always waits for child termination.
Basically, in PHP it would be something like this (wrote it from head, may contain some syntax errors):
<php
declare(ticks = 1);
$data = file('data.txt');
$max=20;
$child=0;
function sig_handler($signo) {
global $child;
switch ($signo) {
case SIGCHLD:
$child -= 1;
}
}
pcntl_signal(SIGCHLD, "sig_handler");
foreach($data as $dataline){
$dataline = trim($dataline);
while($child >= $max){
sleep(1);
}
$child++;
$pid=pcntl_fork();
if($pid){
// SOMETHING WENT WRONG? NEVER HAPPENS!
}else{
exec("php processdata.php \"$dataline\"");
exit;
}//fork
}
while($child != 0){
sleep(1);
}
?>
After the conversation in the comments, here's how to have Node executing your PHP script.
Since you're calling an external command, there's no need to create a new thread. The Node.js runloop understands that calls to external commands are async operations, and it can execute all of them at the same time.
You can see different ways for executing an external process in this SO question (linked answer may be the best in your case).
However, since you're already moving everything to Node, you may even consider rewriting your "process.php" script to Node.js code. Since, as you explained, that script connects to remote servers and databases and uses nslookup (which you may not really need with Node.js), you won't need any separate thread: they're all async operations that Node.js excels at performing.
I am trying to use a perl script to transfer files from one machine to another within a cron job. However for security reasons the cron job has to run as a unprivileged user. Now if I try to establish a connection using this unpriviliged user, Net::SFTP::Foreign always refuses to connect. Here is the part of the script I am having trouble with:
my $host = "hostname";
my %args = (
user => "username",
password => "password",
port => '12345'
);
my $sftp_connection = Net::SFTP::Foreign->new($host, %args);
if( $sftp_connection->error ) {
log_message( "E", "Error " . $sftp_connection->status() . " connecting to " . $host );
die;
}
log_message( "A", "Connected" );
I cannot give a full example, as this would require username and password.
If I execute this script as root, everything works fine, however if I try to use another user, the connection always fails.
Is there a way to get some more diagnostic information? I think there was a way to get more output from the actual sftp process, but I cannot look it up right now as cpan currently does not work from here.
I previously also tried using Net::SFTP instead of Net::SFTP, but the error handling at later parts did not work correctly, so switching to Net::SFTP does not seem a viable option right now.
Use metacpan.org
Debugging:
For debugging purposes you can run ssh in verbose mode passing it the -v option:
my $sftp = Net::SFTP::Foreign->new($host, more => '-v');
"Module description"
Enabling debugging (thanks to all commenters), I could see that the host-key verification failed. So after I added the key to the list of allowed keys (by doing a manual ssh to the host once), everything worked fine.
For root the key must have been in the list already, so that it had nothing to do with priviledge, just with prior usage of the account (I seem have ssh'ed before from root to the other host).
Is there a way to make a script where I can do stuff like $this->EE->db (i.e. using Expression Engine's classes, for example to access the database), but that can be run in the command line?
I tried searching for it, but the docs don't seem to contain this information (please correct me if I'm wrong). I'm using EE 2.4 (the link above should point to 2.4 docs).
The following article seems to have a possible approach: Bootstrapping EE for CLI Access
Duplicate your index.php file and name it cli.php.
Move the index.php file outside your DOCUMENT_ROOT. Now, technically, this isn’t required, but there’s no reason for prying
eyes to see your hard work so why not protect it.
Inside cli.php update the $system_path on line 26 to point to your system folder.
Inside cli.php update the $routing['controller'] on line 96 to be cli.
Inside cli.php update the APPPATH on line 96 to be $system_path.'cli/'.
Duplicate the system/expressionengine directory and name it system/cli.
Duplicate the cli/controllers/ee.php file and name it cli/controllers/cli.php.
Finally, update the class name in cli/controllers/cli.php to be Cli and remove the methods.
By default EE calls the index method, so add in an index method to do what you need.
#Zenbuman This was useful as a starting point although I would add I had issues with all of my requests going to cli -> index, whereas I wanted some that went to cli->task1, cli->task2 etc
I had to update *system\codeigniter\system\core\URI.php*so that it knew how to extract the parameters I was passing via the command line, I got the code below from a more recent version of Codeigniter which supports the CLI
// Is the request coming from the command line?
if (php_sapi_name() == 'cli' or defined('STDIN'))
{
$this->_set_uri_string($this->_parse_cli_args());
return;
}
// Let's try the REQUEST_URI first, this will work in most situations
and also created the function in the same file
private function _parse_cli_args()
{
$args = array_slice($_SERVER['argv'], 1);
return $args ? '/' . implode('/', $args) : '';
}
Also had to comment out the following in my cli.php file as all routing was going to the index method in my cli controller and ignoring my parameters
/*
* ~ line 109 - 111 /cli.php
* ---------------------------------------------------------------
* Disable all routing, send everything to the frontend
* ---------------------------------------------------------------
*/
$routing['directory'] = '';
$routing['controller'] = 'cli';
//$routing['function'] = '';
Even leaving
$routing['function'] = '';
Will force requests to go to index controller
In the end I felt this was a bit hacky but I really need to use the EE API library in my case. Otherwise I would have just created a separate application with Codeigniter to handle my CLI needs, hope the above helps others.
I found #Zenbuman's answer after solving my own variation of this problem. My example allows you to keep the cron script inside a module, so if you need your module to have a cron feature it all stays neatly packaged together. Here's a detailed guide on my blog.
Would it be possible to block Tor users? (https://www.torproject.org/)
Due to the nature of the site I run I should do all I can to stop multiple accounts and block certain locations. Tor is worse than proxies - a total nightmare...
Tor is much easier to block than other open proxies since the list of exit IP addresses is known and published. Read the answer at https://www.torproject.org/docs/faq-abuse.html.en#Bans and if you still want to block users from accessing your site you could use https://www.torproject.org/projects/tordnsel.html.en or the Bulk Exit List exporting tool.
If you use the Bulk Exit List exporting tool be sure to get a fresh list often and expire the old blocks since the list of IP addresses change.
Blocking Tor is wrong because (ab)users and IP addresses are not the same. By blocking Tor you will also block legitimate users and harmless restricted Tor exit nodes configured with conservative exit policies.
For example, if you concerned about attacks on SSH (port 22) then blocking only Tor will do little to increase security. What you really might need is dynamic synchronised blacklist like http://denyhosts.sourceforge.net/ that track offenders disregarding of their affiliation with Tor.
Denyhosts will automatically block Tor exit nodes that allow Tor to access port 22 without unnecessary denying access to anonymous users and operators of Tor exit nodes who never let offenders to attack your SSH services.
The Tor Project actually provides its own list here:
https://check.torproject.org/exit-addresses
In .NET it's possible and simple. I have implemented it on my site.
Let's say your site has an external IP address of 192.168.0.5 for argument's sake. Real TOR IP address at the time of posting: 95.215.44.97
Imports System.Net
Imports System.Net.Sockets
Imports System.Web
Private Function IsTorExitNode(sIP As String) As Boolean
' Reverse your IP (97.44.215.95) then pass to the string, I have just created it as one for this example
Try
Dim strTor As String = "97.44.215.95.80.5.0.168.192.ip-port.exitlist.torproject.org"
Dim host As IPHostEntry = Dns.GetHostEntry(strTor)
If host.AddressList.Length = 0 Then
Return False
Else
If host.AddressList(0).ToString() = "127.0.0.2" Then
Return True
Else
Return False
End If
End If
Catch ex As SocketException
Return False
End Try
End Function
Breakdown
Reversed IP address: 97.44.215.95
Port: 80
Reversed IP address: (your external site IP address)
If the address is a TorExitNode it will return 127.0.0.2.
In your Global.asax file, you can use the Application_Start to check if IP address returns true and then redirect them away from your site:
If IsTorExitNode("97.44.215.95") = True Then Response.Redirect("http://www.google.co.uk")
Now, as soon as they hit your site they are redirected away from it.
TOR has a list of IP addresses, but obviously they change all the time so using my function would be the best way as it's always real-time.
You can use the TorDNSEL service to perform a live query about whether a specific IP address is a Tor exit node. You query the service via a specially-formed DNS request.
Here is some sample PHP code that performs the lookup:
function isTorExitNode() {
$serverPort = $_SERVER['SERVER_PORT'];
$remoteAddr = reverseIp(getClientIp());
$serverAddr = reverseIp($_SERVER['SERVER_ADDR']);
$placeholders = '%s.%s.%s.ip-port.exitlist.torproject.org';
$name = sprintf($placeholders, $remoteAddr, $serverPort, $serverAddr);
return ( gethostbyname($name) === '127.0.0.2' );
}
function getClientIp() {
if (isset($_SERVER['HTTP_CF_CONNECTING_IP'])) {
return $_SERVER['HTTP_CF_CONNECTING_IP'];
}
return $_SERVER['REMOTE_ADDR'];
}
function reverseIp($ip) {
$ipParts = explode('.', $ip);
return $ipParts[3] . '.' . $ipParts[2] . '.' .
$ipParts[1] . '.' . $ipParts[0];
}
if (!isTorExitNode()) {
// Do nothing
} else {
Die("Sorry, You cannot use TOR network!!!");
}
Important Notes:
This example supports IPv4 addresses only, not IPv6.
It could take a couple of seconds to get a response, so be careful about introducing delays into your site rendering.
Since TorDNSEL was deprecated and replaced by a new system in april 2020 [1], most of the answers in this thread are outdated.
After a bit of wrangling I came up with this code that uses the new checker. What it does is it reverses the ip octets and creates a URL for the new checker, then performs a DNS request and checks wether or not the first answer has the "127.0.0.2" IP. If this is the case, the user is deemed to come from Tor, otherwise it returns false.
function IsTorExitPoint(){
$dns_record = dns_get_record(ReverseIPOctets($_SERVER['REMOTE_ADDR']).".dnsel.torproject.org.");
if ($dns_record && $dns_record[0] && $dns_record[0]["ip"] == "127.0.0.2") {
return true;
} else {
return false;
}
}
function ReverseIPOctets($inputip){
$ipoc = explode(".",$inputip);
return $ipoc[3].".".$ipoc[2].".".$ipoc[1].".".$ipoc[0];
}
[1] https://lists.torproject.org/pipermail/tor-project/2020-March/002759.html
PD: It's been a while since I've posted an answer to stackoverflow, so please bear with me and help me improve if possible.
It's a fact, that the best application defence is its code and security, not a firewall blocklist. If it's an essential matter for you to have real true users - you have to use two-factor authentication. Blocklists are totally useless nowadays.
Here (see https://github.com/RD17/DeTor) is a simple REST API to determine whether a request was made from TOR network or not.
The request is:
curl -X GET http://detor.ambar.cloud/.
The response is:
{
"sourceIp": "104.200.20.46",
"destIp": "89.207.89.82",
"destPort": "8080",
"found": true
}
As a bonus you can add a badge to your site to detect whether a user comes from TOR or not:
<img src="http://detor.ambar.cloud/badge" />
(This was written for a PHP specific question that was subsequently deleted and linked here as a duplicate).
Disclaimer: Consider the impact of blocking all Tor users as raised in the best answer here. Consider only blocking functions such as registration, payment, comments etc and not a blanket block on everything.
--
Here are two pure PHP solutions. The first downloads and caches a Tor node list and compares the visitor IP against the list. The second uses the Tor DNS Exit List project to determine if the visitor is using Tor via DNS lookups.
Method #1 (Checking IP against a Tor relay list):
Using the following set of functions we can determine if an IP belongs to the Tor network by checking it against a dynamic exit list that gets downloaded and cached for 10 minutes. Feel free to use this list but please cache for 10 minutes when possible.
Where you want to enforce the Tor check, you can simply use:
$isTorUser = isTorUser($_SERVER['REMOTE_ADDR']);
if ($isTorUser) {
// blocking action
}
Here is the code which you can put in a separate functions file and include when you want to run the check. Note, you may want to adjust some of it to change the path to the cache file.
<?php
function isTorUser($ip)
{
$list = getTorExitList();
if (arrayBinarySearch($ip, $list) !== false) {
return true;
} else {
return false;
}
}
function getTorExitList()
{
$path = __DIR__ . '/tor-list.cache';
if ( file_exists($path) && time() - filemtime($path) < 600 ) {
$list = include $path;
if ($list && is_array($list)) {
return $list;
}
}
$data = file('https://www2.openinternet.io/tor/tor-exit-list.txt');
if (!$data) {
return array();
}
$list = array();
foreach($data as $line) {
$line = trim($line);
if ($line == '' || $line[0] == '#') continue;
list($nick, $ip) = explode("\t", $line);
$list[] = $ip;
}
sort($list);
file_put_contents($path, sprintf("<?php return %s;", var_export($list, true)));
return $list;
}
/**
* Perform binary search of a sorted array.
* Credit: http://php.net/manual/en/function.array-search.php#39115
*
* Tested by VigilanTor for accuracy and efficiency
*
* #param string $needle String to search for
* #param array $haystack Array to search within
* #return boolean|number false if not found, or index if found
*/
function arrayBinarySearch($needle, $haystack)
{
$high = count($haystack);
$low = 0;
while ($high - $low > 1){
$probe = ($high + $low) / 2;
if ($haystack[$probe] < $needle){
$low = $probe;
} else{
$high = $probe;
}
}
if ($high == count($haystack) || $haystack[$high] != $needle) {
return false;
} else {
return $high;
}
}
Method #2 (Checking IP against the Tor DNS Exit List Project):
The DNS exit check is a bit more robust in that it takes into account the relay's exit policy and looks at what IP and port on your server the client is connecting to and if such exit traffic is permitted, it will return a match. The potential downfall is that if the DNS project is down temporarily, DNS requests can hang before timing out slowing things down.
For this example, I will use a class from a library I wrote and maintain called TorUtils.
First, you'll need to install it with Composer using composer require dapphp/torutils and include the standard vendor/autoloader.php code in your application.
The code for the check:
$isTor = false;
try {
// check for Tor using the remote (client IP)
if (TorDNSEL::isTor($_SERVER['REMOTE_ADDR'])) {
// do something special for Tor users
} else {
// not using Tor, educate them! :-D
}
} catch (\Exception $ex) {
// This would likely be a timeout, or possibly a malformed DNS response
error_log("Tor DNSEL query failed: " . $ex->getMessage());
}
if ($isTor) {
// blocking action
}
Additional Considerations
If your application uses PHP sessions, I'd highly suggest caching the "isTorUser" response into the session (along with the source IP) and only run the check initially or when the IP changes (e.g. $_SERVER['REMOTE_ADDR'] != $_SESSION['last_remote_addr']) as not to perform many duplicated lookups. Even though they try to be very efficient, it's a waste to do over and over for the same IP.
I found a list of all the Tor nodes updated every half an hour: https://www.dan.me.uk/tornodes
This SHOULD include the exit, entries and bridge nodes used to connect and browse through Tor.
Use this Perl script to gather the IP addresses from a downloaded webpage:
perl -lne 'print $& if /(\d+\.){3}\d+/' downloadedwebpage.html > listofips.out
It will give you a list of IP addresses , one per line. I have tried to find something that will do this without the Perl script, but after many hours searching I could not find one.
I hope this helps.
I also found some good information here too on the same site:
https://www.dan.me.uk/dnsbl
Detecting Tor traffic is rather easy. The main way to do this is to monitor the Tor exit node list and compare the IP against the list.
I had the need to do such a thing recently and built a small Ruby gem to keep the list of exit nodes up to date and provide a simple way to detect exit nodes. I also wrote a small executable you can use to detect exit nodes.
The gem is open source and can be found here: tor-guard
Installing the gem is simple enough:
$ gem install tor-guard
Using the library in your own Ruby code can be done as follows:
require 'tor-guard'
if TorGuard.exit_node?('108.56.199.13')
puts "Yep, it's an exit node!"
end
The executable is also easy to use:
$ tg 108.56.199.13 && echo "Yep, it's an exit node"
It is possible due to the tor project publishing a list of exit proxies.
The list of exit proxies can be downloaded directly from the project at https://check.torproject.org/exit-addresses in space delimited text form.
I have written a python script to add iptables rules for all exit nodes that reject all packets from them. You can find the script on github here: https://github.com/vab/torblock
If the Tor Project ever decides to stop publishing a list of exit nodes it will be possible to block them. Code would just need to be written to connect to the tor network and discover the exit nodes.
Yes, and in fact here is a script that will do it for all of your windows machines. Like others mentioned above, it's as simple as blocking all the exit nodes, but that takes a little work.
https://github.com/Austin-Src/BlockTor
I have already curated the tor nodes and tor exit nodes list which keep updating hourly. Please refer to https://github.com/SecOps-Institute/Tor-IP-Addresses
You can do a git pull every hour and get the most updated list.
For whatever reason I wasn't able to find another answer on here, as of now (20 Shevat 5781 (from Creation)) that has this particular link, so here it is:
https://check.torproject.org/torbulkexitlist
I got it by downloading Tor, then opening up a find my IP address website, then navigating to that IP address (it happens to be http://195.176.3.20/, if you navigate to it you should find the list also), and navigating to it....