The sure way to protect against XSS? - security

I've looked through the questions and I haven't seen anyone ask this yet.
What is the for sure method to remove any sort of XSS attempts in some user submitted content? I know that < and > should be converted to < and > respectively but I've heard mention that encoding differences can get around this too.
Supposing a whitelist, what are all the steps to completely clean some user submitted content to ensure that no XSS vulnerabilities exist?

There is no absolute security concering XSS since people find new attack vectors every day. Sometimes XSS is even a browser bug you cant do anything about (excep some workarounds).
To get the idea of the complexity look at this (incomplete) xss attack cheat sheet.
http://ha.ckers.org/xss.html
Guess you should make yourself a XSS expert or hire one to reach your goal.
You can start by inspecting the attack vectors from the given link above, try to understand why it can work and make sure you prevent it.
Another great way of preventing XSS is to make sure you accept only stuff you expect instead of blocking stuff you know is bad. (i.e. whitelisting instead of blacklisting)

http://htmlpurifier.org/ - HTMLPurifier could be of help
ps: don't create your own code, there is no way you can cover all the issues. rely on the continually developed libraries such as HTMLPurifier.

Old Question, but XSS isn't old, its still up to date....
So, i would recommend you to have a look for the OWASP XSS Prevention Cheat Sheet and for a good overview the OWASP XSS overview
And if you need a really good way for escaping, have a look for the ESAPI Encoder API

Some of the holes I've seen fixed in different frameworks have been so unreal I don't get how they were found out. If you want to be 100% sure don't let users post content.

Proving a negative is a difficult proposition - as googletorp points out, the only 100% solution is to not have the problem in the first place.
Functionally, the xss attack cheat sheet at ha.ckers.org is a good place to start, as is a whitelisting approach - rather than disallowing things known to be bad, only allow things known to be good.

PHP-Firewall
against XSS etc
http://www.php-firewall.info/
http://code.google.com/p/php-firewall/
<?php
/************************************************************************/
/* PHP Firewall: Universal Firewall for WebSite */
/* ============================================ */
/* Write by Cyril Levert */
/* Copyright (c) 2009-2010 */
/* http://www.php-firewall.info */
/* dev#php-maximus.org */
/* Others projects: */
/* CMS PHP Maximus ( with mysql database ) www.php-maximus.org */
/* Blog PHP Minimus ( with mysqli database ) www.php-minimus.org */
/* Mini CMS PHP Nanomus ( without database ) www.php-nanomus.org */
/* Stop Spam Referer ( without database ) www.stop-spam-referer.info */
/* Twitter clone ( PHP Kweeker CMS ) www.twitter.php-minimus.org */
/* PHP Firewall ( without database ) www.php-firewall.info */
/* Personnal blog www.cyril-levert.info */
/* Release version 1.0.3 */
/* Release date : 12-04-2010 */
/* */
/* This program is free software. */
/************************************************************************/
/** IP Protected */
$IP_ALLOW = array();
/** configuration define */
define('PHP_FIREWALL_LANGUAGE', 'english' );
define('PHP_FIREWALL_ADMIN_MAIL', '' );
define('PHP_FIREWALL_PUSH_MAIL', false );
define('PHP_FIREWALL_LOG_FILE', 'logs' );
define('PHP_FIREWALL_PROTECTION_UNSET_GLOBALS', true );
define('PHP_FIREWALL_PROTECTION_RANGE_IP_DENY', true );
define('PHP_FIREWALL_PROTECTION_RANGE_IP_SPAM', false );
define('PHP_FIREWALL_PROTECTION_URL', true );
define('PHP_FIREWALL_PROTECTION_REQUEST_SERVER', true );
define('PHP_FIREWALL_PROTECTION_SANTY', true );
define('PHP_FIREWALL_PROTECTION_BOTS', true );
define('PHP_FIREWALL_PROTECTION_REQUEST_METHOD', true );
define('PHP_FIREWALL_PROTECTION_DOS', true );
define('PHP_FIREWALL_PROTECTION_UNION_SQL', true );
define('PHP_FIREWALL_PROTECTION_CLICK_ATTACK', true );
define('PHP_FIREWALL_PROTECTION_XSS_ATTACK', true );
define('PHP_FIREWALL_PROTECTION_COOKIES', false );
define('PHP_FIREWALL_PROTECTION_POST', false );
define('PHP_FIREWALL_PROTECTION_GET', false );
define('PHP_FIREWALL_PROTECTION_SERVER_OVH', true );
define('PHP_FIREWALL_PROTECTION_SERVER_KIMSUFI', true );
define('PHP_FIREWALL_PROTECTION_SERVER_DEDIBOX', true );
define('PHP_FIREWALL_PROTECTION_SERVER_DIGICUBE', true );
define('PHP_FIREWALL_PROTECTION_SERVER_OVH_BY_IP', true );
define('PHP_FIREWALL_PROTECTION_SERVER_KIMSUFI_BY_IP', true );
define('PHP_FIREWALL_PROTECTION_SERVER_DEDIBOX_BY_IP', true );
define('PHP_FIREWALL_PROTECTION_SERVER_DIGICUBE_BY_IP', true );
/** end configuration */
/** IPS PROTECTED */
if ( count( $IP_ALLOW ) > 0 ) {
if ( in_array( $_SERVER['REMOTE_ADDR'], $IP_ALLOW ) ) return;
}
/** END IPS PROTECTED */
/** LANGUAGE */
if ( PHP_FIREWALL_LANGUAGE === 'french' ) {
define('_PHPF_PROTECTION_DEDIBOX', 'Protection contre les serveurs DEDIBOX active, cette IP range n\'est pas autorisée !');
define('_PHPF_PROTECTION_DEDIBOX_IP', 'Protection contre les serveurs DEDIBOX active, cette IP range n\'est pas autorisée !');
define('_PHPF_PROTECTION_DIGICUBE', 'Protection contre les serveurs DIGICUBE active, this IP range is not allowed !');
define('_PHPF_PROTECTION_DIGICUBE_IP', 'Protection contre les serveurs DIGICUBE active, cette IP n\'est pas autorisée !');
define('_PHPF_PROTECTION_KIMSUFI', 'Protection contre les serveurs KIMSUFI active, cette IP range n\'est pas autorisée !');
define('_PHPF_PROTECTION_OVH', 'Protection contre les serveurs OVH active, cette IP range n\'est pas autorisée !');
define('_PHPF_PROTECTION_BOTS', 'Attaque Bot détectée ! stop it ...');
define('_PHPF_PROTECTION_CLICK', 'Click attaque détectée ! stop it .....');
define('_PHPF_PROTECTION_DOS', 'Invalide user agent ! Stop it ...');
define('_PHPF_PROTECTION_OTHER_SERVER', 'Poster depuis un autre serveur est interdit !');
define('_PHPF_PROTECTION_REQUEST', 'Méthode de requête interdite ! Stop it ...');
define('_PHPF_PROTECTION_SANTY', 'Attaque Santy detectée ! Stop it ...');
define('_PHPF_PROTECTION_SPAM', 'Protection SPAM IPs active, cette IP range n\'est pas autorisée !');
define('_PHPF_PROTECTION_SPAM_IP', 'Protection SPAM IPs active, cette IP range n\'est pas autorisée !');
define('_PHPF_PROTECTION_UNION', 'Attaque Union détectée ! stop it ......');
define('_PHPF_PROTECTION_URL', 'Protection url active, string non autorisée !');
define('_PHPF_PROTECTION_XSS', 'Attaque XSS détectée ! stop it ...');
} else {
define('_PHPF_PROTECTION_DEDIBOX', 'Protection DEDIBOX Server active, this IP range is not allowed !');
define('_PHPF_PROTECTION_DEDIBOX_IP', 'Protection DEDIBOX Server active, this IP is not allowed !');
define('_PHPF_PROTECTION_DIGICUBE', 'Protection DIGICUBE Server active, this IP range is not allowed !');
define('_PHPF_PROTECTION_DIGICUBE_IP', 'Protection DIGICUBE Server active, this IP is not allowed !');
define('_PHPF_PROTECTION_KIMSUFI', 'Protection KIMSUFI Server active, this IP range is not allowed !');
define('_PHPF_PROTECTION_OVH', 'Protection OVH Server active, this IP range is not allowed !');
define('_PHPF_PROTECTION_BOTS', 'Bot attack detected ! stop it ...');
define('_PHPF_PROTECTION_CLICK', 'Click attack detected ! stop it .....');
define('_PHPF_PROTECTION_DOS', 'Invalid user agent ! Stop it ...');
define('_PHPF_PROTECTION_OTHER_SERVER', 'Posting from another server not allowed !');
define('_PHPF_PROTECTION_REQUEST', 'Invalid request method check ! Stop it ...');
define('_PHPF_PROTECTION_SANTY', 'Attack Santy detected ! Stop it ...');
define('_PHPF_PROTECTION_SPAM', 'Protection SPAM IPs active, this IP range is not allowed !');
define('_PHPF_PROTECTION_SPAM_IP', 'Protection died IPs active, this IP range is not allowed !');
define('_PHPF_PROTECTION_UNION', 'Union attack detected ! stop it ......');
define('_PHPF_PROTECTION_URL', 'Protection url active, string not allowed !');
define('_PHPF_PROTECTION_XSS', 'XSS attack detected ! stop it ...');
}
/** END LANGUAGE*/
if ( PHP_FIREWALL_ACTIVATION === true ) {
FUNCTION PHP_FIREWALL_unset_globals() {
if ( ini_get('register_globals') ) {
$allow = array('_ENV' => 1, '_GET' => 1, '_POST' => 1, '_COOKIE' => 1, '_FILES' => 1, '_SERVER' => 1, '_REQUEST' => 1, 'GLOBALS' => 1);
foreach ($GLOBALS as $key => $value) {
if ( ! isset( $allow[$key] ) ) unset( $GLOBALS[$key] );
}
}
}
if ( PHP_FIREWALL_PROTECTION_UNSET_GLOBALS === true ) PHP_FIREWALL_unset_globals();
/** fonctions de base */
FUNCTION PHP_FIREWALL_get_env($st_var) {
global $HTTP_SERVER_VARS;
if(isset($_SERVER[$st_var])) {
return strip_tags( $_SERVER[$st_var] );
} elseif(isset($_ENV[$st_var])) {
return strip_tags( $_ENV[$st_var] );
} elseif(isset($HTTP_SERVER_VARS[$st_var])) {
return strip_tags( $HTTP_SERVER_VARS[$st_var] );
} elseif(getenv($st_var)) {
return strip_tags( getenv($st_var) );
} elseif(function_exists('apache_getenv') && apache_getenv($st_var, true)) {
return strip_tags( apache_getenv($st_var, true) );
}
return '';
}
FUNCTION PHP_FIREWALL_get_referer() {
if( PHP_FIREWALL_get_env('HTTP_REFERER') )
return PHP_FIREWALL_get_env('HTTP_REFERER');
return 'no referer';
}
FUNCTION PHP_FIREWALL_get_ip() {
if ( PHP_FIREWALL_get_env('HTTP_X_FORWARDED_FOR') ) {
return PHP_FIREWALL_get_env('HTTP_X_FORWARDED_FOR');
} elseif ( PHP_FIREWALL_get_env('HTTP_CLIENT_IP') ) {
return PHP_FIREWALL_get_env('HTTP_CLIENT_IP');
} else {
return PHP_FIREWALL_get_env('REMOTE_ADDR');
}
}
FUNCTION PHP_FIREWALL_get_user_agent() {
if(PHP_FIREWALL_get_env('HTTP_USER_AGENT'))
return PHP_FIREWALL_get_env('HTTP_USER_AGENT');
return 'none';
}
FUNCTION PHP_FIREWALL_get_query_string() {
if( PHP_FIREWALL_get_env('QUERY_STRING') )
return str_replace('%09', '%20', PHP_FIREWALL_get_env('QUERY_STRING'));
return '';
}
FUNCTION PHP_FIREWALL_get_request_method() {
if(PHP_FIREWALL_get_env('REQUEST_METHOD'))
return PHP_FIREWALL_get_env('REQUEST_METHOD');
return 'none';
}
FUNCTION PHP_FIREWALL_gethostbyaddr() {
if ( PHP_FIREWALL_PROTECTION_SERVER_OVH === true OR PHP_FIREWALL_PROTECTION_SERVER_KIMSUFI === true OR PHP_FIREWALL_PROTECTION_SERVER_DEDIBOX === true OR PHP_FIREWALL_PROTECTION_SERVER_DIGICUBE === true ) {
if ( # empty( $_SESSION['PHP_FIREWALL_gethostbyaddr'] ) ) {
return $_SESSION['PHP_FIREWALL_gethostbyaddr'] = #gethostbyaddr( PHP_FIREWALL_get_ip() );
} else {
return strip_tags( $_SESSION['PHP_FIREWALL_gethostbyaddr'] );
}
}
}
/** bases define */
define('PHP_FIREWALL_GET_QUERY_STRING', strtolower( PHP_FIREWALL_get_query_string() ) );
define('PHP_FIREWALL_USER_AGENT', PHP_FIREWALL_get_user_agent() );
define('PHP_FIREWALL_GET_IP', PHP_FIREWALL_get_ip() );
define('PHP_FIREWALL_GET_HOST', PHP_FIREWALL_gethostbyaddr() );
define('PHP_FIREWALL_GET_REFERER', PHP_FIREWALL_get_referer() );
define('PHP_FIREWALL_GET_REQUEST_METHOD', PHP_FIREWALL_get_request_method() );
define('PHP_FIREWALL_REGEX_UNION','#\w?\s?union\s\w*?\s?(select|all|distinct|insert|update|drop|delete)#is');
FUNCTION PHP_FIREWALL_push_email( $subject, $msg ) {
$headers = "From: PHP Firewall: ".PHP_FIREWALL_ADMIN_MAIL." <".PHP_FIREWALL_ADMIN_MAIL.">\r\n"
."Reply-To: ".PHP_FIREWALL_ADMIN_MAIL."\r\n"
."Priority: urgent\r\n"
."Importance: High\r\n"
."Precedence: special-delivery\r\n"
."Organization: PHP Firewall\r\n"
."MIME-Version: 1.0\r\n"
."Content-Type: text/plain\r\n"
."Content-Transfer-Encoding: 8bit\r\n"
."X-Priority: 1\r\n"
."X-MSMail-Priority: High\r\n"
."X-Mailer: PHP/" . phpversion() ."\r\n"
."X-PHPFirewall: 1.0 by PHPFirewall\r\n"
."Date:" . date("D, d M Y H:s:i") . " +0100\n";
if ( PHP_FIREWALL_ADMIN_MAIL != '' )
#mail( PHP_FIREWALL_ADMIN_MAIL, $subject, $msg, $headers );
}
FUNCTION PHP_FIREWALL_LOGS( $type ) {
$f = fopen( dirname(__FILE__).'/'.PHP_FIREWALL_LOG_FILE.'.txt', 'a');
$msg = date('j-m-Y H:i:s')." | $type | IP: ".PHP_FIREWALL_GET_IP." ] | DNS: ".gethostbyaddr(PHP_FIREWALL_GET_IP)." | Agent: ".PHP_FIREWALL_USER_AGENT." | URL: ".PHP_FIREWALL_REQUEST_URI." | Referer: ".PHP_FIREWALL_GET_REFERER."\n\n";
fputs($f, $msg);
fclose($f);
if ( PHP_FIREWALL_PUSH_MAIL === true ) {
PHP_FIREWALL_push_email( 'Alert PHP Firewall '.strip_tags( $_SERVER['SERVER_NAME'] ) , "PHP Firewall logs of ".strip_tags( $_SERVER['SERVER_NAME'] )."\n".str_replace('|', "\n", $msg ) );
}
}
if ( PHP_FIREWALL_PROTECTION_SERVER_OVH === true ) {
if ( stristr( PHP_FIREWALL_GET_HOST ,'ovh') ) {
PHP_FIREWALL_LOGS( 'OVH Server list' );
die( _PHPF_PROTECTION_OVH );
}
}
if ( PHP_FIREWALL_PROTECTION_SERVER_OVH_BY_IP === true ) {
$ip = explode('.', PHP_FIREWALL_GET_IP );
if ( $ip[0].'.'.$ip[1] == '87.98' or $ip[0].'.'.$ip[1] == '91.121' or $ip[0].'.'.$ip[1] == '94.23' or $ip[0].'.'.$ip[1] == '213.186' or $ip[0].'.'.$ip[1] == '213.251' ) {
PHP_FIREWALL_LOGS( 'OVH Server IP' );
die( _PHPF_PROTECTION_OVH );
}
}
if ( PHP_FIREWALL_PROTECTION_SERVER_KIMSUFI === true ) {
if ( stristr( PHP_FIREWALL_GET_HOST ,'kimsufi') ) {
PHP_FIREWALL_LOGS( 'KIMSUFI Server list' );
die( _PHPF_PROTECTION_KIMSUFI );
}
}
if ( PHP_FIREWALL_PROTECTION_SERVER_DEDIBOX === true ) {
if ( stristr( PHP_FIREWALL_GET_HOST ,'dedibox') ) {
PHP_FIREWALL_LOGS( 'DEDIBOX Server list' );
die( _PHPF_PROTECTION_DEDIBOX );
}
}
if ( PHP_FIREWALL_PROTECTION_SERVER_DEDIBOX_BY_IP === true ) {
$ip = explode('.', PHP_FIREWALL_GET_IP );
if ( $ip[0].'.'.$ip[1] == '88.191' ) {
PHP_FIREWALL_LOGS( 'DEDIBOX Server IP' );
die( _PHPF_PROTECTION_DEDIBOX_IP );
}
}
if ( PHP_FIREWALL_PROTECTION_SERVER_DIGICUBE === true ) {
if ( stristr( PHP_FIREWALL_GET_HOST ,'digicube') ) {
PHP_FIREWALL_LOGS( 'DIGICUBE Server list' );
die( _PHPF_PROTECTION_DIGICUBE );
}
}
if ( PHP_FIREWALL_PROTECTION_SERVER_DIGICUBE_BY_IP === true ) {
$ip = explode('.', PHP_FIREWALL_GET_IP );
if ( $ip[0].'.'.$ip[1] == '95.130' ) {
PHP_FIREWALL_LOGS( 'DIGICUBE Server IP' );
die( _PHPF_PROTECTION_DIGICUBE_IP );
}
}
if ( PHP_FIREWALL_PROTECTION_RANGE_IP_SPAM === true ) {
$ip_array = array('24', '186', '189', '190', '200', '201', '202', '209', '212', '213', '217', '222' );
$range_ip = explode('.', PHP_FIREWALL_GET_IP );
if ( in_array( $range_ip[0], $ip_array ) ) {
PHP_FIREWALL_LOGS( 'IPs Spam list' );
die( _PHPF_PROTECTION_SPAM );
}
}
if ( PHP_FIREWALL_PROTECTION_RANGE_IP_DENY === true ) {
$ip_array = array('0', '1', '2', '5', '10', '14', '23', '27', '31', '36', '37', '39', '42', '46', '49', '50', '100', '101', '102', '103', '104', '105', '106', '107', '114', '172', '176', '177', '179', '181', '185', '192', '223', '224' );
$range_ip = explode('.', PHP_FIREWALL_GET_IP );
if ( in_array( $range_ip[0], $ip_array ) ) {
PHP_FIREWALL_LOGS( 'IPs reserved list' );
die( _PHPF_PROTECTION_SPAM_IP );
}
}
if ( PHP_FIREWALL_PROTECTION_COOKIES === true ) {
$ct_rules = Array('applet', 'base', 'bgsound', 'blink', 'embed', 'expression', 'frame', 'javascript', 'layer', 'link', 'meta', 'object', 'onabort', 'onactivate', 'onafterprint', 'onafterupdate', 'onbeforeactivate', 'onbeforecopy', 'onbeforecut', 'onbeforedeactivate', 'onbeforeeditfocus', 'onbeforepaste', 'onbeforeprint', 'onbeforeunload', 'onbeforeupdate', 'onblur', 'onbounce', 'oncellchange', 'onchange', 'onclick', 'oncontextmenu', 'oncontrolselect', 'oncopy', 'oncut', 'ondataavailable', 'ondatasetchanged', 'ondatasetcomplete', 'ondblclick', 'ondeactivate', 'ondrag', 'ondragend', 'ondragenter', 'ondragleave', 'ondragover', 'ondragstart', 'ondrop', 'onerror', 'onerrorupdate', 'onfilterchange', 'onfinish', 'onfocus', 'onfocusin', 'onfocusout', 'onhelp', 'onkeydown', 'onkeypress', 'onkeyup', 'onlayoutcomplete', 'onload', 'onlosecapture', 'onmousedown', 'onmouseenter', 'onmouseleave', 'onmousemove', 'onmouseout', 'onmouseover', 'onmouseup', 'onmousewheel', 'onmove', 'onmoveend', 'onmovestart', 'onpaste', 'onpropertychange', 'onreadystatechange', 'onreset', 'onresize', 'onresizeend', 'onresizestart', 'onrowenter', 'onrowexit', 'onrowsdelete', 'onrowsinserted', 'onscroll', 'onselect', 'onselectionchange', 'onselectstart', 'onstart', 'onstop', 'onsubmit', 'onunload', 'script', 'style', 'title', 'vbscript', 'xml');
if ( PHP_FIREWALL_PROTECTION_COOKIES === true ) {
foreach($_COOKIE as $value) {
$check = str_replace($ct_rules, '*', $value);
if( $value != $check ) {
PHP_FIREWALL_LOGS( 'Cookie protect' );
unset( $value );
}
}
}
if ( PHP_FIREWALL_PROTECTION_POST === true ) {
foreach( $_POST as $value ) {
$check = str_replace($ct_rules, '*', $value);
if( $value != $check ) {
PHP_FIREWALL_LOGS( 'POST protect' );
unset( $value );
}
}
}
if ( PHP_FIREWALL_PROTECTION_GET === true ) {
foreach( $_GET as $value ) {
$check = str_replace($ct_rules, '*', $value);
if( $value != $check ) {
PHP_FIREWALL_LOGS( 'GET protect' );
unset( $value );
}
}
}
}
/** protection de l'url */
if ( PHP_FIREWALL_PROTECTION_URL === true ) {
$ct_rules = array( 'absolute_path', 'ad_click', 'alert(', 'alert%20', ' and ', 'basepath', 'bash_history', '.bash_history', 'cgi-', 'chmod(', 'chmod%20', '%20chmod', 'chmod=', 'chown%20', 'chgrp%20', 'chown(', '/chown', 'chgrp(', 'chr(', 'chr=', 'chr%20', '%20chr', 'chunked', 'cookie=', 'cmd', 'cmd=', '%20cmd', 'cmd%20', '.conf', 'configdir', 'config.php', 'cp%20', '%20cp', 'cp(', 'diff%20', 'dat?', 'db_mysql.inc', 'document.location', 'document.cookie', 'drop%20', 'echr(', '%20echr', 'echr%20', 'echr=', '}else{', '.eml', 'esystem(', 'esystem%20', '.exe', 'exploit', 'file\://', 'fopen', 'fwrite', '~ftp', 'ftp:', 'ftp.exe', 'getenv', '%20getenv', 'getenv%20', 'getenv(', 'grep%20', '_global', 'global_', 'global[', 'http:', '_globals', 'globals_', 'globals[', 'grep(', 'g\+\+', 'halt%20', '.history', '?hl=', '.htpasswd', 'http_', 'http-equiv', 'http/1.', 'http_php', 'http_user_agent', 'http_host', '&icq', 'if{', 'if%20{', 'img src', 'img%20src', '.inc.php', '.inc', 'insert%20into', 'ISO-8859-1', 'ISO-', 'javascript\://', '.jsp', '.js', 'kill%20', 'kill(', 'killall', '%20like', 'like%20', 'locate%20', 'locate(', 'lsof%20', 'mdir%20', '%20mdir', 'mdir(', 'mcd%20', 'motd%20', 'mrd%20', 'rm%20', '%20mcd', '%20mrd', 'mcd(', 'mrd(', 'mcd=', 'mod_gzip_status', 'modules/', 'mrd=', 'mv%20', 'nc.exe', 'new_password', 'nigga(', '%20nigga', 'nigga%20', '~nobody', 'org.apache', '+outfile+', '%20outfile%20', '*/outfile/*',' outfile ','outfile', 'password=', 'passwd%20', '%20passwd', 'passwd(', 'phpadmin', 'perl%20', '/perl', 'phpbb_root_path','*/phpbb_root_path/*','p0hh', 'ping%20', '.pl', 'powerdown%20', 'rm(', '%20rm', 'rmdir%20', 'mv(', 'rmdir(', 'phpinfo()', '<?php', 'reboot%20', '/robot.txt' , '~root', 'root_path', 'rush=', '%20and%20', '%20xorg%20', '%20rush', 'rush%20', 'secure_site, ok', 'select%20', 'select from', 'select%20from', '_server', 'server_', 'server[', 'server-info', 'server-status', 'servlet', 'sql=', '<script', '<script>', '</script','script>','/script', 'switch{','switch%20{', '.system', 'system(', 'telnet%20', 'traceroute%20', '.txt', 'union%20', '%20union', 'union(', 'union=', 'vi(', 'vi%20', 'wget', 'wget%20', '%20wget', 'wget(', 'window.open', 'wwwacl', ' xor ', 'xp_enumdsn', 'xp_availablemedia', 'xp_filelist', 'xp_cmdshell', '$_request', '$_get', '$request', '$get', '&aim', '/etc/password','/etc/shadow', '/etc/groups', '/etc/gshadow', '/bin/ps', 'uname\x20-a', '/usr/bin/id', '/bin/echo', '/bin/kill', '/bin/', '/chgrp', '/usr/bin', 'bin/python', 'bin/tclsh', 'bin/nasm', '/usr/x11r6/bin/xterm', '/bin/mail', '/etc/passwd', '/home/ftp', '/home/www', '/servlet/con', '?>', '.txt');
$check = str_replace($ct_rules, '*', PHP_FIREWALL_GET_QUERY_STRING );
if( PHP_FIREWALL_GET_QUERY_STRING != $check ) {
PHP_FIREWALL_LOGS( 'URL protect' );
die( _PHPF_PROTECTION_URL );
}
}
/** Posting from other servers in not allowed */
if ( PHP_FIREWALL_PROTECTION_REQUEST_SERVER === true ) {
if ( PHP_FIREWALL_GET_REQUEST_METHOD == 'POST' ) {
if (isset($_SERVER['HTTP_REFERER'])) {
if ( ! stripos( $_SERVER['HTTP_REFERER'], $_SERVER['HTTP_HOST'], 0 ) ) {
PHP_FIREWALL_LOGS( 'Posting another server' );
die( _PHPF_PROTECTION_OTHER_SERVER );
}
}
}
}
/** protection contre le vers santy */
if ( PHP_FIREWALL_PROTECTION_SANTY === true ) {
$ct_rules = array('rush','highlight=%','perl','chr(','pillar','visualcoder','sess_');
$check = str_replace($ct_rules, '*', strtolower(PHP_FIREWALL_REQUEST_URI) );
if( strtolower(PHP_FIREWALL_REQUEST_URI) != $check ) {
PHP_FIREWALL_LOGS( 'Santy' );
die( _PHPF_PROTECTION_SANTY );
}
}
/** protection bots */
if ( PHP_FIREWALL_PROTECTION_BOTS === true ) {
$ct_rules = array( '#nonymouse', 'addresses.com', 'ideography.co.uk', 'adsarobot', 'ah-ha', 'aktuelles', 'alexibot', 'almaden', 'amzn_assoc', 'anarchie', 'art-online', 'aspseek', 'assort', 'asterias', 'attach', 'atomz', 'atspider', 'autoemailspider', 'backweb', 'backdoorbot', 'bandit', 'batchftp', 'bdfetch', 'big.brother', 'black.hole', 'blackwidow', 'blowfish', 'bmclient', 'boston project', 'botalot', 'bravobrian', 'buddy', 'bullseye', 'bumblebee ', 'builtbottough', 'bunnyslippers', 'capture', 'cegbfeieh', 'cherrypicker', 'cheesebot', 'chinaclaw', 'cicc', 'civa', 'clipping', 'collage', 'collector', 'copyrightcheck', 'cosmos', 'crescent', 'custo', 'cyberalert', 'deweb', 'diagem', 'digger', 'digimarc', 'diibot', 'directupdate', 'disco', 'dittospyder', 'download accelerator', 'download demon', 'download wonder', 'downloader', 'drip', 'dsurf', 'dts agent', 'dts.agent', 'easydl', 'ecatch', 'echo extense', 'efp#gmx.net', 'eirgrabber', 'elitesys', 'emailsiphon', 'emailwolf', 'envidiosos', 'erocrawler', 'esirover', 'express webpictures', 'extrac', 'eyenetie', 'fastlwspider', 'favorg', 'favorites sweeper', 'fezhead', 'filehound', 'filepack.superbr.org', 'flashget', 'flickbot', 'fluffy', 'frontpage', 'foobot', 'galaxyBot', 'generic', 'getbot ', 'getleft', 'getright', 'getsmart', 'geturl', 'getweb', 'gigabaz', 'girafabot', 'go-ahead-got-it', 'go!zilla', 'gornker', 'grabber', 'grabnet', 'grafula', 'green research', 'harvest', 'havindex', 'hhjhj#yahoo', 'hloader', 'hmview', 'homepagesearch', 'htmlparser', 'hulud', 'http agent', 'httpconnect', 'httpdown', 'http generic', 'httplib', 'httrack', 'humanlinks', 'ia_archiver', 'iaea', 'ibm_planetwide', 'image stripper', 'image sucker', 'imagefetch', 'incywincy', 'indy', 'infonavirobot', 'informant', 'interget', 'internet explore', 'infospiders', 'internet ninja', 'internetlinkagent', 'interneteseer.com', 'ipiumbot', 'iria', 'irvine', 'jbh', 'jeeves', 'jennybot', 'jetcar', 'joc web spider', 'jpeg hunt', 'justview', 'kapere', 'kdd explorer', 'kenjin.spider', 'keyword.density', 'kwebget', 'lachesis', 'larbin', 'laurion(dot)com', 'leechftp', 'lexibot', 'lftp', 'libweb', 'links aromatized', 'linkscan', 'link*sleuth', 'linkwalker', 'libwww', 'lightningdownload', 'likse', 'lwp','mac finder', 'mag-net', 'magnet', 'marcopolo', 'mass', 'mata.hari', 'mcspider', 'memoweb', 'microsoft url control', 'microsoft.url', 'midown', 'miixpc', 'minibot', 'mirror', 'missigua', 'mister.pix', 'mmmtocrawl', 'moget', 'mozilla/2', 'mozilla/3.mozilla/2.01', 'mozilla.*newt', 'multithreaddb', 'munky', 'msproxy', 'nationaldirectory', 'naverrobot', 'navroad', 'nearsite', 'netants', 'netcarta', 'netcraft', 'netfactual', 'netmechanic', 'netprospector', 'netresearchserver', 'netspider', 'net vampire', 'newt', 'netzip', 'nicerspro', 'npbot', 'octopus', 'offline.explorer', 'offline explorer', 'offline navigator', 'opaL', 'openfind', 'opentextsitecrawler', 'orangebot', 'packrat', 'papa foto', 'pagegrabber', 'pavuk', 'pbwf', 'pcbrowser', 'personapilot', 'pingalink', 'pockey', 'program shareware', 'propowerbot/2.14', 'prowebwalker', 'proxy', 'psbot', 'psurf', 'puf', 'pushsite', 'pump', 'qrva', 'quepasacreep', 'queryn.metasearch', 'realdownload', 'reaper', 'recorder', 'reget', 'replacer', 'repomonkey', 'rma', 'robozilla', 'rover', 'rpt-httpclient', 'rsync', 'rush=', 'searchexpress', 'searchhippo', 'searchterms.it', 'second street research', 'seeker', 'shai', 'sitecheck', 'sitemapper', 'sitesnagger', 'slysearch', 'smartdownload', 'snagger', 'spacebison', 'spankbot', 'spanner', 'spegla', 'spiderbot', 'spiderengine', 'sqworm', 'ssearcher100', 'star downloader', 'stripper', 'sucker', 'superbot', 'surfwalker', 'superhttp', 'surfbot', 'surveybot', 'suzuran', 'sweeper', 'szukacz/1.4', 'tarspider', 'takeout', 'teleport', 'telesoft', 'templeton', 'the.intraformant', 'thenomad', 'tighttwatbot', 'titan', 'tocrawl/urldispatcher','toolpak', 'traffixer', 'true_robot', 'turingos', 'turnitinbot', 'tv33_mercator', 'uiowacrawler', 'urldispatcherlll', 'url_spider_pro', 'urly.warning ', 'utilmind', 'vacuum', 'vagabondo', 'vayala', 'vci', 'visualcoders', 'visibilitygap', 'vobsub', 'voideye', 'vspider', 'w3mir', 'webauto', 'webbandit', 'web.by.mail', 'webcapture', 'webcatcher', 'webclipping', 'webcollage', 'webcopier', 'webcopy', 'webcraft#bea', 'web data extractor', 'webdav', 'webdevil', 'webdownloader', 'webdup', 'webenhancer', 'webfetch', 'webgo', 'webhook', 'web.image.collector', 'web image collector', 'webinator', 'webleacher', 'webmasters', 'webmasterworldforumbot', 'webminer', 'webmirror', 'webmole', 'webreaper', 'websauger', 'websaver', 'website.quester', 'website quester', 'websnake', 'websucker', 'web sucker', 'webster', 'webreaper', 'webstripper', 'webvac', 'webwalk', 'webweasel', 'webzip', 'wget', 'widow', 'wisebot', 'whizbang', 'whostalking', 'wonder', 'wumpus', 'wweb', 'www-collector-e', 'wwwoffle', 'wysigot', 'xaldon', 'xenu', 'xget', 'x-tractor', 'zeus' );
$check = str_replace($ct_rules, '*', strtolower(PHP_FIREWALL_USER_AGENT) );
if( strtolower(PHP_FIREWALL_USER_AGENT) != $check ) {
PHP_FIREWALL_LOGS( 'Bots attack' );
die( _PHPF_PROTECTION_BOTS );
}
}
/** Invalid request method check */
if ( PHP_FIREWALL_PROTECTION_REQUEST_METHOD === true ) {
if(strtolower(PHP_FIREWALL_GET_REQUEST_METHOD)!='get' AND strtolower(PHP_FIREWALL_GET_REQUEST_METHOD)!='head' AND strtolower(PHP_FIREWALL_GET_REQUEST_METHOD)!='post' AND strtolower(PHP_FIREWALL_GET_REQUEST_METHOD)!='put') {
PHP_FIREWALL_LOGS( 'Invalid request' );
die( _PHPF_PROTECTION_REQUEST );
}
}
/** protection dos attaque */
if ( PHP_FIREWALL_PROTECTION_DOS === true ) {
if ( !defined('PHP_FIREWALL_USER_AGENT') || PHP_FIREWALL_USER_AGENT == '-' ) {
PHP_FIREWALL_LOGS( 'Dos attack' );
die( _PHPF_PROTECTION_DOS );
}
}
/** protection union sql attaque */
if ( PHP_FIREWALL_PROTECTION_UNION_SQL === true ) {
$stop = 0;
$ct_rules = array( '*/from/*', '*/insert/*', '+into+', '%20into%20', '*/into/*', ' into ', 'into', '*/limit/*', 'not123exists*', '*/radminsuper/*', '*/select/*', '+select+', '%20select%20', ' select ', '+union+', '%20union%20', '*/union/*', ' union ', '*/update/*', '*/where/*' );
$check = str_replace($ct_rules, '*', PHP_FIREWALL_GET_QUERY_STRING );
if( PHP_FIREWALL_GET_QUERY_STRING != $check ) $stop++;
if (preg_match(PHP_FIREWALL_REGEX_UNION, PHP_FIREWALL_GET_QUERY_STRING)) $stop++;
if (preg_match('/([OdWo5NIbpuU4V2iJT0n]{5}) /', rawurldecode( PHP_FIREWALL_GET_QUERY_STRING ))) $stop++;
if (strstr(rawurldecode( PHP_FIREWALL_GET_QUERY_STRING ) ,'*')) $stop++;
if ( !empty( $stop ) ) {
PHP_FIREWALL_LOGS( 'Union attack' );
die( _PHPF_PROTECTION_UNION );
}
}
/** protection click attack */
if ( PHP_FIREWALL_PROTECTION_CLICK_ATTACK === true ) {
$ct_rules = array( '/*', 'c2nyaxb0', '/*' );
if( PHP_FIREWALL_GET_QUERY_STRING != str_replace($ct_rules, '*', PHP_FIREWALL_GET_QUERY_STRING ) ) {
PHP_FIREWALL_LOGS( 'Click attack' );
die( _PHPF_PROTECTION_CLICK );
}
}
/** protection XSS attack */
if ( PHP_FIREWALL_PROTECTION_XSS_ATTACK === true ) {
$ct_rules = array( 'http\:\/\/', 'https\:\/\/', 'cmd=', '&cmd', 'exec', 'concat', './', '../', 'http:', 'h%20ttp:', 'ht%20tp:', 'htt%20p:', 'http%20:', 'https:', 'h%20ttps:', 'ht%20tps:', 'htt%20ps:', 'http%20s:', 'https%20:', 'ftp:', 'f%20tp:', 'ft%20p:', 'ftp%20:', 'ftps:', 'f%20tps:', 'ft%20ps:', 'ftp%20s:', 'ftps%20:', '.php?url=' );
$check = str_replace($ct_rules, '*', PHP_FIREWALL_GET_QUERY_STRING );
if( PHP_FIREWALL_GET_QUERY_STRING != $check ) {
PHP_FIREWALL_LOGS( 'XSS attack' );
die( _PHPF_PROTECTION_XSS );
}
}
}

Related

Receiving error within Smartthings Device Handler

I am receiving the below error when trying to compile the device_handler code for the Sylvania Smart+ Plug. The code comes from https://images-na.ssl-images-amazon.com/images/I/71PrgM-PamL.pdf
The error:
Org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: script_dth_metadata_0631e407_ffd8_4ceb_b49a_877fd47635df: 94: expecting ''', found '\r' # line 94, column 55. nalResult.value == "on" ? '{{ ^ 1 error
Line 94:
def descriptionText = finalResult.value == "on" ? '{{
metadata {
definition (name: "SYLVANIA Smart Plug", namespace: "ledvanceDH", author:
"Ledvance") {
capability "Actuator"
capability "Switch"
capability "Power Meter"
capability "Configuration"
capability "Refresh"
capability "Sensor"
capability "Health Check"
fingerprint profileId: "C05E", inClusters:
"1000,0000,0003,0004,0005,0006,0B04,FC0F", outClusters: "0019", manufacturer: "OSRAM",
model: "Plug 01", deviceJoinName: "SYLVANIA Smart Plug"
fingerprint profileId: "0104", inClusters:
"0000,0003,0004,0005,0006,0B05,FC01,FC08", outClusters: "0003,0019", manufacturer:
"LEDVACE", model: "PLUG", deviceJoinName: "SYLVANIA Smart Plug"
}
// simulator metadata
simulator {
// status messages
status "on": "on/off: 1"
status "off": "on/off: 0"
// reply messages
reply "zcl on-off on": "on/off: 1"
reply "zcl on-off off": "on/off: 0"
}
preferences {
section {
image(name: 'educationalcontent', multiple: true, images: [
"http://cdn.devicegse.smartthings.com/Outlet/US/OutletUS1.jpg",
"http://cdn.devicegse.smartthings.com/Outlet/US/OutletUS2.jpg"
])
}
}
// UI tile definitions
tiles(scale: 2) {
multiAttributeTile(name:"switch", type: "lighting", width: 6, height: 4,
canChangeIcon: true){
tileAttribute ("device.switch", key: "PRIMARY_CONTROL") {
attributeState "on", label: 'On', action: "switch.off",
icon: "st.Appliances.appliances17", backgroundColor: "#79b821", nextState: "turningOff"
attributeState "off", label: 'Off', action: "switch.on",
icon: "st.Appliances.appliances17", backgroundColor: "#565C51", nextState: "turningOn"
attributeState "turningOn", label: 'Turning On', action:
"switch.off", icon: "st.Appliances.appliances17", backgroundColor: "#60903A", nextState:
"turningOff"
attributeState "turningOff", label: 'Turning Off', action:
"switch.on", icon: "st.Appliances.appliances17", backgroundColor: "#CACACA", nextState:
"turningOn"
}
tileAttribute ("power", key: "SECONDARY_CONTROL") {
attributeState "power", label:'${currentValue} W'
}
}
standardTile("refresh", "device.power", inactiveLabel: false, decoration:
"flat", width: 2, height: 2) {
state "default", label:'', action:"refresh.refresh",
icon:"st.secondary.refresh"
}
main "switch"
details(["switch","refresh"])
}
}
// Parse incoming device messages to generate events
def parse(String description) {
log.debug "description is $description"
def finalResult = zigbee.getKnownDescription(description)
def event = [:]
//TODO: Remove this after getKnownDescription can parse it automatically
if (!finalResult && description!="updated")
finalResult =
getPowerDescription(zigbee.parseDescriptionAsMap(description))
if (finalResult) {
log.info "final result = $finalResult"
if (finalResult.type == "update") {
log.info "$device updates: ${finalResult.value}"
event = null
}
else if (finalResult.type == "power") {
def powerValue = (finalResult.value as Integer)/10
event = createEvent(name: "power", value: powerValue,
descriptionText: '{{ device.displayName }} power is {{ value }} Watts', translatable:
true)
/*
Dividing by 10 as the Divisor is 10000 and unit is kW for
the device. AttrId: 0302 and 0300. Simplifying to 10
power level is an integer. The exact power level with
correct units needs to be handled in the device type
to account for the different Divisor value (AttrId: 0302)
and POWER Unit (AttrId: 0300). CLUSTER for simple metering is 0702
*/
}
else {
def descriptionText = finalResult.value == "on" ? '{{
device.displayName }} is On' : '{{ device.displayName }} is Off'
event = createEvent(name: finalResult.type, value:
finalResult.value, descriptionText: descriptionText, translatable: true)
}
}
else {
def cluster = zigbee.parse(description)
if (cluster && cluster.clusterId == 0x0006 && cluster.command == 0x07){
if (cluster.data[0] == 0x00) {
log.debug "ON/OFF REPORTING CONFIG RESPONSE: " + cluster
event = createEvent(name: "checkInterval", value: 60 * 12,
displayed: false, data: [protocol: "zigbee", hubHardwareId: device.hub.hardwareID])
}
else {
log.warn "ON/OFF REPORTING CONFIG FAILED- error
code:${cluster.data[0]}"
event = null
}
}
else {
log.warn "DID NOT PARSE MESSAGE for description : $description"
log.debug "${cluster}"
}
}
return event
}
def off() {
zigbee.off()
}
def on() {
zigbee.on()
}
/**
* PING is used by Device-Watch in attempt to reach the Device
* */
def ping() {
return zigbee.onOffRefresh()
}
def refresh() {
zigbee.onOffRefresh() + zigbee.electricMeasurementPowerRefresh()
}
def configure() {
// Device-Watch allows 2 check-in misses from device + ping (plus 1 min lag time)
// enrolls with default periodic reporting until newer 5 min interval is confirmed
sendEvent(name: "checkInterval", value: 2 * 10 * 60 + 1 * 60, displayed: false,
data: [protocol: "zigbee", hubHardwareId: device.hub.hardwareID])
// OnOff minReportTime 0 seconds, maxReportTime 5 min. Reporting interval if no
activity
refresh() + zigbee.onOffConfig(0, 300) + powerConfig()
}
//power config for devices with min reporting interval as 1 seconds and reporting
interval if no activity as 10min (600s)
//min change in value is 01
def powerConfig() {
[
"zdo bind 0x${device.deviceNetworkId} 1 ${endpointId} 0x0B04
{${device.zigbeeId}} {}", "delay 2000",
"zcl global send-me-a-report 0x0B04 0x050B 0x29 1 600 {05 00}",
//The send-me-a-report is custom to the attribute type for CentraLite
"delay 200",
"send 0x${device.deviceNetworkId} 1 ${endpointId}", "delay 2000"
]
}
private getEndpointId() {
new BigInteger(device.endpointId, 16).toString()
}
//TODO: Remove this after getKnownDescription can parse it automatically
def getPowerDescription(descMap) {
def powerValue = "undefined"
if (descMap.cluster == "0B04") {
if (descMap.attrId == "050b") {
if(descMap.value!="ffff")
powerValue = zigbee.convertHexToInt(descMap.value)
}
}
else if (descMap.clusterId == "0B04") {
if(descMap.command=="07"){
return [type: "update", value : "power (0B04) capability configured
successfully"]
}
}
if (powerValue != "undefined"){
return [type: "power", value : powerValue]
}
else {
return [:]
}
}
I should have thought of it sooner... The problem is with the copy/paste. The long lines were broken by a "return". I removed the returns and the script compiled.

Translator Text API | Microsoft Azure | Always ERROR 401000

I tried to test Microsoft Translator API Text v3.0 but failed with 401 Access denied. I do standard cURL requests (HTTP POST) using PHP 7.3.
$key = "************************"; // secret key here (from the Azure Portal)
$host = "https://api.cognitive.microsofttranslator.com";
$path = "/translate?api-version=3.0";
$params = "&to=en&from=bg";
$text = "За мен лично хора, които задават такива въпроси са несъобразителни.";
$requestBody = array(
array(
'Text' => $text,
),
);
$content = json_encode($requestBody);
if (!function_exists('com_create_guid')) {
function com_create_guid()
{
return sprintf('%04x%04x-%04x-%04x-%04x-%04x%04x%04x',
mt_rand(0, 0xffff), mt_rand(0, 0xffff),
mt_rand(0, 0xffff),
mt_rand(0, 0x0fff) | 0x4000,
mt_rand(0, 0x3fff) | 0x8000,
mt_rand(0, 0xffff), mt_rand(0, 0xffff), mt_rand(0, 0xffff)
);
}
}
$curl_headers = array(
'Content-type: application/json',
'Content-length: ' . strlen($content),
'Ocp-Apim-Subscription-Key: ' . $key,
'X-ClientTraceId: ' . com_create_guid()
);
$url = $host . $path . $params;
$ch = curl_init();
$curl_content = array('content', $content);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, $curl_headers);
//curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
//curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_POSTFIELDS, $content);
// Receive server response ...
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
//curl_setopt($ch, CURLINFO_HEADER_OUT, TRUE);
$result = curl_exec($ch);
//dd(curl_getinfo($ch, CURLINFO_HEADER_OUT));
if (curl_exec($ch) === false) {
dd(curl_error($ch));
}
Laravel 5.6, the code is placed on api.php route file.
Response Code:
"{"error":{"code":401000,"message":"The request is not authorized because credentials are missing or invalid."}}"
Where is the mistake? Should I enable any settings on the developer portal and so on?
Maybe this can help. In the official documentation you can find two examples of CURL requests. The difference between the two is that the second one has also the parameter region. Only the second request works in my case. Maybe the region parameter is essential.

Jenkinsfile with email in loop fails on first iteration

Latest Jenkins used.
edit: pastebin of full Java exception - https://pastebin.com/zZDNj18E
Goal: loop through all nodes, check for offline, email alert for each offline node.
(tried emailext alerts as well, could not use "offline")
Failed: My jenkinsfile runs perfectly with no email.
With email in the for loop or separately defined in a function, the job crashes after the first email is sent.
[Pipeline] End of Pipeline an exception which occurred: in field hudson.model.Slave.launcher in object hudson.slaves.DumbSlave#ae938e61 .... and many more
My jenkinsfile:
pipeline {
agent{
label 'master'
}
options {
// Enable timestamps in log
timestamps()
skipDefaultCheckout()
timeout(time: 4, unit: 'MINUTES')
}
stages {
stage('Monitor') {
steps{
script{
def offlineSlaves = []
for (aSlave in hudson.model.Hudson.instance.slaves) {
def thisSlave = aSlave.name
echo 'Name: ' + thisSlave + ' is being checked.'
if ( aSlave.getComputer().isOffline().toString() == 'true') {
slaveState = 'OFFLINE'
echo 'Name: ' + thisSlave + ' is ' + slaveState + ' !'
emailext (
mimeType: 'text/html',
body: "${env.JOB_NAME} found an OFFLINE node: ${name} ",
subject: "Jenkins ERROR: Build Node ${name} is OFFLINE " ,
to: 'jfisher#xxx')
}
}
}
}
}
}
post {
failure {
emailext (
body: 'Monitor Nodes Jenkins Job failed !',
presendScript: '$DEFAULT_PRESEND_SCRIPT',
recipientProviders: [requestor(),culprits()],
subject: 'Monitor Nodes Jenkins Failed',
to: 'jfisher#intouchhealth.com')
}
}
}
The problem with this code is the getComputer() part. In the pipeline you should only use Serializable and the SlaveComputer returned from getComputer() isn't.
https://javadoc.jenkins.io/hudson/slaves/SlaveComputer.html
What you should do is move this part to a function annotated with NonCPS
#NonCPS
def shallTrigger() {
for (aSlave in hudson.model.Hudson.instance.slaves) {
def thisSlave = aSlave.name
echo 'Name: ' + thisSlave + ' is being checked.'
if ( aSlave.getComputer().isOffline().toString() == 'true') {
slaveState = 'OFFLINE'
echo 'Name: ' + thisSlave + ' is ' + slaveState + ' !'
emailext (
mimeType: 'text/html',
body: "${env.JOB_NAME} found an OFFLINE node: ${name} ",
subject: "Jenkins ERROR: Build Node ${name} is OFFLINE " ,
to: 'jfisher#xxx')
}
}
}

Install Magento 2 and Varnish Cache on different server

I have 2 servers, one which has magento 2 installed (ip - 129.89.188.244 port 80) and Varnish on another (ip - 129.89.188.245 port 80)
My Varnish Configuration:
File /etc/default/varnish:-
DAEMON_OPTS="-a :80 \
-T 127.0.0.1:6082 \
-b 129.89.188.244:80 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s malloc,256m"
netstat -tulpn :-
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1288/sshd
tcp 0 0 127.0.0.1:6082 0.0.0.0:* LISTEN 11115/varnishd
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 11115/varnishd
tcp6 0 0 :::22 :::* LISTEN 1288/sshd
tcp6 0 0 :::80 :::* LISTEN 11115/varnishd
/etc/varnish/default.vcl : -
# VCL version 5.0 is not supported so it should be 4.0 even though actually used Varnish version is 5
vcl 4.0;
import std;
# The minimal Varnish version is 5.0
# For SSL offloading, pass the following header in your proxy server or load balancer: 'X-Forwarded-Proto: https'
backend default {
.host = "129.89.188.244";
.port = "80";
.first_byte_timeout = 600s;
.probe = {
.url = "/pub/health_check.php";
.timeout = 2s;
.interval = 5s;
.window = 10;
.threshold = 5;
}
}
acl purge {
"129.89.188.245";
"127.0.0.1";
"localhost";
}
sub vcl_recv {
if (req.method == "PURGE") {
if (client.ip !~ purge) {
return (synth(405, "Method not allowed"));
}
# To use the X-Pool header for purging varnish during automated deployments, make sure the X-Pool header
# has been added to the response in your backend server config. This is used, for example, by the
# capistrano-magento2 gem for purging old content from varnish during it's deploy routine.
if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
}
if (req.http.X-Magento-Tags-Pattern) {
ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
}
if (req.http.X-Pool) {
ban("obj.http.X-Pool ~ " + req.http.X-Pool);
}
return (synth(200, "Purged"));
}
if (req.method != "GET" &&
req.method != "HEAD" &&
req.method != "PUT" &&
req.method != "POST" &&
req.method != "TRACE" &&
req.method != "OPTIONS" &&
req.method != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}
# We only deal with GET and HEAD by default
if (req.method != "GET" && req.method != "HEAD") {
return (pass);
}
# Bypass shopping cart, checkout and search requests
if (req.url ~ "/checkout" || req.url ~ "/catalogsearch") {
return (pass);
}
# Bypass health check requests
if (req.url ~ "/pub/health_check.php") {
return (pass);
}
# Set initial grace period usage status
set req.http.grace = "none";
# normalize url in case of leading HTTP scheme and domain
set req.url = regsub(req.url, "^http[s]?://", "");
# collect all cookies
std.collect(req.http.Cookie);
# Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
if (req.http.Accept-Encoding) {
if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
# No point in compressing these
unset req.http.Accept-Encoding;
} elsif (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
set req.http.Accept-Encoding = "deflate";
} else {
# unkown algorithm
unset req.http.Accept-Encoding;
}
}
# Remove Google gclid parameters to minimize the cache objects
set req.url = regsuball(req.url,"\?gclid=[^&]+$",""); # strips when QS = "?gclid=AAA"
set req.url = regsuball(req.url,"\?gclid=[^&]+&","?"); # strips when QS = "?gclid=AAA&foo=bar"
set req.url = regsuball(req.url,"&gclid=[^&]+",""); # strips when QS = "?foo=bar&gclid=AAA" or QS = "?foo=bar&gclid=AAA&bar=baz"
# Static files caching
if (req.url ~ "^/(pub/)?(media|static)/") {
# Static files should not be cached by default
return (pass);
# But if you use a few locales and don't use CDN you can enable caching static files by commenting previous line (#return (pass);) and uncommenting next 3 lines
#unset req.http.Https;
#unset req.http.X-Forwarded-Proto;
#unset req.http.Cookie;
}
return (hash);
}
sub vcl_hash {
if (req.http.cookie ~ "X-Magento-Vary=") {
hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
}
# For multi site configurations to not cache each other's content
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
# To make sure http users don't see ssl warning
if (req.http.X-Forwarded-Proto) {
hash_data(req.http.X-Forwarded-Proto);
}
}
sub vcl_backend_response {
set beresp.grace = 3d;
if (beresp.http.content-type ~ "text") {
set beresp.do_esi = true;
}
if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
set beresp.do_gzip = true;
}
if (beresp.http.X-Magento-Debug) {
set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
}
# cache only successfully responses and 404s
if (beresp.status != 200 && beresp.status != 404) {
set beresp.ttl = 0s;
set beresp.uncacheable = true;
return (deliver);
} elsif (beresp.http.Cache-Control ~ "private") {
set beresp.uncacheable = true;
set beresp.ttl = 86400s;
return (deliver);
}
# validate if we need to cache it and prevent from setting cookie
# images, css and js are cacheable by default so we have to remove cookie also
if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
unset beresp.http.set-cookie;
}
# If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass
if (beresp.ttl <= 0s ||
beresp.http.Surrogate-control ~ "no-store" ||
(!beresp.http.Surrogate-Control &&
beresp.http.Cache-Control ~ "no-cache|no-store") ||
beresp.http.Vary == "*") {
# Mark as Hit-For-Pass for the next 2 minutes
set beresp.ttl = 120s;
set beresp.uncacheable = true;
}
return (deliver);
}
sub vcl_deliver {
if (resp.http.X-Magento-Debug) {
if (resp.http.x-varnish ~ " ") {
set resp.http.X-Magento-Cache-Debug = "HIT";
set resp.http.Grace = req.http.grace;
} else {
set resp.http.X-Magento-Cache-Debug = "MISS";
}
} else {
unset resp.http.Age;
}
# Not letting browser to cache non-static files.
if (resp.http.Cache-Control !~ "private" && req.url !~ "^/(pub/)?(media|static)/") {
set resp.http.Pragma = "no-cache";
set resp.http.Expires = "-1";
set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
}
unset resp.http.X-Magento-Debug;
unset resp.http.X-Magento-Tags;
unset resp.http.X-Powered-By;
unset resp.http.Server;
unset resp.http.X-Varnish;
unset resp.http.Via;
unset resp.http.Link;
}
sub vcl_hit {
if (obj.ttl >= 0s) {
# Hit within TTL period
return (deliver);
}
if (std.healthy(req.backend_hint)) {
if (obj.ttl + 300s > 0s) {
# Hit after TTL expiration, but within grace period
set req.http.grace = "normal (healthy server)";
return (deliver);
} else {
# Hit after TTL and grace expiration
return (miss);
}
} else {
# server is not healthy, retrieve from cache
set req.http.grace = "unlimited (unhealthy server)";
return (deliver);
}
}
Now the issue is, when I open the URL 129.89.188.244 , magento opens but it's not getting cached in varnish. But when I call varnish URL 129.89.188.245, it will redirect to my magento url 129.89.188.244. In my varnish log, it shows that the page is cached already but magento is not getting served from that varnish cache.
In order for the cache to work, your requests always have to go through the proxy/cache (Varnish).
Varnish evaluates your requests and decides if the request is already cached (then returns it from the cache) or not - then it will redirect it to Magento and cache the response, before returning it to the client.
That's how most caches work. You can read in detail how Varnish cache mechanism works here.
If you want to hit your cache, you always have to go through Varnish (129.89.188.245:80).
Consider this diagram form the official documentation:
this is the expected behaviour.
Varnish sits in front of you Magento server and if you ping your Magento server bypassing Varnish, opening 129.89.188.244, then Magento will send you a response without involving Varnish, while if you ping varnish, Varnish will then call Magento and cache. The other way around is not possible and makes no sense.
This solution worked for me. To configure Varnish and Magento on a different server.
Varnish server: xxx.xxx.xxx.xxx port 80
Magento server: yyy.yyy.yyy.yyy port 80
Changes need to be made on the varnish server:
1. login to varnish server
2. go to file /etc/varnish/default.vcl
3. under the "backend default" update
.host = "yyy.yyy.yyy.yyy";//(use Magento server IP for better network)
.port = "80";//(Magento web server port)
4. Restart the Varnish (systemctl restart varnish)
Note: Kindly use the default VCL that is generated during the varnish installation and don't update it with Magento generated VCL for varnish ( available from Magento Admin)
Changes need to be made on the Magento server:
1. Log in to the Magento server
2. Go to the env.php file located in the app/etc directory
3. Update the values in 'http_cache_hosts' => [
[
'host' => 'xxx.xxx.xxx.xxx', //(varnish server public ip)
'port' => '80' // ( varnish server port)
]
]
Now update your base URL's on the core_config_data table to your varnish public Ip (http://xxx.xxx.xxx.xxx/)
flush the Magento caches ( bin/magento ca:fl)

Content API for Shopping not working in OpenCart

I am trying to integrate the Content API for Shopping in OpenCart into my PHP app and getting this error:
Fatal error: Uncaught exception 'GSC_ParseError' with message 'Not
Found' in
/home/public_html/admin/controller/module/GShoppingContent.php:2805
Stack trace: #0
/home/public_html/admin/controller/module/GShoppingContent.php(980):
_GSC_AtomParser::parse('Not Found') #1 /home/public_html/admin/controller/module/contentapi.php(76):
GSC_Client->insertProduct(Object(GSC_Product)) #2 [internal function]:
ControllerModuleContentApi->index(Array) #3
/home//public_html/vqmod/vqcache/vq2-system_modification_system_engine_action.php(71):
call_user_func(Array, Array) #4
/home//public_html/vqmod/vqcache/vq2-system_engine_front.php(89):
Action->execute(Object(Registry)) #5
/home//public_html/vqmod/vqcache/vq2-system_engine_front.php(63):
Front->execute(Object(Action)) #6
/home//public_html/admin/index.php(175):
Front->dispatch(Object(Action), Object(Action)) #7 {main} thrown in
/home//public_html/admin/controller/module/GShoppingContent.php on
line 2805
Here's my code:
require_once('GShoppingContent.php');
class ControllerModuleContentApi extends Controller {
private $error = array();
public function index() {
$this->load->language('module/contentapi');
$this->db->query("
CREATE TABLE IF NOT EXISTS `contentapi` (
`contentapiid` int(11) NOT NULL AUTO_INCREMENT,
`contentapimerchantid` varchar(256) NOT NULL,
`contentapiemail` varchar(256) NOT NULL,
`contentapipassword` varchar(256) NOT NULL,
PRIMARY KEY (`contentapiid`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
");
$contentAPI11=$this->db->query("select * from contentapi");
$data['contentapidata'] = $contentAPI11->rows;
if (isset($this->request->post['merchantid']) && isset($this->request->post['GoogleMerchantEmail']) && isset($this->request->post['GoogleMerchantPassword'] )) {
$merchantid=$this->request->post['merchantid'];
$GoogleMerchantEmail=$this->request->post['GoogleMerchantEmail'];
$GoogleMerchantPassword=$this->request->post['GoogleMerchantPassword'];
$primaryid=$this->request->post['primaryid'];
$this->db->query("UPDATE contentapi SET contentapimerchantid='$merchantid',contentapiemail='$GoogleMerchantEmail',contentapipassword='$GoogleMerchantPassword' where contentapiid='$primaryid'");
}
$retrieveapidetails=$this->db->query("select * from contentapi");
if($retrieveapidetails->num_rows>0)
{
$retrieveapidetails1=$retrieveapidetails->rows;
foreach($retrieveapidetails1 as $retrieveapidetails2)
{
$contentapimerchantid=$retrieveapidetails2["contentapimerchantid"];
$contentapiemail=$retrieveapidetails2["contentapiemail"];
$contentapipassword=$retrieveapidetails2["contentapipassword"];
}
$retrieveproducts=$this->db->query("select oc_product.model,oc_product.sku,oc_product.price,oc_product.image,oc_product_description.name,oc_product_description.description from oc_product,oc_product_description where oc_product.product_id=oc_product_description.product_id");
$retrieveproducts1=$retrieveproducts->rows;
$client = new GSC_Client($contentapimerchantid);
$client->login($contentapiemail, $contentapipassword);
foreach($retrieveproducts1 as $retrieveproducts2)
{
$productid=$retrieveproducts2["product_id"];
$productname=$retrieveproducts2["name"];
$productmodel=$retrieveproducts2["model"];
$productprice=$retrieveproducts2["price"];
$productimage=$retrieveproducts2["image"];
$productsku=$retrieveproducts2["sku"];
$productdescription=$retrieveproducts2["description"];
$product = new GSC_Product();
$product->setTitle($productname);
$product->setDescription($productdescription);
$link = 'https://testingsite.com/index.php?route=product/product&product_id=' . $productid;
$product->setProductLink($link);
$product->setSKU($productsku);
$product->setImageLink('https://testingsite.com/'.$productimage);
$product->setBrand($productimage);
$product->setPrice($productprice, 'usd');
$entry = $client->insertProduct($product);
}
}

Resources