Sed is drawing me nuts !!
I'm trying to append a file with a input data from keyboard. This data needs to be put in the line after some text matches criteria.
My file:
location ~* ^/(index\.php/admin|admin) {
allow 10.10.1.25;
try_files $uri $uri/ /index.php?$args;
location ~* \.php$ { try_files /dummy #proxy; }
deny all;
}
That is the part of my script where I'm collecting data from keyboard and run sed:
echo "Type IP you want to whitelist > "
read ip
sed -i '/allow 10.10.1.25;/i allow ${ip};' test.txt
My file now looks like:
location ~* ^/(index\.php/admin|admin) {
allow $ip
allow 10.10.1.25;
try_files $uri $uri/ /index.php?$args;
location ~* \.php$ { try_files /dummy #proxy; }
deny all;
}
What I'm doing wrong ? I want to value from this variable was in the file not $ip. How my sed command should looks like if I wanted this to be inserted after this line:
location ~* ^/(index\.php/admin|admin) {
Many thanks for your help
Related
I have been searching thoruhg nginx documentation and posts online but I can't find the answer to this.
I am running a python application and I want one of the paths altered slightly. I can't do this in python for various reasons.
I want the input url:
/public/web/apidocs***
(where * could be anything - including nothing)
to be passed to the python app as
/apidocs*
This is my configuration:
server {
listen 80;
server_name localhost; ##ignored if there is only one server block
charset utf-8;
client_max_body_size 75M;
location = /frontend/webfrontendConnectionData {
try_files $uri #yourapplication;
}
location /public/web/frontend {
alias /frontend/;
autoindex off;
}
location /public/web/adminfrontend {
alias /adminfrontend/;
autoindex off;
}
location ^(/public/web)(/apidocs.*)$ {
try_files $2 #yourapplication;
}
location / {
try_files $uri #yourapplication;
}
location #yourapplication {
include uwsgi_params;
uwsgi_pass unix:/app/uwsgi.sock;
}
}
Update based on answer so far:
Thank you for pointing out the mistake or missing out the ~ in the config.
The config has changed as follows:
location ~ ^(/public/web)(/apidocs.*)$ {
try_files $2 #yourapplication;
}
location / {
try_files $uri #yourapplication;
}
location #yourapplication {
include uwsgi_params;
uwsgi_pass unix:/app/uwsgi.sock;
}
Unfortunately it still doesn't work as expected:
wget 127.0.0.1:80/apidocs -> Goes to python App and works as expected
wget 127.0.0.1:80/apidocs/ -> Goes to python App and works as expected
wget 127.0.0.1:80/apidocs/swagger.json -> Goes to python App and works as expected
wget 127.0.0.1:80/public/web/apidocs -> FAILS I want this to give the same response as wget 127.0.0.1:80/apidocs
wget 127.0.0.1:80/public/web/apidocs/ -> FAILS I want this to give the same response as wget 127.0.0.1:80/apidocs/
wget 127.0.0.1:80/public/web/apidocs/swagger.json -> FAILS I want this to give the same response as wget 127.0.0.1:80/apidocs/swagger.json
All the failed responses give me:
robert#ansiblerunner:~/t$ wget 127.0.0.1:80/public/web/apidocs/swagger.json
--2019-04-08 09:50:39-- http://127.0.0.1/public/web/apidocs/swagger.json
Connecting to 127.0.0.1:80... connected.
HTTP request sent, awaiting response... 404 NOT FOUND
2019-04-08 09:50:39 ERROR 404: NOT FOUND.
Can anyone suggest the correct syntax for the rule I want.
Update #2
After reading more nginx documentation I found out that it doesn't take the location rules in order and use the first one it hits, rather it has a complex longest match algorithm. I was worried the "location /" block was then overriding all the paths so I have altered the config so it is like the following:
location /api/public {
try_files $uri #yourapplication;
}
location /api/authed {
try_files $uri #yourapplication;
}
location ~* ^(/public/web)(/apidocs.*)$ {
try_files $2 #yourapplication;
}
#location / {
# try_files $uri #yourapplication;
#}
location #yourapplication {
include uwsgi_params;
uwsgi_pass unix:/app/uwsgi.sock;
}
/api/public and /api/authed both work as normal.
it's the /public/web/apidocs that doesn't. The difference is that I need nginx to change the path provided to the python app in this case. This is why I have a regexp with two parts and it passing $2 rather than $uri. I don't think it's doing what I expect which is to pass /apidoc to the app.
I don't see a regex, there, as it lacks a tilde.
You probably want a clause that starts with:
location ~ ^(/public/web)(/apidocs.*)$ {
You didn't mention which docs you were reading.
Consulting a tutorial,
Understanding Nginx Server and Location Block Selection Algorithms,
we see the syntax is:
location optional_modifier location_match {
and we optionally can specify:
~: If a tilde modifier is present, this location will be interpreted as a case-sensitive regular expression match.
I cannot explain your reported 404 symptom,
as the location / prefix clause should have absorbed the GET request
and at least given #yourapplication an opportunity to log the raw URL.
I have an answer
I was completely off base with my understanding of what try_files did not work as I expected. I found an article saying if I use it with an underscore it won't try files and will just use it's fallback.
I also found out that I needed to use rewrite to change the path.
Getting rid of the root location may have not been needed but I prefer not to have it in any case.
My final config is as follows:
server {
listen 80;
server_name localhost; ##ignored if there is only one server block
charset utf-8;
client_max_body_size 75M;
#location = /frontend/webfrontendConnectionData {
# try_files $uri #yourapplication;
#}
location /public/web/frontend {
alias /frontend/;
autoindex off;
}
location /public/web/adminfrontend {
alias /adminfrontend/;
autoindex off;
}
# try_files _ #xxx means don't look at any file go to xxx block
location /api/public {
try_files _ #yourapplication;
}
location /api/authed {
try_files _ #yourapplication;
}
location /public/web/apidocs {
rewrite ^/public/web(/apidocs.*)$ /$1? break;
try_files _ #yourapplication;
}
#location / {
# try_files _ #yourapplication;
#}
location #yourapplication {
include uwsgi_params;
uwsgi_pass unix:/app/uwsgi.sock;
}
}
Thanks to J_H for comments and answers.
Installed Pimcore5.1, PHP7 and extensions, MySQL, Nginx
Nginx Virtual Host config is as:
upstream php-pimcore5 {
server 127.0.0.1:9000;
}
server {
listen 80;
server_name s3pim.local;
root /var/www/html/s3pim/web;
index app.php index.php;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log error;
# Pimcore Head-Link Cache-Busting
rewrite ^/cache-buster-(?:\d+)/(.*) /$1 last;
# Stay secure
#
# a) don't allow PHP in folders allowing file uploads
location ~* /var/assets/*\.php(/|$) {
return 404;
}
# b) Prevent clients from accessing hidden files (starting with a dot)
# Access to `/.well-known/` is allowed.
# https://www.mnot.net/blog/2010/04/07/well-known
# https://tools.ietf.org/html/rfc5785
location ~* /\.(?!well-known/) {
deny all;
log_not_found off;
access_log off;
}
# c) Prevent clients from accessing to backup/config/source files
location ~* (?:\.(?:bak|conf(ig)?|dist|fla|in[ci]|log|psd|sh|sql|sw[op])|~)$ {
deny all;
}
# Some Admin Modules need this:
# Database Admin, Server Info
location ~* ^/admin/(adminer|external) {
rewrite .* /app.php$is_args$args last;
}
# Thumbnails
location ~* .*/(image|video)-thumb__\d+__.* {
try_files /var/tmp/$1-thumbnails$request_uri /app.php;
location ~* .*/(image|video)-thumb__\d+__.* {
try_files /var/tmp/$1-thumbnails$request_uri /app.php;
expires 2w;
access_log off;
add_header Cache-Control "public";
}
# Assets
# Still use a whitelist approach to prevent each and every missing asset to go through the PHP Engine.
location ~* (.+?)\.((?:css|js)(?:\.map)?|jpe?g|gif|png|svgz?|eps|exe|gz|zip|mp\d|ogg|ogv|webm|pdf|docx?|xlsx?|pptx?)$ {
try_files /var/assets$uri $uri =404;
expires 2w;
access_log off;
log_not_found off;
add_header Cache-Control "public";
}
# Installer
# Remove this if you don't need the web installer (anymore)
if (-f $document_root/install.php) {
rewrite ^/install(/?.*) /install.php$1 last;
}
location / {
error_page 404 /meta/404;
add_header "X-UA-Compatible" "IE=edge";
try_files $uri /app.php$is_args$args;
}
# Use this location when the installer has to be run
# location ~ /(app|install)\.php(/|$) {
#
# Use this after initial install is done:
location ~ ^/app\.php(/|$) {
send_timeout 1800;
fastcgi_read_timeout 1800;
# regex to split $uri to $fastcgi_script_name and $fastcgi_path
fastcgi_split_path_info ^(.+\.php)(/.+)$;
# Check that the PHP script exists before passing it
try_files $fastcgi_script_name =404;
include fastcgi.conf;
# Bypass the fact that try_files resets $fastcgi_path_info
# see: http://trac.nginx.org/nginx/ticket/321
set $path_info $fastcgi_path_info;
fastcgi_param PATH_INFO $path_info;
# Activate these, if using Symlinks and opcache
# fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
# fastcgi_param DOCUMENT_ROOT $realpath_root;
fastcgi_pass php-pimcore5;
# Prevents URIs that include the front controller. This will 404:
# http://domain.tld/app.php/some-path
# Remove the internal directive to allow URIs like this
internal;
}
location ~ \.php$ {
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
# PHP-FPM Status and Ping
location /fpm- {
access_log off;
include fastcgi_params;
location /fpm-status {
allow 127.0.0.1;
# add additional IP's or Ranges
deny all;
fastcgi_pass php-pimcore5;
}
location /fpm-ping {
fastcgi_pass php-pimcore5;
}
}
# nginx Status
# see: https://nginx.org/en/docs/http/ngx_http_stub_status_module.html
# location /nginx-status {
# allow 127.0.0.1;
# deny all;
# access_log off;
# stub_status;
# }
}
Excluded .htaccess file, start PHP FPM, Start Nginx
Now when hitting s3pim.local, it redirects to s3pim.local/install which redirects to s3pim.local/install/ which redirects to s3pim.local/install and so on..
If in install.php I write some code and die then that code get printed. Its all occuring where it is
$response->send
in Project_Root/web/install.php that is redirecting it again and again on install php.
How to resolve this issue?
As mentioned in comments, you have to uncomment following line:
# location ~ /(app|install)\.php(/|$) {
And comment one below:
location ~ ^/app\.php(/|$) {
Remember to undo this after installation succeeds.
In the end, your file should look like this:
upstream php-pimcore5 {
server 127.0.0.1:9000;
}
server {
listen 80;
server_name s3pim.local;
root /var/www/html/s3pim/web;
index app.php index.php;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log error;
# Pimcore Head-Link Cache-Busting
rewrite ^/cache-buster-(?:\d+)/(.*) /$1 last;
# Stay secure
#
# a) don't allow PHP in folders allowing file uploads
location ~* /var/assets/*\.php(/|$) {
return 404;
}
# b) Prevent clients from accessing hidden files (starting with a dot)
# Access to `/.well-known/` is allowed.
# https://www.mnot.net/blog/2010/04/07/well-known
# https://tools.ietf.org/html/rfc5785
location ~* /\.(?!well-known/) {
deny all;
log_not_found off;
access_log off;
}
# c) Prevent clients from accessing to backup/config/source files
location ~* (?:\.(?:bak|conf(ig)?|dist|fla|in[ci]|log|psd|sh|sql|sw[op])|~)$ {
deny all;
}
# Some Admin Modules need this:
# Database Admin, Server Info
location ~* ^/admin/(adminer|external) {
rewrite .* /app.php$is_args$args last;
}
# Thumbnails
location ~* .*/(image|video)-thumb__\d+__.* {
try_files /var/tmp/$1-thumbnails$request_uri /app.php;
expires 2w;
access_log off;
add_header Cache-Control "public";
}
# Assets
# Still use a whitelist approach to prevent each and every missing asset to go through the PHP Engine.
location ~* (.+?)\.((?:css|js)(?:\.map)?|jpe?g|gif|png|svgz?|eps|exe|gz|zip|mp\d|ogg|ogv|webm|pdf|docx?|xlsx?|pptx?)$ {
try_files /var/assets$uri $uri =404;
expires 2w;
access_log off;
log_not_found off;
add_header Cache-Control "public";
}
# Installer
# Remove this if you don't need the web installer (anymore)
if (-f $document_root/install.php) {
rewrite ^/install(/?.*) /install.php$1 last;
}
location / {
error_page 404 /meta/404;
add_header "X-UA-Compatible" "IE=edge";
try_files $uri /app.php$is_args$args;
}
# Use this location when the installer has to be run
location ~ /(app|install)\.php(/|$) {
#
# Use this after initial install is done:
# location ~ ^/app\.php(/|$) {
send_timeout 1800;
fastcgi_read_timeout 1800;
# regex to split $uri to $fastcgi_script_name and $fastcgi_path
fastcgi_split_path_info ^(.+\.php)(/.+)$;
# Check that the PHP script exists before passing it
try_files $fastcgi_script_name =404;
include fastcgi.conf;
# Bypass the fact that try_files resets $fastcgi_path_info
# see: http://trac.nginx.org/nginx/ticket/321
set $path_info $fastcgi_path_info;
fastcgi_param PATH_INFO $path_info;
# Activate these, if using Symlinks and opcache
# fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
# fastcgi_param DOCUMENT_ROOT $realpath_root;
fastcgi_pass php-pimcore5;
# Prevents URIs that include the front controller. This will 404:
# http://domain.tld/app.php/some-path
# Remove the internal directive to allow URIs like this
internal;
}
location ~ \.php$ {
fastcgi_pass php-pimcore5;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
}
Its a very basic thing i need to do, but i cant figure this one out.
i have following rule in my nginx.conf
location ^/search {
#try_files index.php?search=$1 last;
rewrite ^/search/(.*)/ index.php?search=$1;
#rewrite ^/search/([^/]+)/page/([^/]+)/?$ /index.php?search=$1&page=$2;
}
location ^/article {
rewrite ^/article/([^/]+)/([^/]+)/$ /index.php?layout=article&id=$1;
}
location ^/images {
rewrite ^/images/([^/]+)$ /images/$1 break;
rewrite ^/images/([^/]+)/([^/]+)$ /images/$1/$2 break;
}
location ^/css {
rewrite ^/css/([^/]+)$ /css/$1 break;
rewrite ^/css/([^/]+)/([^/]+)$ /css/$1/$2 break;
}
location ~* ^/(css|img|js|flv|swf|download)/(.+)$ {
root $root_path;
}
location ~ /\.ht {
deny all;
}
location ~ \.php$ {
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include /etc/nginx/fastcgi_params;
}
basically all i want to do is
match below query on search to this above.
example.com/search-term1/
or
example.com/search term/
or
example.com/search/search-term1/
or
example.com/search/search-term1/page/1/
above url patterns ?how do i do this ? above rule gives me error 500 Internal server error.
in the Nginx log i get following as
2015/08/05 12:56:39 [error] 11136#0: *2 could not find named location "#rewrite", client: 202.172.107.194, server: example.com, request: "GET /search/search-term1/page/1/ HTTP/1.1", host: "example.com"
i fixed this issue by moving them out of the location block and putting them directly into the server block.
I am attempting to use Bash in order to run a form of an install process. During this process, a configuration file is copied and certain values are replaced inside of it. Such a config can be found below:
server {
listen 80;
root ${INSTALLPATH};
server_name ${SITEURL};
client_max_body_size 20m;
client_body_timeout 120s;
location / {
try_files /public/router.php =404;
fastcgi_split_path_info ^(.+?\.php)(/.*)$;
fastcgi_pass ${PHPSERV};
fastcgi_index router.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include /etc/nginx/fastcgi_params;
}
location /assets {
try_files /app/$uri =404;
}
}
#Enables HTTPS access
#This requires you to install certificates and is not enabled by default
#If you wish to enable HTTPS, uncomment (remove the #s) from the below lines
#And change the ssl_certificate and ssl_certificate_key to point to the correct
#certificates.
#server {
# listen 443;
# root ${INSTALLPATH};
# server_name ${SITEURL};
#
# ssl on;
# ssl_certificate /etc/nginx/ssl/site.crt;
# ssl_certificate_key /etc/nginx/ssl/site.key;
#
# location / {
# try_files /public/router.php =404;
# fastcgi_split_path_info ^(.+?\.php)(/.*)$;
# fastcgi_pass ${PHPSERV};
# fastcgi_index router.php;
# fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
# include /etc/nginx/fastcgi_params;
# }
#
# location /assets {
# try_files /app/$uri =404;
# }
#
#}
I have been finding that most examples stem off the use of eval, and I have tried using that to do this replacement, however it appears that the file here does not properly expand, and bash tries to execute some stuff and can't.
Currently, I have this
INSTALLPATH="/var/www/html/mycustomsite/"
PHPSERV="127.0.0.1:9000"
SITEURL="example.com"
while read -r line; do
eval echo -e "${line}"
done < template
However, this does not properly replace the values that are declared, nor does it generate the file correctly. It loses for example any line that starts with # and tries to execute others (along with losing some spacing).
What would be the correct way to do this, using just Bash and commands available on most Linux systems?
Security tips
This don't take care of security issues! Using eval is evil!
The compatible answer is not better!
Of course, you have to be confident about the content of your template!!
If else, try using sed! (see my last answer)
Quick way bash only!:
Under bash you can simply:
eval "INSTALLPATH='/somepath/somewhere' SITEURL='example.com' PHPSERV='127.0.0.1:9000'; echo \"$(<template)\""
or
eval "INSTALLPATH='/somepath/somewhere'
SITEURL='example.com'
PHPSERV='127.0.0.1:9000';
echo \"$(<template)\""
As you're using eval, you could store your resulting config file into one variable:
eval "INSTALLPATH='/somepath/somewhere'
SITEURL='example.com'
PHPSERV='127.0.0.1:9000';
cfgBody=\"$(<template)\""
Then
echo "$cfgBody"
and/or
echo "$cfgBody" >/cfgpath/cfgfile
Doing this into a loop
tmplBody="$(<template)"
while read INSTALLPATH SITEURL PHPSERV CFGFILE;do
[ "$CFGFILE" ] && eval "echo \"$tmplBody\"" >"$CFGFILE"
done <<<"
/somepath/somewhere example.com 127.0.0.1:9000 /tmp/file1
'/some\ other\ path/elsewhere' sample2.com 127.0.0.1:9001 /tmp/file2
"
Note: On second line, there are escaped spaces (prepanded with a backshash \ and quotes '. The backslash tell read to not split the variable, and the quotes have to be added into the resulting /tmp/file2.
Soft way (compatible answer)
Under posix shell, you may do this way:
#!/bin/sh
(
cat <<eohead
#!/bin/sh
INSTALLPATH='/somepath/somewhere'
SITEURL='example.com'
PHPSERV='127.0.0.1:9000';
cat <<eof
eohead
cat template
echo eof
) | /bin/sh
This don't require bash, was tested under dash and busybox.
bash Without eval !
sedcmd=''
for var in INSTALLPATH SITEURL PHPSERV;do
printf -v sc 's/${%s}/%s/;' $var "${!var//\//\\/}"
sedcmd+="$sc"
done
sed -e "$sedcmd" <template
Could work into a loop:
while read INSTALLPATH SITEURL PHPSERV CFGFILE;do
if [ "$CFGFILE" ] ;then
sedcmd=''
for var in INSTALLPATH SITEURL PHPSERV;do
printf -v sc 's/${%s}/%s/;' $var "${!var//\//\\/}"
sedcmd+="$sc"
done
sed -e "$sedcmd" <template >"$CFGFILE"
fi
done <<<"
/somepath/somewhere example.com 127.0.0.1:9000 /tmp/file1
'/some\ other\ path/elsewhere' sample2.com 127.0.0.1:9001 /tmp/file2
"
Compatible answer, using sed
This could work without so called bashisms, into a loop:
#!/bin/sh
while read INSTALLPATH SITEURL PHPSERV CFGFILE;do
sedcmd="s|\\\${INSTALLPATH}|${INSTALLPATH}|;"
sedcmd="${sedcmd}s|\\\${SITEURL}|${SITEURL}|;"
sedcmd="${sedcmd}s|\\\${PHPSERV}|${PHPSERV}|;"
sed -e "$sedcmd" template >"$CFGFILE"
done <<eof
/somepath/somewhere example.com 127.0.0.1:9000 /tmp/file1
'/some\ other\ path/elsewhere' sample2.com 127.0.0.1:9001 /tmp/file2
eof
Comparing outputs:
diff -u99 /tmp/file{1,2}
--- /tmp/file1 2015-05-31 11:02:03.407463963 +0200
+++ /tmp/file2 2015-05-31 11:02:03.407463963 +0200
## -1,22 +1,22 ##
server {
listen 80;
- root /somepath/somewhere;
- server_name example.com;
+ root '/some other path/elsewhere';
+ server_name sample2.com;
client_max_body_size 20m;
client_body_timeout 120s;
location / {
try_files /public/router.php =404;
fastcgi_split_path_info ^(.+?\.php)(/.*)$;
- fastcgi_pass 127.0.0.1:9000;
+ fastcgi_pass 127.0.0.1:9001;
fastcgi_index router.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include /etc/nginx/fastcgi_params;
}
location /assets {
try_files /app/$uri =404;
}
}
I'd like to protect the root with passwd and make the urlfriendly rewrite for modx cms.
only the rewriting works but the password won't be requested.
my ispconfig nginx directives looks like this
location / {
index index.html index.php
auth_basic "Protected Area";
auth_basic_user_file /var/www/clients/client21/web22/web/htpasswd;
client_max_body_size 0;
if (!-e $request_filename) {
rewrite ^/(.*)$ /index.php?q=$1 last;
}
}
location ~ /\.ht {
deny all;
}
location ~ \.php$ {
include /etc/nginx/fastcgi_params;
fastcgi_pass unix:/var/run/php5-fpm.sock;
fastcgi_read_timeout 600;
fastcgi_buffers 16 16k;
fastcgi_buffer_size 32k;
client_max_body_size 0;
}
seems like the auth_basic won't be executed or overwriten by the rewrite rule. anyone got a idea?
after alot of ppl viewing this question but i didn't received a answer here i post the solution to the problem. The problem was that the basic auth needs to be done in the php directive.
location / {
index index.html index.php
client_max_body_size 0;
if (!-e $request_filename) {
rewrite ^/(.*)$ /index.php?q=$1 last;
}
}
location ~ /\.ht {
deny all;
}
location ~ \.php$ {
auth_basic "Protected Area";
auth_basic_user_file /var/www/clients/client21/web22/web/htpasswd;
include /etc/nginx/fastcgi_params;
fastcgi_pass unix:/var/run/php5-fpm.sock;
fastcgi_read_timeout 600;
fastcgi_buffers 16 16k;
fastcgi_buffer_size 32k;
client_max_body_size 0;
}
Try this set of rules. No PHP directives and leverages FURLS.
location / {
auth_basic "Restricted";
# Add path to .htpasswd file
auth_basic_user_file /var/www/clients/client21/web22/web/.htpasswd;
# if the protected location is modx, then ...
try_files $uri $uri/ #modx-rewrite;
# otherwise, if static files, you’d use
# try_files $uri $uri/ =404;
# This line should always be placed at the end
try_files $uri $uri/ #modx-rewrite;
}