I have an issue where pages from two distinct websites are being indexed by Google as one site.
So, we have two websites; let's call them siteone.com and sitetwo.com. When I do a search like "site:sitetwo.com" I get a few SERP pages of results, all appearing as from the www.sitetwo.com domain.
However, some of the results are actually content from siteone.com. These pages exist on siteone.com, but do not on sitetwo.com. If requested, they return a 404. As an example, there may be a page in the results such as "http://www.sitetwo.com/foo/bar" but that page doesn't exist and never has. The page "http://www.siteone.com/foo/bar does exist.
The websites are on the same server, at the same IP address. Sitetwo.com has been up for about a week.
Stack is: CentOS, NGINX, Node.js, MySQL. The sites are in separate directories, running separate instances of Node, and while both DBs are on the same MySQL server, they are separate DBs. The only thing they share is the server and the IP. I've included a portion of my nginx.conf file as this is, IMO, the most likely place I could've messed up.
server {
listen 80;
server_name siteone.com;
return 301 http://www.siteone.com$request_uri;
}
server {
listen 80;
server_name sitetwo.com;
return 301 http://www.sitetwo.com$request_uri;
}
server {
listen 80;
listen [::]:80;
server_name sitetwo.com www.sitetwo.com;
root /usr/share/nginx/html;
# Load configuration files for the default server block.
include /etc/nginx/default.d/\*.conf;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-NginX-Proxy true;
proxy_set_header X-Real-IP $remote_addr;
proxy_buffering off;
proxy_pass http://127.0.0.1:9002;
proxy_pass_header X-CSRF-TOKEN;
proxy_redirect off;
}
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
server {
listen 80 default_server;
listen 443 ssl;
listen [::]:80 default_server;
server_name siteone.com www.siteone.com;
root /usr/share/nginx/html;
ssl_certificate /etc/ssl/nginx/www.siteone.com.crt;
ssl_certificate_key /etc/ssl/nginx/www.siteone.com.key;
# Load configuration files for the default server block.
include /etc/nginx/default.d/\*.conf;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-NginX-Proxy true;
proxy_set_header X-Real-IP $remote_addr;
proxy_buffering off;
proxy_pass http://127.0.0.1:9001;
proxy_pass_header X-CSRF-TOKEN;
proxy_redirect off;
}
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
The "\" in "include /etc/nginx/default.d/*.conf;" is not really there, but it was affecting StackOverflow's formatting.
Related
I want to redirect a URL e.g domain.com/api/ to a specific Node.js server, the root URL shows my website. At the moment I use this config:
server {
listen 80 default_server;
listen [::]:80 default_server;
root /var/www/html;
server_name _;
location / {
try_files $uri $uri/ =404;
}
location /api {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://localhost:3031/;
proxy_ssl_session_reuse off;
proxy_set_header Host $http_host;
proxy_cache_bypass $http_upgrade;
proxy_redirect off;
}
but it does not work.
What has gone wrong?
Thanks for help and best regards :)
You fogot closing bracket after location /api section.
Your config working at my machine.
server {
listen 80 default_server;
listen [::]:80 default_server;
root /var/www/html;
server_name _;
location / {
try_files $uri $uri/ =404;
}
location /api {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://localhost:3031/;
proxy_ssl_session_reuse off;
proxy_set_header Host $http_host;
proxy_cache_bypass $http_upgrade;
proxy_redirect off;
}
}
Also your nodejs backend must handle '/api' requests.
Please note that proxy_pass directive gets argument for backend and optional URI.
That means proxy_pass http://localhost:3031; will get the URI from the user, e.g /api/res.json so final URL for the NODE JS is: http://localhost:3031/api/res.json
But when you provide the URI to the directive itself, it overrides the requested URI in the matching location. e.g location /api and proxy_pass http://localhost:3031/; (note the suffixed slash). so the part /api is replaced by / and final URL is: http://localhost:3031/res.json.
from the NGINX docs:
If the proxy_pass directive is specified with a URI, then when a request is passed to the server, the part of a normalized request URI matching the location is replaced by a URI specified in the directive:
location /name/ {
proxy_pass http://127.0.0.1/remote/;
}
So it's important to understand that the part of the requested URI /name/ is replaced by /remote/ and then the rest of the requested URI is added to the final sent URI.
I've spent all day yesterday and today learning how nginx works and I got two different domains working, one with Ghost blogging platform and a static page (future NodeJS app), now I'm trying to setup the subdomain, but I'm kind of frustrated because I feel like I'm almost there but it's not working... This is my current setup:
#Main Domain
server {
listen 80;
listen [::]:80;
server_name example.com;
root /var/www/portfolio;
index index.html;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
# proxy_pass http://127.0.0.1:2222;
}
location ~ /.well-known {
allow all;
}
client_max_body_size 50m;
}
#Sub domain
server {
listen 80;
listen [::]:80;
server_name example.com/blog;
root /var/www/ghost/system/nginx-root;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
proxy_pass http://127.0.0.1:2368;
}
location ~ /.well-known {
allow all;
}
client_max_body_size 50m;
}
The idea is to create mysite.com/blog where eventually mysite would be a nodejs app, prob linking the route later will be another problem but... one at a time lol, how can I establish that subdomain?
If I separate the config file into a separate file, I would get the other domain working :/
Thanks
EDIT: I've found that with bucket in S3 on AWS I could acomplish that, but now I don't need it for what I'm doing jeje, but it's good to know.
First: it's not a subdomain, but a subfolder called blog.
If you want to run two apps where one appears in a subfolder, you could do the following
Define two upstreams / proxy pass them to different ports the
Have them in the same config file then
Have two location blocks (location / and location /blog)
Does that make sense? Otherwise one will probably shadow the other.
Note: This is not a complete answer, you probably will need to tinker a bit
As #Jonathan mentioned, from nginx's point of view this is the same site, but you need nginx to handle both locations differently.
Here's how it would look like
server {
listen 80;
listen [::]:80;
server_name example.com;
root /var/www/portfolio;
index index.html;
client_max_body_size 50m;
location / {
# your normal location settings
}
# your blog is defined here
location /blog {
root /var/www/ghost/system/nginx-root;
# You'll probably need to do a rewrite here, because a
# /blog/article needs to be passed as `/article` to the
# app server
# rewrite ^/blog/(.*) $1;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
proxy_pass http://127.0.0.1:2368;
}
}
}
I'm trying to host 2 applications in the same droplet in digitalocean by using nginx
But so far I've only been able to get the root application running (the one without example.com/secondapp)
I want to be able to not use a subdomain and just use example.com/secondmeteorapp to be able to access it.
My sites-enabled/default looks like this:
server {
listen 80;
#listen 443 ssl;
server_name example.com/;
#ssl_certificate /etc/nginx/ssl/ssl-bundle-myApp-domain-com.crt;
#ssl_certificate_key /etc/nginx/ssl/myApp_domain_com.key;
location /dragonfire {
proxy_pass http://localhost:4000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forward-Proto http;
proxy_set_header X-Nginx-Proxy true;
proxy_redirect off;
}
location /images {
alias /home/dragonfire-build/bundle/programs/web.browser/assets/images;
access_log off;
expires max;
}
location /fonts {
alias /home/dragonfire-build/bundle/programs/web.browser/assets/images;
access_log off;
expires max;
}
location "^/[a-z0-9]{40}\.(css|js)$" {
root /home/dragonfire-build/bundle/programs/web.browser;
access_log off;
expires max;
}
}
however, when I access http://serverIpAddress/dragonfire it can't find the css or javascript giving me this error:
GET http://myipaddress/1f3848edee9e199050b9b1965b9e697aa714b9f3.css?meteor_css_resource=true
GET http://myipaddress/6e48198c6b584ff79c86e0c624a65b4853faaf50.js?meteor_js_resource=true 404 (Not Found)
I can access the app if I go directly through the IP address and port but not via the nginx way
QUESTION
How can I access a second app using the same domain but with a /mysecondapp (in this case /dragonfire at the end?
I've set up an Nginx server which does a proxy_pass to a Node server if no html files are found. I can access my static html file just fine but when I hit the Node server I get a 403: Forbidden. Here is my Nginx conf if that helps:
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=one:8m max_size=3000m inactive=600m;
upstream mysite_upstream {
server 127.0.0.1:8000;
keepalive 64;
}
server {
listen 80;
server_name staging.mysite.org;
root /var/www/staging.mysite.org/public;
access_log /var/logs/staging.mysite.org.access.log;
error_log /var/logs/staging.mysite.org.error.log;
error_page 404 /404.html;
error_page 500 503 /500.html;
location ~ ^/(images/|javascript/|js/|css/|style/|flash/|robots.txt|sitemap.xml|humans.txt|favicon.ico) {
access_log off;
expires max;
}
location #node {
proxy_redirect off;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_set_header Connection "";
proxy_http_version 1.1;
proxy_cache one;
proxy_cache_key sfs$request_uri$scheme;
proxy_pass http://mysite_upstream;
}
location / {
try_files $uri $uri/ $uri.html #node;
}
}
I should mention that if I take out try_files and just proxy_pass directly from / the Node app is served up but I get a 404 for all my static html files. Also, this is working fine on my local machine (osx).
I have a nodejs server and SSL enabled nginx on 2 separate machines. Request/response all work properly, however I have some problems getting nginx to cache stuff. My server configuration is below. Initially, I had the proxy cache statement in the 'location /' block, and at the time it was caching only my index page. I read that nginx won't cache requests with set-cookie headers, so I ignored them as well (although it didn't stop my index page from getting cached earlier). I tried fiddling with this for a whole day, but couldn't get nginx to cache my js and css files. All such requests are getting routed back to my node server. Access logs and error logs don't have any unusual entries. What am I doing wrong?
server {
listen 443 ssl;
server_name example.com;
ssl_certificate /webserver/nginx/credentials/cert;
ssl_certificate_key /webserver/nginx/credentials/key;
ssl_session_cache shared:SSL:10m;
location ~ .*\.(ico|css|js|gif|jpe?g|png)$ {
proxy_pass http://somewhere:80;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_redirect http:// https://;
proxy_ignore_headers "Set-Cookie";
proxy_cache one;
proxy_cache_valid 200 1d;
proxy_cache_valid any 1m;
expires 7d;
proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504;
}
location / {
proxy_pass http://somewhere:80;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_redirect http:// https://;
}
}
This is what I'm using (I don't have SSL enabled but I don't think that is the problem). You're missing the try_files line that tells nginx to look for the files in the root before passing off to the proxy. Also, it's not really a caching problem - none of the static file requests should ever be hitting your node.js backend with this configuration.
server {
root /public;
listen 80;
server_name _;
index index.html index.htm;
charset utf-8;
# proxy request to node
location #proxy {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://127.0.0.1:3010;
proxy_redirect off;
break;
}
location / {
try_files $uri.html $uri $uri/ #proxy;
}
# static content
location ~ \.(?:ico|jpg|css|png|js|swf|woff|eot|svg|ttf|html|gif)$ {
access_log off;
log_not_found off;
add_header Pragma "public";
add_header Cache-Control "public";
expires 30d;
}
location ~ /\. {
access_log off;
log_not_found off;
deny all;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
error_page 404 /404.html;
location = /404.html {
}
}