I have a DigitalOcean droplet where Django project with Postgres/PostGIS database is running.
Suddenly it stopped working. When I want to connect to psql it says:
psql
psql: error: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I have no idea where is the problem.
Can this be an attack? I guess no as UFW wouldn't allow postgres connections from outside.
UFW
sudo ufw status
Status: active
To Action From
-- ------ ----
Nginx Full ALLOW Anywhere
22/tcp ALLOW Anywhere
Nginx Full (v6) ALLOW Anywhere (v6)
22/tcp (v6) ALLOW Anywhere (v6)
I'd like to know if the data are still there and I want to connect again.
I've rebooted the server and now I can't see anything except today in:
sudo tail -n 1000 /var/log/auth.log
I don't know where to start looking for the problem. Can you help me?
The last logs:
sudo tail -200 /var/log/postgresql/postgresql-12-main.log.1
2021-11-12 12:36:16.439 UTC [792] LOG: received fast shutdown request
2021-11-12 12:36:16.440 UTC [792] LOG: aborting any active transactions
2021-11-12 12:36:16.449 UTC [792] LOG: background worker "logical replication launcher" (PID 800) exited with exit code 1
2021-11-12 12:36:16.452 UTC [795] LOG: shutting down
2021-11-12 12:36:16.482 UTC [792] LOG: database system is shut down
/var/run/postgresql/
ls -la /var/run/postgresql/
total 0
drwxrwsr-x 3 postgres postgres 60 Nov 13 13:08 .
drwxr-xr-x 27 root root 900 Nov 13 13:23 ..
drwxr-s--- 2 postgres postgres 40 Nov 13 13:08 12-main.pg_stat_tmp
syslog | grep "postgres"
sudo tail -n 20000 /var/log/syslog | grep "postgres"
Nov 13 13:08:46 myproject-ubuntu kernel: [ 4.082735] systemd[1]: Created slice system-postgresql.slice.
Nov 13 13:08:46 myproject-ubuntu postgresql#12-main[773]: Error: Could not open logfile /var/log/postgresql/postgresql-12-main.log
Nov 13 13:08:46 myproject-ubuntu postgresql#12-main[708]: Error: /usr/lib/postgresql/12/bin/pg_ctl /usr/lib/postgresql/12/bin/pg_ctl start -D /var/lib/postgresql/12/main -l /var/log/postgresql/postgresql-12-main.log -s -o -c config_file="/etc/postgresql/12/main/postgresql.conf" exited with status 1:
Nov 13 13:08:46 myproject-ubuntu systemd[1]: postgresql#12-main.service: Can't open PID file /run/postgresql/12-main.pid (yet?) after start: Operation not permitted
Nov 13 13:08:46 myproject-ubuntu systemd[1]: postgresql#12-main.service: Failed with result 'protocol'.
Nov 13 13:18:26 myproject-ubuntu postgresql#12-main[1231]: Error: Could not open logfile /var/log/postgresql/postgresql-12-main.log
Nov 13 13:18:26 myproject-ubuntu postgresql#12-main[1215]: Error: /usr/lib/postgresql/12/bin/pg_ctl /usr/lib/postgresql/12/bin/pg_ctl start -D /var/lib/postgresql/12/main -l /var/log/postgresql/postgresql-12-main.log -s -o -c config_file="/etc/postgresql/12/main/postgresql.conf" exited with status 1:
Nov 13 13:18:26 myproject-ubuntu systemd[1]: postgresql#12-main.service: Can't open PID file /run/postgresql/12-main.pid (yet?) after start: Operation not permitted
Nov 13 13:18:26 myproject-ubuntu systemd[1]: postgresql#12-main.service: Failed with result 'protocol'.
Nov 13 13:24:27 myproject-ubuntu gunicorn[1326]: File "/home/master/.virtualenvs/xxx_django/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
Nov 13 13:24:27 myproject-ubuntu gunicorn[1326]: File "/home/master/.virtualenvs/xxx_django/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
Nov 13 13:24:27 myproject-ubuntu gunicorn[1326]: File "/home/master/.virtualenvs/xxx_django/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
Nov 13 13:24:27 myproject-ubuntu gunicorn[1326]: File "/home/master/.virtualenvs/xxx_django/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
Nov 13 13:24:27 myproject-ubuntu gunicorn[1326]: File "/home/master/.virtualenvs/xxx_django/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
Nov 13 13:24:27 myproject-ubuntu gunicorn[1326]: File "/home/master/.virtualenvs/xxx_django/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
Nov 13 13:24:27 myproject-ubuntu gunicorn[1326]: File "/home/master/.virtualenvs/xxx_django/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
Nov 13 13:24:27 myproject-ubuntu gunicorn[1326]: File "/home/master/.virtualenvs/xxx_django/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
Nov 13 13:47:42 myproject-ubuntu gunicorn[1330]: File "/home/master/.virtualenvs/xxx_django/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
Nov 13 13:47:42 myproject-ubuntu gunicorn[1330]: File "/home/master/.virtualenvs/xxx_django/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
Related
I have a simple python script that runs a sql query and converts the query results into csv (and parquet) leveraging pandas.to_csv(path/to/csv-file) (and pandas.to_parquet(path/to/parquet-file)
I'm using Docker Desktop in Windows 10. I have WSL installed -which I believe Docker Desktop somehow connects to that.
My python script 'main.py`:
import vertica_python as vp
import pandas as pd
from datetime import datetime as dt
connection_settings = {
'host': 'host.host',
'port': 5433,
'user': "user",
'password': "password",
'database': 'db'
}
def pull_data_vertica():
sql_file = "sample.sql"
with open(sql_file) as f:
query = f.read()
connection = vp.connect(**connection_settings)
temp = pd.read_sql(query, connection)
connection.close()
_pre = "/app/data_from_vertical_"
_dt = dt.now().strftime('%Y-%m-%d-%H-%M')
csv_out = _pre + _dt + '.csv'
temp.to_csv(csv_out)
parquet_out = _pre + _dt + '.parquet'
temp.to_parquet(parquet_out)
return csv_out, parquet_out
if __name__ == "__main__":
a, b = pull_data_vertica()
The Dcokerfile is as following:
FROM ubuntu:latest
RUN apt-get update
RUN apt-get install -y build-essential python3.8 python3-pip python3-dev cron vim busybox
WORKDIR /app
RUN pip3 -q install pip --upgrade
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
RUN rm requirements.txt
COPY crontab /etc/cron.d/crontab
RUN chmod 0644 /etc/cron.d/crontab
COPY script/main.py main.py
COPY script/sample.sql sample.sql
RUN chmod 0644 /app/main.py
RUN /usr/bin/crontab /etc/cron.d/crontab
CMD ["cron", "-f"]
here is my crontab:
* * * * * /usr/bin/python3 /app/main.py
I docker build the image, then docker run it in detached mode, and by docker exe ... bash I get into the container bash.
In there, I can manually execute my main.py and everything works. However, the cron job seems to be failing to produce the csv file, i.e., no file is written. I also don't have access to any log file to check if there is any issue with the cron job.
notes:
I see cron is the main process in the container (using top).
I did try to put the corntab content in a .sh script and execute that but all is the same with above: the .sh alone runs perfectly and files are written, however in the cronjob nothing happens.
I added shebang #!/usr/bin/env python3 on top of main.py script.
Update:
I installed busybox and modified the last line of Dockerfile to the following:
CMD busybox syslogd && cron -f
then in the container, I found the content /var/log/messages as following:
Oct 28 02:11:59 0ad085300c89 syslog.info syslogd started: BusyBox v1.30.1
Oct 28 02:11:59 0ad085300c89 cron.info cron[9]: (CRON) INFO (pidfile fd = 3)
Oct 28 02:11:59 0ad085300c89 cron.err cron[9]: Error: bad username; while reading /etc/cron.d/crontab
Oct 28 02:11:59 0ad085300c89 cron.info cron[9]: (*system*crontab) ERROR (Syntax error, this crontab file will be ignored)
Oct 28 02:11:59 0ad085300c89 cron.info cron[9]: (CRON) INFO (Running #reboot jobs)
Oct 28 02:12:01 0ad085300c89 authpriv.err CRON[10]: pam_env(cron:session): Unable to open env file: /etc/default/locale: No such file or directory
Oct 28 02:12:01 0ad085300c89 authpriv.info CRON[10]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 02:12:01 0ad085300c89 cron.info CRON[11]: (root) CMD (/usr/bin/python3 /app/main.py pull^M)
Oct 28 02:12:02 0ad085300c89 cron.info CRON[10]: (CRON) info (No MTA installed, discarding output)
Oct 28 02:12:02 0ad085300c89 authpriv.info CRON[10]: pam_unix(cron:session): session closed for user root
Oct 28 02:12:31 0ad085300c89 cron.info crontab[37]: (root) LIST (root)
Oct 28 02:13:01 0ad085300c89 authpriv.err CRON[40]: pam_env(cron:session): Unable to open env file: /etc/default/locale: No such file or directory
Oct 28 02:13:01 0ad085300c89 authpriv.info CRON[40]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 02:13:01 0ad085300c89 cron.info CRON[41]: (root) CMD (/usr/bin/python3 /app/main.py pull^M)
Oct 28 02:13:01 0ad085300c89 cron.info CRON[40]: (CRON) info (No MTA installed, discarding output)
Oct 28 02:13:01 0ad085300c89 authpriv.info CRON[40]: pam_unix(cron:session): session closed for user root
Oct 28 02:14:01 0ad085300c89 authpriv.err CRON[59]: pam_env(cron:session): Unable to open env file: /etc/default/locale: No such file or directory
Oct 28 02:14:01 0ad085300c89 authpriv.info CRON[59]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 02:14:01 0ad085300c89 cron.info CRON[60]: (root) CMD (/usr/bin/python3 /app/main.py pull^M)
Oct 28 02:14:01 0ad085300c89 cron.info CRON[59]: (CRON) info (No MTA installed, discarding output)
Oct 28 02:14:01 0ad085300c89 authpriv.info CRON[59]: pam_unix(cron:session): session closed for user root
Oct 28 02:15:01 0ad085300c89 authpriv.err CRON[81]: pam_env(cron:session): Unable to open env file: /etc/default/locale: No such file or directory
Oct 28 02:15:01 0ad085300c89 authpriv.info CRON[81]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 02:15:01 0ad085300c89 cron.info CRON[82]: (root) CMD (/usr/bin/python3 /app/main.py pull^M)
Oct 28 02:15:02 0ad085300c89 cron.info CRON[81]: (CRON) info (No MTA installed, discarding output)
Oct 28 02:15:02 0ad085300c89 authpriv.info CRON[81]: pam_unix(cron:session): session closed for user root
I am not sure I clearly get what cron is upset for but hope it helps.
Update 2:
Thanks to #jabbson 's comment I updfate the following:
absolute path to output file: /app/... instead of ./..
crontab is * * * * * root /usr/bin/python3 /app/main.py pull >> /app/exec.log 2>&1
here is the new /var/log/messages content:
Oct 28 03:10:36 898c63c3a4e6 syslog.info syslogd started: BusyBox v1.30.1
Oct 28 03:10:36 898c63c3a4e6 cron.info cron[10]: (CRON) INFO (pidfile fd = 3)
Oct 28 03:10:36 898c63c3a4e6 cron.info cron[10]: (CRON) INFO (Running #reboot jobs)
Oct 28 03:11:01 898c63c3a4e6 authpriv.info CRON[20]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 03:11:01 898c63c3a4e6 authpriv.info CRON[21]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 03:11:01 898c63c3a4e6 cron.info CRON[22]: (root) CMD (/usr/bin/python3 /app/main.py pull >> /app/exec.log 2>&1^M)
Oct 28 03:11:01 898c63c3a4e6 cron.info CRON[23]: (root) CMD (root /usr/bin/python3 /app/main.py pull >> /app/exec.log 2>&1^M)
Oct 28 03:11:01 898c63c3a4e6 cron.info CRON[20]: (CRON) info (No MTA installed, discarding output)
Oct 28 03:11:01 898c63c3a4e6 cron.info CRON[21]: (CRON) info (No MTA installed, discarding output)
Oct 28 03:11:01 898c63c3a4e6 authpriv.info CRON[21]: pam_unix(cron:session): session closed for user root
Oct 28 03:11:01 898c63c3a4e6 authpriv.info CRON[20]: pam_unix(cron:session): session closed for user root
Oct 28 03:12:01 898c63c3a4e6 authpriv.info CRON[25]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 03:12:01 898c63c3a4e6 authpriv.info CRON[26]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 03:12:01 898c63c3a4e6 cron.info CRON[27]: (root) CMD (/usr/bin/python3 /app/main.py pull >> /app/exec.log 2>&1^M)
Oct 28 03:12:01 898c63c3a4e6 cron.info CRON[28]: (root) CMD (root /usr/bin/python3 /app/main.py pull >> /app/exec.log 2>&1^M)
Oct 28 03:12:01 898c63c3a4e6 cron.info CRON[25]: (CRON) info (No MTA installed, discarding output)
Oct 28 03:12:01 898c63c3a4e6 cron.info CRON[26]: (CRON) info (No MTA installed, discarding output)
Oct 28 03:12:01 898c63c3a4e6 authpriv.info CRON[25]: pam_unix(cron:session): session closed for user root
Oct 28 03:12:01 898c63c3a4e6 authpriv.info CRON[26]: pam_unix(cron:session): session closed for user root
Oct 28 03:13:01 898c63c3a4e6 authpriv.info CRON[32]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 03:13:01 898c63c3a4e6 authpriv.info CRON[33]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 03:13:01 898c63c3a4e6 cron.info CRON[34]: (root) CMD (root /usr/bin/python3 /app/main.py pull >> /app/exec.log 2>&1^M)
Oct 28 03:13:01 898c63c3a4e6 cron.info CRON[35]: (root) CMD (/usr/bin/python3 /app/main.py pull >> /app/exec.log 2>&1^M)
Oct 28 03:13:01 898c63c3a4e6 cron.info CRON[33]: (CRON) info (No MTA installed, discarding output)
Oct 28 03:13:01 898c63c3a4e6 cron.info CRON[32]: (CRON) info (No MTA installed, discarding output)
Oct 28 03:13:01 898c63c3a4e6 authpriv.info CRON[32]: pam_unix(cron:session): session closed for user root
Oct 28 03:13:01 898c63c3a4e6 authpriv.info CRON[33]: pam_unix(cron:session): session closed for user root
AND NOTHING!!!!!! no log file in /app
Tried the same in Mac OS and it works in there. So this is not related to dataframe.to_csv (or dataframe.to_parquet) but I assume related to Docker Desktop on Windows and Docker Daemon.
I appreciate if you could kindly assist me with this issue.
Thanks,
I am trying to set up vsftpd on a Centos 7 server. We have a bunch of linux users with /usr/sbin/nologin shells just for the purpose of FTP. I also created a regular user testuser with a bash shell.
Anonymous logins are disabled. When I try to login to the FTP server through Chrome I get a password prompt. When I submit the password prompt I just get another password prompt, over and over. However, the contents of vsftpd's log file are as followed:
Wed Aug 5 10:32:05 2020 [pid 30282] CONNECT: Client "my.ip.goes.here"
Wed Aug 5 10:32:05 2020 [pid 30282] FTP response: Client "my.ip.goes.here", "220 SUP GUY"
Wed Aug 5 10:32:05 2020 [pid 30282] FTP command: Client "my.ip.goes.here", "USER anonymous"
Wed Aug 5 10:32:05 2020 [pid 30282] [anonymous] FTP response: Client "my.ip.goes.here", "331 Please specify the password."
Wed Aug 5 10:32:05 2020 [pid 30282] [anonymous] FTP command: Client "my.ip.goes.here", "PASS <password>"
Wed Aug 5 10:32:07 2020 [pid 30281] [anonymous] FAIL LOGIN: Client "my.ip.goes.here"
Wed Aug 5 10:32:08 2020 [pid 30282] [anonymous] FTP response: Client "my.ip.goes.here", "530 Login incorrect."
Wed Aug 5 10:32:08 2020 [pid 30282] FTP command: Client "my.ip.goes.here", "QUIT"
Wed Aug 5 10:32:08 2020 [pid 30282] FTP response: Client "my.ip.goes.here", "221 Goodbye."
Wed Aug 5 10:32:08 2020 [pid 30285] CONNECT: Client "my.ip.goes.here"
Wed Aug 5 10:32:08 2020 [pid 30285] FTP response: Client "my.ip.goes.here", "220 SUP GUY"
Wed Aug 5 10:32:08 2020 [pid 30285] FTP command: Client "my.ip.goes.here", "USER testuser"
Wed Aug 5 10:32:08 2020 [pid 30285] [testuser] FTP response: Client "my.ip.goes.here", "331 Please specify the password."
Wed Aug 5 10:32:08 2020 [pid 30285] [testuser] FTP command: Client "my.ip.goes.here", "PASS <password>"
Wed Aug 5 10:32:08 2020 [pid 30284] [testuser] OK LOGIN: Client "my.ip.goes.here"
As you can see, the last line is OK LOGIN which is funny because the browser sure isn't acting like I logged in successfully.
Here's my vsftpd.conf:
anonymous_enable=NO
local_enable=YES
dirmessage_enable=YES
xferlog_enable=YES
connect_from_port_20=YES
xferlog_std_format=YES
listen=YES
listen_ipv6=NO
pam_service_name=vsftpd
tcp_wrappers=YES
ssl_enable=NO
pasv_enable=YES
pasv_address=my.server.ip.here
pasv_min_port=49152
pasv_max_port=65535
ftpd_banner=SUP GUY
chroot_local_user=YES
chroot_list_enable=NO
allow_writeable_chroot=NO
write_enable=NO
userlist_enable=NO
log_ftp_protocol=YES
dual_log_enable=YES
Here's my /etc/pam.d/vsftpd file:
#%PAM-1.0
session optional pam_keyinit.so force revoke
auth required pam_listfile.so item=user sense=deny file=/etc/vsftpd/ftpusers onerr=succeed
auth required pam_nologin.so
auth include password-auth
account include password-auth
session required pam_loginuid.so
session include password-auth
My testuser account is not in that ftpusers file.
And here are the directory permissions of that testuser's home directory, in case that makes a difference:
total 16K
drwx------. 2 testuser testuser 91 Aug 5 10:27 .
drwxr-xr-x. 6 root root 65 Aug 4 10:42 ..
-rw-------. 1 testuser testuser 25 Aug 5 10:27 .bash_history
-rw-r--r--. 1 testuser testuser 18 Mar 31 21:17 .bash_logout
-rw-r--r--. 1 testuser testuser 193 Mar 31 21:17 .bash_profile
-rw-r--r--. 1 testuser testuser 231 Mar 31 21:17 .bashrc
-rw-rw-r--. 1 testuser testuser 0 Aug 5 10:27 hello```
Any idea what is going on here?
Turns out the culprit was SELinux. Everything was fine as far as vsftpd was concerned, but SELinux was blocking access to that home directory. My /var/log/audit/audit.log was full of entries like this:
type=AVC msg=audit(1596625942.966:385491): avc: denied { read } for pid=6778 comm="vsftpd" name="vsftpd"
dev="sda2" ino=2013664268 scontext=system_u:system_r:ftpd_t:s0-s0:c0.c1023
tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file permissive=0
In my case I do not need SELinux, so all I needed to do was set setenforce 0 and set SELINUX=disabled in my /etc/selinux/config.
I have a production LAMP server on Gentoo that's been running a personal intranet website. Since yesterday afternoon, it's hanging shortly after any connection; the relatively small login page loads fine, but upon logging in which involves a lot of MySQL queries and data, it hangs indefinitely.
SSH is still able to connect but oddly, that suddenly hangs too after a few pages' worth of characters are transmitted. Thus to get the below information, I have to keep logging back in. I tried /etc/init.d/apache2 restart and /etc/init.d/mysql restart, and then did a full reboot of the system; yet alas the problem persists. Details follow.
Top:
top - 12:23:52 up 1:34, 2 users, load average: 0.16, 0.09, 0.06 Tasks: 81 total, 1 running, 80 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st Mem: 3920788k total, 123476k used, 3797312k free, 4676k buffers Swap: 1227772k total, 0k used, 1227772k free, 48524k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 447 root 20 0 0 0 0 S 0 0.0 0:00.16 khubd
1 root 20 0 2020 640 568 S 0 0.0 0:00.51 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
5 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/u:0
6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
7 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
9 root 20 0 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
10 root 20 0 0 0 0 S 0 0.0 0:00.69 kworker/0:1
Apache logs show the usual hacking attempts:
# tail -50 /var/log/apache2/error_log
[Mon Mar 17 19:03:48 2014] [error] [client 116.58.240.169] File does not exist: /var/www/mysite/pma
[Mon Mar 17 19:03:48 2014] [error] [client 116.58.240.169] File does not exist: /var/www/mysite/myadmin
[Tue Mar 18 05:58:42 2014] [error] [client 202.53.8.82] File does not exist: /var/www/mysite/admin.cgi
[Tue Mar 18 07:19:42 2014] [error] [client 74.63.220.132] File does not exist: /var/www/mysite/phpTest
[Tue Mar 18 07:19:43 2014] [error] [client 74.63.220.132] File does not exist: /var/www/mysite/phpMyAdmin
[Tue Mar 18 07:19:43 2014] [error] [client 74.63.220.132] File does not exist: /var/www/mysite/pma
[Tue Mar 18 07:19:44 2014] [error] [client 74.63.220.132] File does not exist: /var/www/mysite/myadmin
[Tue Mar 18 08:24:16 2014] [error] [client 222.5.204.73] invalid request-URI \xcc\\\xa4/\x83\x8f\x90:\x84\x90\x0f\xc4\x8dfe\xecb\x94v\x1f[\xd7Z\x95$X\xaby\x13k\x88\xf2\xeb\xf7\x1b\xfc\xe8a\xff
[Tue Mar 18 08:29:49 2014] [error] [client 76.3.191.245] invalid request-URI
[Tue Mar 18 08:38:00 2014] [error] [client 35.2.240.149] invalid request-URI
[Tue Mar 18 08:50:52 2014] [error] [client 173.26.148.34] invalid request-URI
[Tue Mar 18 10:57:48 2014] [error] [client 110.175.79.216] invalid request-URI
[Tue Mar 18 10:57:53 2014] [error] [client 110.248.140.59] invalid request-URI D\xe8\x91a\xbc\xe5WZ\xd0C]\x9f~\xb5\x89\bd\x9e"[w,\xc6\xd9\xde\x8b]#JJ\xbf\x12
[Tue Mar 18 14:24:54 2014] [error] [client 108.14.2.113] invalid request-URI
[Tue Mar 18 14:40:08 2014] [error] [client 86.217.136.41] invalid request-URI \x94FI-\x02;4JVOV\x0f\xba\b
[Tue Mar 18 14:45:42 2014] [error] [client 98.119.127.76] invalid request-URI
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon-precomposed.png
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon.png
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon-120x120-precomposed.png
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon-120x120.png
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon-precomposed.png
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon.png
[Tue Mar 18 16:20:45 2014] [error] [client 103.24.32.14] File does not exist: /var/www/mysite/phpTest
[Tue Mar 18 16:20:46 2014] [error] [client 103.24.32.14] File does not exist: /var/www/mysite/phpMyAdmin
[Tue Mar 18 16:20:46 2014] [error] [client 103.24.32.14] File does not exist: /var/www/mysite/pma
[Tue Mar 18 16:20:46 2014] [error] [client 103.24.32.14] File does not exist: /var/www/mysite/myadmin
[Tue Mar 18 16:40:58 2014] [error] [client 122.170.93.35] invalid request-URI
[Tue Mar 18 16:57:54 2014] [error] [client 124.107.151.190] invalid request-URI
[Tue Mar 18 17:36:17 2014] [error] [client 68.147.250.90] invalid request-URI \x1d\x1e;&\x9e\xd2\xa8\xc2GNQ\\
[Tue Mar 18 23:38:20 2014] [error] [client 92.240.68.153] request failed: error reading the headers
[Wed Mar 19 02:52:43 2014] [error] [client 162.213.24.36] File does not exist: /var/www/mysite/CFIDE
[Wed Mar 19 06:26:06 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 06:26:07 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 06:26:07 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 06:26:09 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 06:26:15 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 07:48:28 2014] [error] [client 201.161.37.93] File does not exist: /var/www/crownware/manager
[Wed Mar 19 09:27:08 2014] [error] [client 113.184.228.73] invalid request-URI \xad_X\xdf\x9aIM6x\x01ti\xf6Ko\xebi
[Wed Mar 19 09:36:06 2014] [error] [client 162.213.24.36] File does not exist: /var/www/crownware/CFIDE
[Wed Mar 19 10:28:15 2014] [notice] caught SIGTERM, shutting down
[Wed Mar 19 10:28:17 2014] [notice] Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.0j PHP/5.4.6--pl0-gentoo configured -- resuming normal operations
[Wed Mar 19 10:43:31 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 10:43:31 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 10:43:35 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 10:43:35 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 10:43:36 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 10:47:16 2014] [notice] caught SIGTERM, shutting down
[Wed Mar 19 10:49:32 2014] [notice] Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.0j PHP/5.4.6--pl0-gentoo configured -- resuming normal operations
[Wed Mar 19 10:53:45 2014] [error] [client 65.60.209.141] Invalid URI in request \x13\xe0\x94\xc4\xa4o\xd1\xd3*\xe0\xe7\x1a\xce\xd9\xe8\t\xca\xc3k\x9f\xb0\x06\x13\xbcE\x17\xbb\x02\x9c:\xffD\x8d\x1f\x85Wv\x14\xfd\x8f\xe3k\xc6\xfe\xf7\x1bu
[Wed Mar 19 12:20:07 2014] [error] [client 173.24.52.209] invalid request-URI
Last message of interest from /var/log/mysql/mysqld.err (5 days ago):
140314 9:56:02 InnoDB: ERROR: the age of the last checkpoint is 9448765,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.
Versions:
# uname -a
Linux myhost 3.3.8-gentoo #1 SMP Fri Sep 28 09:34:42 MYT 2012 i686 Intel(R) Xeon(R) CPU E31220 # 3.10GHz GenuineIntel GNU/Linux
# mysqld -V
140319 12:37:13 [Warning] '--default-character-set' is deprecated and will be removed in a future release. Please use '--character-set-server' instead.
140319 12:37:13 [Warning] '--default-collation' is deprecated and will be removed in a future release. Please use '--collation-server' instead.
mysqld Ver 5.1.62-log for pc-linux-gnu on i686 (Gentoo Linux mysql-5.1.62-r1)
# apache2 -V
Server version: Apache/2.2.23 (Unix)
Server built: Oct 27 2012 19:17:52
Server's Module Magic Number: 20051115:31
Server loaded: APR 1.4.5, APR-Util 1.3.12
Compiled using: APR 1.4.5, APR-Util 1.3.12
Architecture: 32-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/usr"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="/var/run/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="/var/run/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="/etc/apache2/mime.types"
-D SERVER_CONFIG_FILE="/etc/apache2/httpd.conf"
# php -v
PHP 5.4.6--pl0-gentoo (cli) (built: Oct 27 2012 18:42:24)
Copyright (c) 1997-2012 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2012 Zend Technologies
Disk appears to have plenty of space still:
# df
Filesystem 1K-blocks Used Available Use% Mounted on
rootfs 960125048 84604800 826748732 10% /
udev 10240 0 10240 0% /dev
/dev/sda3 960125048 84604800 826748732 10% /
tmpfs 1960392 220 1960172 1% /run
rc-svcdir 1024 64 960 7% /lib/rc/init.d
cgroup_root 10240 0 10240 0% /sys/fs/cgroup
shm 1960392 0 1960392 0% /dev/shm
Apache processes:
# ps -ef|grep -i apache
root 2060 1 0 10:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2062 2060 0 10:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2066 2060 0 10:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2067 2060 0 10:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2068 2060 0 10:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2069 2060 0 10:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2070 2060 0 10:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2123 2060 0 10:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2124 2060 0 10:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2125 2060 0 10:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2148 2060 0 10:50 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2149 2060 0 10:50 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
Stracing the parent (root) process displays this repeatedly, not sure if it's normal:
# strace -p 2060
Process 2060 attached
select(0, NULL, NULL, NULL, {0, 669445}) = 0 (Timeout)
waitpid(-1, 0xbffb4b6c, WNOHANG|WSTOPPED) = 0
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
waitpid(-1, 0xbffb4b6c, WNOHANG|WSTOPPED) = 0
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
waitpid(-1, 0xbffb4b6c, WNOHANG|WSTOPPED) = 0
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
The fact that SSH also hangs after a couple thousand bytes indicates that I should be looking wider than Apache. What's the next step to diagnose?
It is obvious from the Apache log that your site was the target of typical exploit scripts that just try to bombard the server with requests for known applications looking for vulnerabilities.
This may have led to a compromise - this part is difficult to tell because you haven't detailed what other scripts are running on your machine.
I would suggest running a rootkit analyzer or similar on your server.
Also, this kind of question is better suited for serverfault.com as its not related to programming (what stackoverflow is about) but rather system administration/server management.
To prevent such requests from hitting your server, a WAF (Web Application Firewall) or other proxy is recommended which will throttle and block such requests before they reach your machine.
naxsi is a module for nginx that provides an open source WAF.
My localhost does not showing any previous contents it has, after normal system update.
What could be the possible reason for this.
However, I was able to connect to phpmyadmin.
If localhost location changed from /var/www/ to another, what could be the possible location?
System details:
Operating system - debian testing (Jessie) x86_64
Index of /
[ICO] Name Last modified Size Description
Apache/2.4.7 (Debian) Server at localhost Port 80
# /usr/sbin/apache2 -V
[Tue Mar 11 21:41:55.901363 2014] [core:warn] [pid 19737] AH00111: Config variable ${APACHE_LOCK_DIR} is not defined
[Tue Mar 11 21:41:55.901541 2014] [core:warn] [pid 19737] AH00111: Config variable ${APACHE_PID_FILE} is not defined
[Tue Mar 11 21:41:55.901569 2014] [core:warn] [pid 19737] AH00111: Config variable ${APACHE_RUN_USER} is not defined
[Tue Mar 11 21:41:55.901583 2014] [core:warn] [pid 19737] AH00111: Config variable ${APACHE_RUN_GROUP} is not defined
[Tue Mar 11 21:41:55.901616 2014] [core:warn] [pid 19737] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Tue Mar 11 21:41:55.932506 2014] [core:warn] [pid 19737] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Tue Mar 11 21:41:55.932942 2014] [core:warn] [pid 19737] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Tue Mar 11 21:41:55.932966 2014] [core:warn] [pid 19737] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
AH00526: Syntax error on line 74 of /etc/apache2/apache2.conf:
Invalid Mutex directory in argument file:${APACHE_LOCK_DIR}
# ps -ef | grep apache
root 16811 1 0 20:47 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 16815 16811 0 20:47 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 16816 16811 0 20:47 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 16817 16811 0 20:47 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 16818 16811 0 20:47 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 16819 16811 0 20:47 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 16820 16811 0 20:47 ? 00:00:00 /usr/sbin/apache2 -k start
This is possibly because your system's root has changed from /var/wwww to /var/www/html due to the system update.
To solve this,go to:
/etc/apache2/sites-available/000-default.conf
and set
ServerAdmin webmaster#localhost
DocumentRoot /var/www/html
You need root permission to edit the file. From terminal
sudo gedit /etc/apache2/sites-available/000-default.conf
and then edit the file and save.
After this restart your server from the terminal.
sudo service apache2 restart
I want to restart apache from a bash in linux (OpenSUSE).
I am trying to read the root password from a file (stdin). This is how I do it:
exec < /opt/otrsadm/stdin
read a1
sudo apache2ctl -k graceful
echo $a1
I get this in my log:
[Mon Sep 12 23:23:12 2011] [error] [client 192.168.1.101] Restarting apache
[Mon Sep 12 23:23:13 2011] [error] [client 192.168.1.101] sudo
[Mon Sep 12 23:23:13 2011] [error] [client 192.168.1.101] :
[Mon Sep 12 23:23:13 2011] [error] [client 192.168.1.101] no tty present and no askpass program specified
What's my problem and how do I fix it?
The sudo -S option reads the password from stdin.
The -S (stdin) option causes sudo to read the password from the
standard input instead of the terminal device.
echo "password" | sudo -S apache2ctl -l graceful