I think my webapp is pretty cool. It's a natural language playlist generator. It takes in a description of a playlist, like:
"midwest emo songs to cry to in the shower because my girlfriend broke up with me"
and converts it into an embedding generated by a NLP transformer model (specifically SentenceTransformers) and does recommender system stuff to return songs in a playlist for a user.
My website hangs after the user has submitted their description, and I get a 504 load balancer error after 5 minutes. After tracing where the code hangs, it seems to stop during model.encode(text), which runs the user's query through the ML model to get the embedding.
This code runs no problem on my local machine, and when I run it in the console it also has no problem processing the text through the ML model.
What should I do? Add more workers? Free up space in the program? Let me know.
Below are my server logs after model.encode() is run.
2022-11-26 07:53:26 entered the get embedding function
2022-11-26 07:53:27 announcing my loyalty to the Emperor...
2022-11-26 07:54:11 Sat Nov 26 07:54:10 2022 - HARAKIRI ON WORKER 4 (pid: 18, try: 1)
2022-11-26 07:54:11 Sat Nov 26 07:54:10 2022 - HARAKIRI !!! worker 4 status !!!
2022-11-26 07:54:11 Sat Nov 26 07:54:10 2022 - HARAKIRI [core 0] 10.0.0.75 - POST / since 1669448649
2022-11-26 07:54:11 Sat Nov 26 07:54:10 2022 - HARAKIRI !!! end of worker 4 status !!!
2022-11-26 07:54:11 DAMN ! worker 4 (pid: 18) died, killed by signal 9 :( trying respawn ...
2022-11-26 07:54:11 Respawned uWSGI worker 4 (new pid: 33)
2022-11-26 07:54:11 spawned 2 offload threads for uWSGI worker 4
2022-11-26 08:03:28 Sat Nov 26 08:03:27 2022 - HARAKIRI ON WORKER 3 (pid: 15, try: 1)
2022-11-26 08:03:28 Sat Nov 26 08:03:27 2022 - HARAKIRI !!! worker 3 status !!!
2022-11-26 08:03:28 Sat Nov 26 08:03:27 2022 - HARAKIRI [core 0] 10.0.0.75 - POST / since 1669449206
2022-11-26 08:03:28 Sat Nov 26 08:03:27 2022 - HARAKIRI !!! end of worker 3 status !!!
2022-11-26 08:03:28 DAMN ! worker 3 (pid: 15) died, killed by signal 9 :( trying respawn ...
2022-11-26 08:03:28 Respawned uWSGI worker 3 (new pid: 36)
2022-11-26 08:03:28 spawned 2 offload threads for uWSGI worker 3
I tried running this code in the console of pythonanywhere, and it ran just fine. I'm stuck!
I used an always on task to run the queries through the model and spit them back into the main script.
Works like a dream!
Related
cassandra.service: main process exited, code=killed, status=11/SEGV
env:
apache-cassandra-4.0.0
jdk-11.0.12
ZGC
jvm:
-Xms31G
-Xmx31G
host:
16core 128G
/var/log/message:
Jul 4 13:57:10 iZ2zec1q29sosy4bdv893qZ systemd-logind: Removed session 277.
Jul 4 13:57:12 iZ2zec1q29sosy4bdv893qZ cassandra: INFO [CompactionExecutor:4] 2022-07-04 13:57:12,074 CompactionTask.java:245 - Compacted (24af5250-fb5e-11ec-aa2a-6b96728ba428)
4 sstables to [/data/cassandra/data/data/spaceport/xm_coupon_code_realtime1-d77e7f10ebcc11ecae252faeea3c28c4/nb-6494-big,] to level=0. 27.414MiB to 27.412MiB (~99% of original) i
n 1,812ms. Read Throughput = 15.127MiB/s, Write Throughput = 15.126MiB/s, Row Throughput = ~123,625/s. 32,718 total partitions merged to 32,689. Partition merge counts were {1:
32663, 2:23, 3:3, }
Jul 4 13:57:12 iZ2zec1q29sosy4bdv893qZ cassandra: INFO [NonPeriodicTasks:1] 2022-07-04 13:57:12,083 SSTable.java:111 - Deleting sstable: /data/cassandra/data/data/spaceport/xm_c
oupon_code_realtime1-d77e7f10ebcc11ecae252faeea3c28c4/nb-6490-big
Jul 4 13:57:12 iZ2zec1q29sosy4bdv893qZ cassandra: INFO [NonPeriodicTasks:1] 2022-07-04 13:57:12,084 SSTable.java:111 - Deleting sstable: /data/cassandra/data/data/spaceport/xm_c
oupon_code_realtime1-d77e7f10ebcc11ecae252faeea3c28c4/nb-6493-big
Jul 4 13:57:12 iZ2zec1q29sosy4bdv893qZ cassandra: INFO [NonPeriodicTasks:1] 2022-07-04 13:57:12,085 SSTable.java:111 - Deleting sstable: /data/cassandra/data/data/spaceport/xm_c
oupon_code_realtime1-d77e7f10ebcc11ecae252faeea3c28c4/nb-6491-big
Jul 4 13:57:12 iZ2zec1q29sosy4bdv893qZ cassandra: INFO [NonPeriodicTasks:1] 2022-07-04 13:57:12,085 SSTable.java:111 - Deleting sstable: /data/cassandra/data/data/spaceport/xm_c
oupon_code_realtime1-d77e7f10ebcc11ecae252faeea3c28c4/nb-6492-big
Jul 4 14:00:01 iZ2zec1q29sosy4bdv893qZ systemd: Started Session 293 of user root.
Jul 4 14:01:01 iZ2zec1q29sosy4bdv893qZ systemd: Started Session 294 of user root.
Jul 4 14:01:59 iZ2zec1q29sosy4bdv893qZ systemd: cassandra.service: main process exited, code=killed, status=11/SEGV
Jul 4 14:02:00 iZ2zec1q29sosy4bdv893qZ systemd: Unit cassandra.service entered failed state.
Jul 4 14:02:00 iZ2zec1q29sosy4bdv893qZ systemd: cassandra.service failed.
Jul 4 14:02:05 iZ2zec1q29sosy4bdv893qZ systemd: cassandra.service holdoff time over, scheduling restart.
Jul 4 14:02:05 iZ2zec1q29sosy4bdv893qZ systemd: Stopped Cassandra Server Service.
Jul 4 14:02:05 iZ2zec1q29sosy4bdv893qZ systemd: Started Cassandra Server Service.
Jul 4 14:02:55 iZ2zec1q29sosy4bdv893qZ cassandra: CompileCommand: dontinline org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset(Lorg/apache/cassandra/io/util/DataI
nputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns;
The log entries you posted on their own don't explain what the problem is. You will need to review the Cassandra system.log for clues.
A friendly reminder that Stack Overflow is for getting help with coding, algorithm, or programming language problems. For future reference, you should post DB admin/ops questions on dba.stackexchange.com. If you post it there, I'd be happy to help. Cheers!
I just updated my arch linux to gnome 3.22 and it broke many things.
Most annoying is that gnome-terminal won't start.
I checked journalctl and found that :
oct. 18 15:11:05 jarvis dbus-daemon[727]: Successfully activated service 'org.gnome.Terminal'
oct. 18 15:11:05 jarvis systemd[711]: Started GNOME Terminal Server.
oct. 18 15:11:06 jarvis org.gnome.Shell.desktop[745]: Window manager warning: Could not import pending buffer, ignoring commit: Failed to create texture 2d due to size/format constraints
oct. 18 15:11:06 jarvis gnome-terminal-[1569]: Error flushing display: Broken pipe
oct. 18 15:11:06 jarvis kernel: traps: gnome-terminal-[1569] trap int3 ip:7f21f3389ff1 sp:7ffd9c3e2bb0 error:0
oct. 18 15:11:06 jarvis systemd[1]: Started Process Core Dump (PID 1592/UID 0).
oct. 18 15:11:06 jarvis systemd[711]: gnome-terminal-server.service: Main process exited, code=dumped, status=5/TRAP
oct. 18 15:11:06 jarvis systemd[711]: gnome-terminal-server.service: Unit entered failed state.
oct. 18 15:11:06 jarvis systemd[711]: gnome-terminal-server.service: Failed with result 'core-dump'.
oct. 18 15:11:06 jarvis systemd-coredump[1593]: Process 1569 (gnome-terminal-) of user 1000 dumped core.
Stack trace of thread 1569:
#0 0x00007f21f3389ff1 n/a (libglib-2.0.so.0)
#1 0x00007f21f338c731 g_log_writer_default (libglib-2.0.so.0)
#2 0x00007f21f338ab8c g_log_structured_array (libglib-2.0.so.0)
...
Going on and on on different libc.so.6 calls.
I can't find how to fix this.
Does any one have an idea of what happened ? and of what I should do ?
Try switch login screen to use Xorg,
edit file: /etc/gdm/custom.conf and uncomment WaylandEnable=false:
[daemon]
# Uncoment the line below to force the login screen to use Xorg
# WaylandEnable=false
It seems like sge tries start before lustre is mounted when the server boots, which brings an error to start automatically when it reboots.
Can somebody tell me how to change the order when it boots, so sge starts after lustre is mounted?
Error message from the log:
Aug 12 11:46:21 dragen1 systemd: Configuration file /usr/lib/systemd/system/sge_execd.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Aug 12 11:46:40 dragen1 sge_execd: error: SGE_ROOT directory "/cm/shared/apps/sge/2011.11p1" doesn't exist
Aug 12 11:46:40 dragen1 systemd: sge_execd.service: control process exited, code=exited status=1
Aug 12 11:46:40 dragen1 systemd: Unit sge_execd.service entered failed state.
Aug 12 11:46:40 dragen1 systemd: sge_execd.service failed
I added in the following under [Unit] from the sge service
RequiresMountsFor=(Mount Point)
This fixed the problem.
When I try to start the apache this happens:
Job for apache2.service failed. See ‘systemctl status apache2.service’ and ‘journalctl -xn’ for details.
system.ctl status apache2.service -l returns me this:
Mar 24 23:41:57 glauber-pc.site start_apache2[3249]: httpd2-prefork: Syntax error on line 179 of /etc/apache2/httpd.conf: Syntax error on line 102 of /etc/apache2/default-server.conf: Syntax error on line 1 of /etc/apache2/conf.d/mod_evasive.conf: Cannot load /usr/lib64/apache2/mod_evasive20.so into server: /usr/lib64/apache2/mod_evasive20.so: cannot open shared object file: No such file or directory
Mar 24 23:41:57 glauber-pc.site systemd[1]: apache2.service: main process exited, code=exited, status=1/FAILURE
Mar 24 23:41:57 glauber-pc.site systemd[1]: Failed to start The Apache Webserver.
Mar 24 23:41:57 glauber-pc.site systemd[1]: Unit apache2.service entered failed state.
Everything seems right in those lines, any clue what it can be?
Change the first line in
/etc/apache2/conf.d/mod_evasive.conf
to
LoadModule evasive20_module /usr/lib64/apache2/mod_evasive24.so
(...24.so instead of ...20.so)
I am trying to set up nginx to work with uwsgi, I am on Arch Linux. I have /etc/uwsgi/emperor.ini and /etc/uwsgi/vassals/myapp.ini. My emperor.ini looks like:
[uwsgi]
emperor = /etc/uwsgi/vassals
emperor-pidfile = /run/uwsgi/uwsgi.pid
uid = http
gid = http
logto = /var/log/uwsgi/uwsgi.log
myapp.ini
[uwsgi]
chdir = /srv/http/myapp/src/
wsgi-file = run.py
callable = app
processes = 4
threads = 2
offload-threads = 2
stats = 127.0.0.1:9191
max-requests = 5000
master = True
vacuum = True
socket = 127.0.0.1:3031
enable-threads = true
harakiri = 60
logto = /var/log/uwsgi/myapp.log
uwsgi.service [for systemd]
[Unit]
Description=uWSGI Emperor
After=syslog.target
[Service]
PIDFile=/run/uwsgi/uwsgi.pid
ExecStartPre=/bin/mkdir -p /run/uwsgi
ExecStartPre=/bin/chown http:http /run/uwsgi
ExecStart=/usr/bin/uwsgi --ini /etc/uwsgi/emperor.ini
Restart=always
Type=notify
StandardError=syslog
NotifyAccess=main
[Install]
WantedBy=multi-user.target
Yet when the uwsgi.service runs I get:
uwsgi.service - uWSGI Emperor
Loaded: loaded (/etc/systemd/system/uwsgi.service; disabled)
Active: active (running) since Sun 2014-02-02 04:37:04 PST; 34min ago
Process: 31537 ExecStartPre=/bin/chown http:http /run/uwsgi (code=exited, status=0/SUCCESS)
Process: 31535 ExecStartPre=/bin/mkdir -p /run/uwsgi (code=exited, status=0/SUCCESS)
Main PID: 31540 (uwsgi)
Status: "The Emperor is governing 0 vassals"
CGroup: /system.slice/uwsgi.service
ââ31540 /usr/bin/uwsgi --ini /etc/uwsgi/emperor.ini
Feb 02 04:37:04 MyMachine uwsgi[31540]: [uWSGI] getting INI configuration from /etc/uwsgi/emperor.ini
Feb 02 04:37:04 MyMachine systemd[1]: Started uWSGI Emperor.
That bit right there? "The Emperor is governing 0 vassals" for some reason it will not look at the myapp.ini in the vassal directory. This is what I get in a command line "log":
*** Starting uWSGI 2.0 (64bit) on [Sun Feb 2 05:27:37 2014] ***
compiled with version: 4.8.2 20131219 (prerelease) on 09 January 2014 11:43:55
os: Linux-3.12.7-2-ARCH #1 SMP PREEMPT Sun Jan 12 13:09:09 CET 2014
nodename: MyMachine
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 2
current working directory: /home/me
detected binary path: /usr/bin/uwsgi
your processes number limit is 26743
your memory page size is 4096 bytes
*** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers ***
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:3031 fd 3
Python version: 2.7.6 (default, Nov 26 2013, 12:52:49) [GCC 4.8.2]
Python main interpreter initialized at 0xcfd340
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 415280 bytes (405 KB) for 8 cores
*** Operational MODE: preforking+threaded ***
WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0xcfd340 pid: 32089 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 32089)
spawned uWSGI worker 1 (pid: 32092, cores: 2)
spawned 2 offload threads for uWSGI worker 1
spawned uWSGI worker 2 (pid: 32095, cores: 2)
spawned uWSGI worker 3 (pid: 32097, cores: 2)
spawned 2 offload threads for uWSGI worker 3
spawned uWSGI worker 4 (pid: 32100, cores: 2)
spawned 2 offload threads for uWSGI worker 2
spawned 2 offload threads for uWSGI worker 4
I cannot seem to get uwsgi to run see its vassals no matter what I do. Thanks.
EDIT
OK so I decided to go back to the command line so I don't have to deal with log and pid permission errors and I noticed something strange. When I make write out uwsgi --ini /etc/uwsgi/emperor.ini the entire thing works, I can access the site through nginx, beautiful, but I saw this in the console output:
*** Starting uWSGI 2.0 (64bit) on [Sun Feb 2 15:19:43 2014] ***
compiled with version: 4.8.2 20131219 (prerelease) on 09 January 2014 11:43:55
os: Linux-3.12.9-2-ARCH #1 SMP PREEMPT Fri Jan 31 10:22:54 CET 2014
nodename: MyMachine
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 2
current working directory: /home/me
detected binary path: /usr/bin/uwsgi
*** WARNING: you are running uWSGI without its master process manager ***
your processes number limit is 26743
your memory page size is 4096 bytes
detected max file descriptor number: 1024
*** starting uWSGI Emperor ***
*** has_emperor mode detected (fd: 5) ***
[uWSGI] getting INI configuration from /etc/uwsgi/emperor.ini
*** Starting uWSGI 2.0 (64bit) on [Sun Feb 2 15:19:43 2014] ***
compiled with version: 4.8.2 20131219 (prerelease) on 09 January 2014 11:43:55
os: Linux-3.12.9-2-ARCH #1 SMP PREEMPT Fri Jan 31 10:22:54 CET 2014
nodename: MyMachine
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 2
current working directory: /home/me
detected binary path: /usr/bin/uwsgi
your processes number limit is 26743
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
*** starting uWSGI sub-Emperor ***
Python version: 2.7.6 (default, Nov 26 2013, 12:52:49) [GCC 4.8.2]
*** has_emperor mode detected (fd: 7) ***
[uWSGI] getting INI configuration from myapp.ini
*** Starting uWSGI 2.0 (64bit) on [Sun Feb 2 15:19:43 2014] ***
compiled with version: 4.8.2 20131219 (prerelease) on 09 January 2014 11:43:55
os: Linux-3.12.9-2-ARCH #1 SMP PREEMPT Fri Jan 31 10:22:54 CET 2014
nodename: MyMachine
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 2
current working directory: /etc/uwsgi/vassals
detected binary path: /usr/bin/uwsgi
your processes number limit is 26743
your memory page size is 4096 bytes
*** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers ***
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:3031 fd 3
Python version: 2.7.6 (default, Nov 26 2013, 12:52:49) [GCC 4.8.2]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x2403ab0
your mercy for graceful operations on workers is 60 seconds
*** Operational MODE: no-workers ***
spawned uWSGI master process (pid: 1640)
Sun Feb 2 15:19:43 2014 - [emperor] vassal /etc/uwsgi/emperor.ini has been spawned
Python main interpreter initialized at 0x17682a0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 415280 bytes (405 KB) for 8 cores
*** Operational MODE: preforking+threaded ***
WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x17682a0 pid: 1642 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 1642)
Sun Feb 2 15:19:45 2014 - [emperor] vassal myapp.ini has been spawned
spawned uWSGI worker 1 (pid: 1645, cores: 2)
spawned 2 offload threads for uWSGI worker 1
spawned uWSGI worker 2 (pid: 1648, cores: 2)
Sun Feb 2 15:19:45 2014 - [emperor] vassal myapp.ini is ready to accept requests
spawned 2 offload threads for uWSGI worker 2
spawned uWSGI worker 3 (pid: 1651, cores: 2)
spawned 2 offload threads for uWSGI worker 3
spawned uWSGI worker 4 (pid: 1657, cores: 2)
*** Stats server enabled on 127.0.0.1:9191 fd: 16 ***
spawned 2 offload threads for uWSGI worker 4
[pid: 1648|app: 0|req: 1/1] 192.168.1.156 () {40 vars in 795 bytes} [Sun Feb 2 15:20:40 2014] GET /articles => generated 2943 bytes in 164 msecs (HTTP/1.1 200) 3 headers in 283 bytes (1 switches on core 0)
announcing my loyalty to the Emperor...
Sun Feb 2 15:20:40 2014 - [emperor] vassal myapp.ini is now loyal
[pid: 1657|app: 0|req: 1/2] 192.168.1.156 () {40 vars in 792 bytes} [Sun Feb 2 15:20:41 2014] GET /index => generated 3216 bytes in 184 msecs (HTTP/1.1 200) 3 headers in 283 bytes (1 switches on core 0)
announcing my loyalty to the Emperor...
Sun Feb 2 15:20:42 2014 - [emperor] vassal myapp.ini is now loyal
I started noticing some odd ball behavior in the output. I finally came down to this configuration which let's me see my site.
emperor.ini
[uwsgi]
emperor = /etc/uwsgi/vassals
master = true
#gid = http
#uid = http
I commented out the gid and uid because with them there I get a bad gateway response when I try to connect to my website, it warns me about it in the log uWSGI running as root, you can use --uid/--gid/--chroot options but I am not sure what to do about it.
myapp.ini
[uwsgi]
chdir = /srv/http/myapp/src/
wsgi-file = run.py
callable = app
processes = 4
threads = 2
offload-threads = 2
stats = 127.0.0.1:9191
max-requests = 5000
master = true
vacuum = true
socket = 127.0.0.1:3031
harakiri = 60
uwsgi.service
[Unit]
Description=uWSGI Emperor
After=syslog.target
[Service]
ExecStart=/usr/bin/uwsgi --ini /etc/uwsgi/emperor.ini --enable-threads
Restart=always
KillSignal=SIGQUIT
Type=notify
StandardError=syslog
NotifyAccess=main
[Install]
WantedBy=multi-user.target
No idea what to do about it running as root.
create a vassals in /etc/uwsgi/vassals/