I have a error with uwsgi
when i start my config - uwsgi bottle.ini
!!! no internal routing support, rebuild with pcre support !!!
setgid() to 80
setuid() to 80
your processes number limit is 5547
your memory page size is 4096 bytes
detected max file descriptor number: 58982
lock engine: ipcsem
uwsgi_lock_ipcsem_init()/semget(): No space left on device [core/lock.c line 507]
uwsgi_ipcsem_clear()/semctl(): Invalid argument [core/lock.c line 631]
my bottle.ini
[uwsgi]
socket = 185.21.214.275:80
chdir = /usr/local/www/myapp/
virtualenv = /usr/local/www/mypython
master = true
wsgi-file = /usr/local/www/myapp/app.py
uid = www
gid = www
I have had reinstalled uwsgi and pcre but proble is still appeare
It is explained here: http://uwsgi-docs.readthedocs.org/en/latest/ThingsToKnow.html
On OpenBSD, NetBSD and FreeBSD < 9, SysV IPC semaphores are used as the locking subsystem. These operating systems tend to limit the number of allocable semaphores to fairly small values. You should raise the default limits if you plan to run more than one uWSGI instance. FreeBSD 9 has POSIX semaphores, so you do not need to bother with that.
Related
I have a program calling madvise [1] system call to give OS hint about how the memory will be accessed.
When randomly accessing an mmapped file, some conflict advise is given to OS:
197 mmap(NULL, 125554026, PROT_READ, MAP_PRIVATE, 6, 0) = 0x7f023f242000
198 madvise(0x7f023f242000, 125554026, MADV_RANDOM) = 0
199 madvise(0x7f023f242000, 6, MADV_WILLNEED) = 0
200 madvise(0x7f02469fe000, 3434, MADV_WILLNEED) = 0
.....
204 madvise(0x7f023f242000, 928, MADV_WILLNEED) = 0
205 madvise(0x7f023f242000, 125553424, MADV_WILLNEED) = 0
As you can see:
#198 system call madvise tells OS to randomly access this file
but #205 system call madvise tells OS that almost entire file is WILLNEED (asking for prefetch)
I am not sure what the expected behavior is for Linux kernel in this case.
I have two different deployment environments:
one is CentOS 7.6 with Linux Kernel 3.10.0
and the other is Ubuntu 20.04 with Linux Kernel 5.4.0
And I observed different behaviors:
For CentOS 7.6, the entire file is pre-loaded to page cache (#205 system call wins)
while for Ubuntu 20.04, only the randomly accessed parts are loaded to page cache (#198 system call wins).
I don't know where to find such information about this behavior when two madvise system call give conflict hints. Could some one shed some light on this? Thanks.
References
[1] https://linux.die.net/man/2/madvise
I am in the process of setting up a Riak Cluster on Raspberry Pis.
Unfortunately I get the following error message after changing the IP address.
Versions I used:
Debian Jessie (Raspberry PI)
riak (Github Clone Mar2017)
riak-cs2.1.1
stanchion-2.1.1
Using this guide I tried to change the IP addresses in the various .conf files.
https://docs.riak.com/riak/kv/latest/using/cluster-operations/changing-cluster-info/index.html
Works on 127.0.0.1:
$ ~/riak/rel/riak/bin/riak-admin test
Successfully completed 1 read/write cycle to 'riak#127.0.0.1'
Error Message (after changing IP:192.168.178.61):
sudo ./riak console
config is OK
-config /home/pi/neu/riak/rel/riak/data/generated.configs/app.2020.01.02.23.37.52.config -args_file /home/pi/neu/riak/rel/riak/data/generated.configs/vm.2020.01.02.23.37.52.args -vm_args /home/pi/neu/riak/rel/riak/data/generated.configs/vm.2020.01.02.23.37.52.args
Exec: /home/pi/neu/riak/rel/riak/bin/../erts-5.10.3/bin/erlexec -boot /home/pi/neu/riak/rel/riak/bin/../releases/2.2.3/riak -config /home/pi/neu/riak/rel/riak/data/generated.configs/app.2020.01.02.23.37.52.config -args_file /home/pi/neu/riak/rel/riak/data/generated.configs/vm.2020.01.02.23.37.52.args -vm_args /home/pi/neu/riak/rel/riak/data/generated.configs/vm.2020.01.02.23.37.52.args -pa /home/pi/neu/riak/rel/riak/bin/../lib/basho-patches -- console
Root: /home/pi/neu/riak/rel/riak/bin/..
Erlang R16B02_basho10 (erts-5.10.3) [source] [smp:4:4] [async-threads:64] [hipe] [kernel-poll:true] [frame-pointer]
[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
{"Kernel pid terminated",application_controller,"{application_start_failure,riak_core,{bad_return,{{riak_core_app,start,[normal,[]]},{'EXIT',{{function_clause,[{orddict,fetch,['riak#192.168.178.61',[]],[{file,\"orddict.erl\"},{line,72}]},{riak_core_capability,renegotiate_capabilities,1,[{file,\"src/riak_core_capability.erl\"},{line,441}]},{riak_core_capability,handle_call,3,[{file,\"src/riak_core_capability.erl\"},{line,213}]},{gen_server,handle_msg,5,[{file,\"gen_server.erl\"},{line,585}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,239}]}]},{gen_server,call,[riak_core_capability,{register,{riak_core,vnode_routing},{capability,[proxy,legacy],legacy,{riak_core,legacy_vnode_routing,[{true,legacy},{false,proxy}]}}},infinity]}}}}}}"}
Crash dump was written to: ./log/erl_crash.dump
Kernel pid terminated (application_controller) ({application_start_failure,riak_core,{bad_return,{{riak_core_app,start,[normal,[]]},{'EXIT',{{function_clause,[{orddict,fetch,['riak#192.168.178.61',[
https://github.com/basho/riak/issues/999
martinsumner commented 3 days ago:
I might expect to see this if you hadn't done the step of either renaming (or deleting the contents of) the ring directory. Did you do this?
Also can you confirm if you're in the single-node or multi-node renaming scenario?
Ei3rb0mb3r commented 1 minute ago:
Many thanks for the quick feedback!
The error has been solved after I deleted the ring directory files.
../riak/rel/riak/data/ring/ rm -rf *
Python 3.6.8 to create venv
uwsgi installed in venv and is properly symlinked and using the correct python version
nginx installed system wide
just trying to get the most basic uwsgi --ini myinifile.ini to function properly.
Despite days of trying to fix this problem I always end back up here, with my django application not being recognized by uwsgi when I launch it.
uwsgi works properly when uwsgi --uwsgi-file test.py --http-socket :8001
I can python manage.py runserver 0.0.0.0:8001 just fine and django launches the application with 0 permission issues on navigating the application website.
I've seen this error so many times I think I'm taking crazy pills and missing something silly and obvious.
[uWSGI] getting INI configuration from /home/saleor/saleor/saleor/wsgi/uwsgi.ini
[uwsgi-static] added mapping for /static => /app/static
*** Starting uWSGI 2.0.18 (64bit) on [Mon Nov 18 07:36:36 2019] ***
compiled with version: 7.4.0 on 18 November 2019 04:47:07
os: Linux-4.15.0 #1 SMP Thu Aug 23 19:33:51 MSK 2018
nodename: myurl.com
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 2
current working directory: /home/saleor/saleor/venv/bin
detected binary path: /home/saleor/saleor/venv/bin/uwsgi
chdir() to /home/saleor/saleor/saleor
your processes number limit is 62987
your memory page size is 4096 bytes
detected max file descriptor number: 1024
building mime-types dictionary from file /etc/mime.types...554 entry found
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address :8001 fd 3
Python version: 3.6.8 (default, Oct 7 2019, 12:59:55) [GCC 8.3.0]
PEP 405 virtualenv detected: /home/saleor/saleor/venv
Set PythonHome to /home/saleor/saleor/venv
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x558314cc3dd0
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145808 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
ModuleNotFoundError: No module named 'saleor'
unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. going in full dynamic mode ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 7969)
spawned uWSGI worker 1 (pid: 7970, cores: 1)
^CSIGINT/SIGQUIT received...killing workers...
worker 1 buried after 1 seconds
goodbye to uWSGI.
This is my uwsgi.py file named:__init__.py
"""WSGI config for saleor project.
This module contains the WSGI application used by Django's development server
and any production WSGI deployments. It should expose a module-level variable
named ``application``. Django's ``runserver`` and ``runfcgi`` commands discover
this application via the ``WSGI_APPLICATION`` setting.
Usually you will have the standard Django WSGI application here, but it also
might make sense to replace the whole Django WSGI application with a custom one
that later delegates to the Django one. For example, you could introduce WSGI
middleware here, or combine a Django application with an application of another
framework.
"""
from django.core.wsgi import get_wsgi_application
from saleor.wsgi.health_check import health_check
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "saleor.settings")
# This application object is used by any WSGI server configured to use this
# file. This includes Django's development server, if the WSGI_APPLICATION
# setting points here.
application = get_wsgi_application()
# Apply WSGI middleware here.
# from helloworld.wsgi import HelloWorldApplication
# application = HelloWorldApplication(application)
application = health_check(application, "/health/")
And this is my uwsgi.ini
[uwsgi]
die-on-term = true
http-socket = :$(PORT)
log-format = UWSGI uwsgi "%(method) %(uri) %(proto)" %(status) %(size) %(msecs)ms [PID:%(pid):Worker-%(wid)] [RSS:%(rssM)MB]
master = true
max-requests = 100
memory-report = true
module = saleor.wsgi:application
processes = 1
static-map = /static=/app/static
mimefile = /etc/mime.types
env = /home/saelor/saleor/venv
virtualenv = /home/saleor/saleor/venv
chdir = /home/saleor/saleor/saleor
master = true
vacuum = true
chmod-socket = 666
uid = saleor
gid = saleor
I'm just trying to get uwsgi to run my saleor app so I can start to work on getting it and nginx to communicate properly :( I'm lost and my brain seriously hurts.
update to python 3.8
install uwsgi inside your venv
uwsgi --http :8000 --module saleor.wsgi
do not touch any other settings
I have got this error and I solved it this way
Install uwsgi inside your venv
If it is already installed, then deactivate your virtual
environment, then again activate it.
3.uwsgi --http :8000 --module saleor.wsgi
I am running MPI job in linux server. I got error:
--------------------------------------------------------------------------
The OpenFabrics (openib) BTL failed to initialize while trying to
allocate some locked memory. This typically can indicate that the
memlock limits are set too low. For most HPC installations, the
memlock limits should be set to "unlimited". The failure occured
here:
Local host: yw0431
OMPI source: ../../../../../ompi/mca/btl/openib/btl_openib_component.c:1216
Function: ompi_free_list_init_ex_new()
Device: mlx4_0
Memlock limit: 65536
You may need to consult with your system administrator to get this
problem fixed. This FAQ entry on the Open MPI web site may also be
helpful:
http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.
Local host: yw0431
Local device: mlx4_0
--------------------------------------------------------------------------
[yw0431:20193] 11 more processes have sent help message help-mpi-btl-openib.txt / init-fail-no-mem
[yw0431:20193] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[yw0431:20193] 11 more processes have sent help message help-mpi-btl-openib.txt / error in device init
forrtl: error (78): process killed (SIGTERM)
it means that my linux server have locked memory with 65M, but my job needed more memory. I think 2G should be emough.
I have found a solution about ulimiting the memory:
ulimit -l unlimited
But i am worried that i will cause system crash or some bad things happen.
so can i set "ulimit -l umlimited"?
When you set ulimit as unlimited and your process starting using memory exhaustively then OOM killer will kill ur job for system stability,I would set the ulimit as 80 to 90% of RAM of instead of unlimited.
I am facing a problem now, unable to run any program in the cluster. It gives error.
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 64 current, 64 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 64 current, 64 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 64 current, 64 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 64 current, 64 max
Traceback (most recent call last):
File "hello-world.py", line 1, in <module>
from keras.models import Sequential
File "/home/amalli2s/anaconda3/lib/python3.6/site-packages/keras/__init__.py", line 3, in <module>
from . import utils
File "/home/amalli2s/anaconda3/lib/python3.6/site-packages/keras/utils/__init__.py", line 2, in <module>
from . import np_utils
File "/home/amalli2s/anaconda3/lib/python3.6/site-packages/keras/utils/np_utils.py", line 6, in <module>
import numpy as np
File "/home/amalli2s/.local/lib/python3.6/site-packages/numpy/__init__.py", line 142, in <module>
from . import add_newdocs
File "/home/amalli2s/.local/lib/python3.6/site-packages/numpy/add_newdocs.py", line 13, in <module>
from numpy.lib import add_newdoc
File "/home/amalli2s/.local/lib/python3.6/site-packages/numpy/lib/__init__.py", line 8, in <module>
from .type_check import *
File "/home/amalli2s/.local/lib/python3.6/site-packages/numpy/lib/type_check.py", line 11, in <module>
import numpy.core.numeric as _nx
File "/home/amalli2s/.local/lib/python3.6/site-packages/numpy/core/__init__.py", line 16, in <module>
from . import multiarray
SystemError: initialization of multiarray raised unreported exception
This problem, i assume to be same as this one Multiple instances of Python running simultaneously limited to 35
So according to the solution when I set
export OPENBLAS_NUM_THREADS=1
then I end up getting following error:
terminate called after throwing an instance of 'std::system_error'
what(): Resource temporarily unavailable
Aborted
Is there anybody else facing same issue or has idea on how to solve this ? Thank you.
EDIT:
Ok, Seems like this happened because of some config restrictions the admins were trying to implement. Now it works again.
I had this problem running numpy on an ubuntu server. I got all of the following errors, depending on whether I tried to import numpy in a shell or running my django app:
PyCapsule_Import could not import module "datetime" from
numpy.core._multiarray_umath import ( OpenBLAS blas_thread_init:
pthread_create failed for thread 25 of 32: Resource temporarily unavailable
I'm posting this answer since it drove me crazy. What helped for me was to add:
import os
os.environ['OPENBLAS_NUM_THREADS'] = '1'
before
import numpy as np
I guess the server had some limit for the amount of threads it allows(?). Hope it helps someone!
This is for others in the future who encounter this error. The cluster setup most likely limits the number processes that can be run by a user on an interactive node. The clue is in the second line of the error:
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 64 current, 64 max
Here the limit is set to 64. While this is quite sufficient for normal CLI use, it's probably not sufficient for interactively running Keras jobs (like the OP); or in my case, trying to run an interactive Dask cluster.
It may be possible to increase the limit from your shell using, say ulimit -u 10000, but that's not guaranteed to work. It's best to notify the admins like the OP.
Often this issue is related to the limit number of processes available through ulimit (in Linux):
→ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 127590
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096 # <------------------culprit
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
A temporary solution is to increase this limit:
ulimit -u unlimited
Most servers I've encountered have this values set to the number of pending signals. E.g. ulimit -i. So, on my example above I did instead:
ulimit -u 127590
Then, added a line on my ~/.bashrc file to set it on login.
For more info on how to permanently fix this, check out: https://serverfault.com/a/485277
If you are a manager, you can:
change temporarily the limit number of processes with the command ulimit -u [number]
change permanently the limit number of processes, i.e. the parameter nproc in /etc/security/limit.conf
If you are a user, you can:
In bash
$ export OPENBLAS_NUM_THREADS=2
$ export GOTO_NUM_THREADS=2
$ export OMP_NUM_THREADS=2
In Python
>>> import os
>>> os.environ['OPENBLAS_NUM_THREADS'] = '1`
Then the problem due to multiple-threads in Python should be solved. The key is to set the number of threads to be less than the limit for you in the cluster.
Building on Ylor's answer, rather than limiting yourself to a single thread, read through the error outputs (here are the first few lines of mine):
OpenBLAS blas_thread_init: pthread_create failed for thread 13 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2048 current, 384066 max
OpenBLAS blas_thread_init: pthread_create failed for thread 58 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2048 current, 384066 max
...
and find the minimum thread number which failed–then set the thread count to one fewer (12 for me here):
>>> import os
>>> os.environ['OPENBLAS_NUM_THREADS'] = '12`
This will maximize your code's ability to use threads while still staying within current system limits (if unable to change).