How to work around Python 3 maximum string size? - python-3.x

On a 64 bit Python build with a sys.maxsize of 9223372036854775807 the
interpreter nevertheless throws a MemoryError if I allocate a string of more
than INT_MAX - 512 MiB chars:
$ python3
#Python 3.6.6 (default, Jul 19 2018, 14:25:17)
[GCC 8.1.1 20180712 (Red Hat 8.1.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> s = "*" * 2684354560
>>> s = "*" * 2684354561
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
MemoryError
(The limit is the same for bytes whose element type is definitely 8 bits.)
There is plenty of free memory and swap so I am certain the system is not
hitting any physical limits.
What is happening here and how can I increase this cap?

Resolution: turns out to be the
data segment size limit
$ ulimit -d
4194304
For some reason, these 4294967296 B translate to a 2684354560 B per-allocation
cap in Python.
Setting this value to unlimited removes the cap. This can be done externally
by the parent process (e. g. ulimit -d unlimited from the shell) or
in Python itself using the
wrapper library for resource.h:
resource.setrlimit (resource.RLIMIT_DATA,
(resource.RLIM_INFINITY
,resource.RLIM_INFINITY))
Apparently on more
recent kernels (4.7 and later) RLIMIT_DATA affects anonymous mappings too which
explains both the observed failure of large-ish allocations and my being
surprised.

Related

os.major() and os.minor() on block device incorrect?

Using Python 3.9.2 on Linux I try to find major and minor device numbers of a block device, e.g. /dev/sda.
For result checking I first note the expected result using shell utils:
$ ls -ltr /dev/sda
brw-rw---- 1 root disk 8, 0 Mar 15 13:34 /dev/sda
major-----^ ^----minor
$ stat -c%t:%T /dev/sda
8:0
But then in python3:
python3 -c "import os ; print( os.major(os.stat('/dev/sda').st_dev) )"
0
python3 -c "import os ; print( os.minor(os.stat('/dev/sda').st_dev) )"
5
Major is 0 and minor is 5 according to python...?
The subject of many (if not all) questions similar to mine, is getting the major/minor numbers for some directory that resides on e.g. /dev/sda2. That somehow does get me the expected results:
python3 -c "import os ; print( os.major(os.stat('/etc').st_dev) )"
8
Can't find my mistake. How can I get, using python, the major and minor numbers of a block device file like /dev/sda?
You want os.stat("/dev/sda").st_rdev.

how to determine ulimits - linux

how to determine ulimits (linux)?
Im using ubuntu 16.04,
kernel version 4.4.0-21-generic
I set the nofile to maximum for root (in /etc/security/limits.conf)
the line is: * hard nofile NUMBER
according to file /proc/sys/fs/file-max
the value is : 32854728
when Im running the command ulimit -a
i found that the limitation is 1024.
i tested it , and i found that the highest value of max open file is 1048575.
If I set it to higher value the limit is 1024.
how to determine ulimit of openfiles? why I can't set it to higher limit than 1048575?
To determine the maximum number of file handles for the entire system, run:
cat /proc/sys/fs/file-max
To determine the current usage of file handles, run:
$ cat /proc/sys/fs/file-nr
1154 133 8192
|   |  |
|   |   |
|        | maximum open file descriptors
   | total free allocated file descriptors
total allocated file descriptors
(the number of file descriptors allocated since boot)

shell command returning an sg device size (Linux)

I need a shell command that would return an sg device size for me.
I am searching for it over the Internet for almost two hours without any success.
I can do it by writing a program but there has to be a way to get it through a command! I simply cannot find it!
Unfortunatelly, on the dell server I work, sg_inq, and sginfo return Dell's PERC information, and not the device I try to get the information of.
I tried to use the smartctl but it returns a static (and not true) infomation about the device (SSD,) for the number of bytes it returns divided by 512 shows bigger number of sectors that I can access (sic!)
Any [shell command] hint would be greatly appreciated!
I am putting an answer here for anyone looking for it.
The ultimate way to either write own program or use smartctl (http://sourceforge.net/projects/smartmontools/files/smartmontools/6.0/) and call the following command:
#>./smartctl -a /dev/sdc
smartctl 6.0 2012-10-10 r3643 [x86_64-linux-2.6.18-274.18.1.el5] (local build)
Copyright (C) 2002-12, Bruce Allen, Christian Franke, www.smartmontools.org
Vendor: DELL
Product: PERC H710P
Revision: 3.13
User Capacity: 179,443,728,384 bytes [179 GB]
Logical block size: 512 bytes
Logical Unit id: --------------------------------
Serial number: --------------------------------
Device type: disk
Local Time is: Tue Sep 24 17:54:13 2013 EDT
Device does not support SMART
Error Counter logging not supported
Device does not support Self Test logging
use #fdisk -l /dev/sg2
or # parted -l /dev/sg2

Maximum length of command line argument that can be passed to SQL*Plus?

I am calling SQL*Plus from Linux C Shell:
sqlplus username/password #file.sql var1 var2 var3
If I pass a string as var1, how long can this string be?
Is it governed by the OS? In this case:
Linux version 2.6.9-100.ELsmp (mockbuild#x86-010.build.bos.redhat.com) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-11)) #1 SMP Tue Feb 1 12:17:32 EST 2011
Update: Empirical testing yielded the following results:
A command line argument of 5200 characters gave the error, "Word too long."
1300 characters then produced the SQL*Plus error, "string beginning "(000796384..." is too long. maximum size is 239 characters."
As soon as I got under 239 chars all was well.
I think I'll use sqlldr to overcome this.
Try with: xargs --show-limits </dev/null
Your environment variables take up 2446 bytes
POSIX upper limit on argument length (this system): 2092658
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2090212
Size of command buffer we are actually using: 131072
There is no limit per argument, but a total for the whole command line length. In my system (Fedora 15/zsh) its closer to 2Mb. (line 4).
I came across "How long an argument list your kernel can take on the command line before it chokes?":
getconf ARG_MAX
which gives the following on my system:
131072

How to determine whether a given Linux is 32 bit or 64 bit?

When I type uname -a, it gives the following output.
Linux mars 2.6.9-67.0.15.ELsmp #1 SMP Tue Apr 22 13:50:33 EDT 2008 i686 i686 i386 GNU/Linux
How can I know from this that the given OS is 32 or 64 bit?
This is useful when writing configure scripts, for example: what architecture am I building for?
Try uname -m. Which is short of uname --machine and it outputs:
x86_64 ==> 64-bit kernel
i686 ==> 32-bit kernel
Otherwise, not for the Linux kernel, but for the CPU, you type:
cat /proc/cpuinfo
or:
grep flags /proc/cpuinfo
Under "flags" parameter, you will see various values: see "What do the flags in /proc/cpuinfo mean?"
Among them, one is named lm: Long Mode (x86-64: amd64, also known as Intel 64, i.e. 64-bit capable)
lm ==> 64-bit processor
Or using lshw (as mentioned below by Rolf of Saxony), without sudo (just for grepping the cpu width):
lshw -class cpu|grep "^ width"|uniq|awk '{print $2}'
Note: you can have a 64-bit CPU with a 32-bit kernel installed.
(as ysdx mentions in his/her own answer, "Nowadays, a system can be multiarch so it does not make sense anyway. You might want to find the default target of the compiler")
If you were running a 64 bit platform you would see x86_64 or something very similar in the output from uname -a
To get your specific machine hardware name run
uname -m
You can also call
getconf LONG_BIT
which returns either 32 or 64
lscpu will list out these among other information regarding your CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
...
Another useful command for easy determination is as below:
Command:
getconf LONG_BIT
Answer:
32, if OS is 32 bit
64, if OS is 64 bit
The command
$ arch
is equivalent to
$ uname -m
but is twice as fast to type
I was wondering about this specifically for building software in Debian (the installed Debian system can be a 32-bit version with a 32 bit kernel, libraries, etc., or it can be a 64-bit version with stuff compiled for the 64-bit rather than 32-bit compatibility mode).
Debian packages themselves need to know what architecture they are for (of course) when they actually create the package with all of its metadata, including platform architecture, so there is a packaging tool that outputs it for other packaging tools and scripts to use, called dpkg-architecture. It includes both what it's configured to build for, as well as the current host. (Normally these are the same though.) Example output on a 64-bit machine:
DEB_BUILD_ARCH=amd64
DEB_BUILD_ARCH_OS=linux
DEB_BUILD_ARCH_CPU=amd64
DEB_BUILD_GNU_CPU=x86_64
DEB_BUILD_GNU_SYSTEM=linux-gnu
DEB_BUILD_GNU_TYPE=x86_64-linux-gnu
DEB_HOST_ARCH=amd64
DEB_HOST_ARCH_OS=linux
DEB_HOST_ARCH_CPU=amd64
DEB_HOST_GNU_CPU=x86_64
DEB_HOST_GNU_SYSTEM=linux-gnu
DEB_HOST_GNU_TYPE=x86_64-linux-gnu
You can print just one of those variables or do a test against their values with command line options to dpkg-architecture.
I have no idea how dpkg-architecture deduces the architecture, but you could look at its documentation or source code (dpkg-architecture and much of the dpkg system in general are Perl).
#include <stdio.h>
int main(void)
{
printf("%d\n", __WORDSIZE);
return 0;
}
If you have a 64-bit OS, instead of i686, you have x86_64 or ia64 in the output of uname -a. In that you do not have any of these two strings; you have a 32-bit OS (note that this does not mean that your CPU is not 64-bit).
That system is 32bit. iX86 in uname means it is a 32-bit architecture. If it was 64 bit, it would return
Linux mars 2.6.9-67.0.15.ELsmp #1 SMP Tue Apr 22 13:50:33 EDT 2008 x86_64 i686 x86_64 x86_64 GNU/Linux
Nowadays, a system can be multiarch so it does not make sense anyway. You might want to find the default target of the compiler:
$ cc -v 2>&1 | grep ^Target
Target: x86_64-pc-linux-gn
You can try to compile a hello world:
$ echo 'int main() { return 0; }' | cc -x c - -o foo
$ file foo
foo: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=b114e029a08abfb3c98db93d3dcdb7435b5bba0c, not stripped
With respect to the answer "getconf LONG_BIT".
I wrote a simple function to do it in 'C':
/*
* check_os_64bit
*
* Returns integer:
* 1 = it is a 64-bit OS
* 0 = it is NOT a 64-bit OS (probably 32-bit)
* < 0 = failure
* -1 = popen failed
* -2 = fgets failed
*
* **WARNING**
* Be CAREFUL! Just testing for a boolean return may not cut it
* with this (trivial) implementation! (Think of when it fails,
* returning -ve; this could be seen as non-zero & therefore true!)
* Suggestions?
*/
static int check_os_64bit(void)
{
FILE *fp=NULL;
char cb64[3];
fp = popen ("getconf LONG_BIT", "r");
if (!fp)
return -1;
if (!fgets(cb64, 3, fp))
return -2;
if (!strncmp (cb64, "64", 3)) {
return 1;
}
else {
return 0;
}
}
Good idea, the 'getconf'!
You can also check using a environment variable:
echo $HOSTTYPE
Result:
i386 -> 32 bits
x86_64 -> 64 bits
Extracted from: http://www.sysadmit.com/2016/02/linux-como-saber-si-es-32-o-64-bits.html
In Bash, using integer overflow:
if ((1 == 1<<32)); then
echo 32bits
else
echo 64bits
fi
It's much more efficient than invoking another process or opening files.
getconf uses the fewest system calls:
$ strace getconf LONG_BIT | wc -l
253
$ strace arch | wc -l
280
$ strace uname -m | wc -l
281
$ strace grep -q lm /proc/cpuinfo | wc -l
301
If you shift 1 left by 32 and you get 1, your system is 32 bit.
If you shift 1 left by 64 and you get 1, your system is 64 bit.
In other words,
if echo $((1<<32)) gives 1 then your system is 32 bit.
if echo $((1<<64)) gives 1 then your system is 64 bit.
If one is severely limited in available binaries (e.g. in initramfs), my colleagues suggested:
$ ls -l /lib*/ld-linux*.so.2
On my ALT Linux systems, i586 has /lib/ld-linux.so.2 and x86_64 has /lib64/ld-linux-x86-64.so.2.
$ grep "CONFIG_64" /lib/modules/*/build/.config
# CONFIG_64BIT is not set
I can't believe that in all this time, no one has mentioned:
sudo lshw -class cpu
to get details about the speed, quantity, size and capabilities of the CPU hardware.
Simple script to get 64 bit or 32 bit
if $(getconf LONG_BIT | grep '64'); then
echo "64 bit system"
else
echo "32 bit system"
fi
[ -z `uname -m | grep 64` ] && echo "32-bit" || echo "64-bit"
Based on the fact that 64-bit is usually x86_64 and 32-bit is i686 etc.
First you have to download Virtual Box. Then select new and a 32-bit Linux. Then boot the linux using it. If it boots then it is 32 bit if it doesn't then it is a 64 bit.

Resources