I am trying to implement a double buffer using ioctl(fd, FBIOPAN_DISPLAY...
my single buffer code works fine
screensize = vinfo.xres * vinfo.yres * vinfo.bits_per_pixel / 8;
fbp = (char*)mmap(0, screensize, PROT_READ | PROT_WRITE, MAP_SHARED, fbfd, 0);
when I try to increase the "length parameter" by using screensize*2, the mmap fails with EINVAL. I think it doesn't like my length parameter.
The screen size for a single buffer is 6,220,800 and for the double buffer is 12,441600. This is an embedded system but it has 1 Gig of RAM.
The length parameter is size_t which on this system is only 4 bytes which would make me think that the max size I could use would be 4 Meg, yet 6 Meg works fine so I think I am missing something really simple. Is there a way to mmap a buffer larger than size_t?
The man page says that length (the 2nd parameter) is of type size_t, so I don't think you are safe to pass a larger type.
I would suggest you to just map the first part, and then remap the second part as shown in this SO Q&A.
Regarding the EINVAL:
Following is stated in the man page:
EINVAL
We don't like addr, length, or offset (e.g., they are too large, or not aligned on a page boundary).
EINVAL
(since Linux 2.6.12) length was 0.
EINVAL
flags contained neither MAP_PRIVATE or MAP_SHARED, or contained both of these values.
Are you sure you are page alligned?
Related
I have to pass 256Kb of text as an argument to the "aws sqs" command but am running into a limit in the command-line at around 140Kb. This has been discussed in many places that it been solved in the Linux kernel as of 2.6.23 kernel.
But cannot get it to work. I am using 3.14.48-33.39.amzn1.x86_64
Here's a simple example to test:
#!/bin/bash
SIZE=1000
while [ $SIZE -lt 300000 ]
do
echo "$SIZE"
VAR="`head -c $SIZE < /dev/zero | tr '\0' 'a'`"
./foo "$VAR"
let SIZE="( $SIZE * 20 ) / 19"
done
And the foo script is just:
#!/bin/bash
echo -n "$1" | wc -c
And the output for me is:
117037
123196
123196
129680
129680
136505
./testCL: line 11: ./foo: Argument list too long
143689
./testCL: line 11: ./foo: Argument list too long
151251
./testCL: line 11: ./foo: Argument list too long
159211
So, the question how do I modify the testCL script is it can pass 256Kb of data? Btw, I have tried adding ulimit -s 65536 to the script and it didn't help.
And if this is plain impossible I can deal with that but can you shed light on this quote from my link above
"While Linux is not Plan 9, in 2.6.23 Linux is adding variable
argument length. Theoretically you shouldn't hit frequently "argument
list too long" errors again, but this patch also limits the maximum
argument length to 25% of the maximum stack limit (ulimit -s)."
edit:
I was finally able to pass <= 256 KB as a single command line argument (see edit (4) in the bottom). However, please read carefully how I did it and decide for yourself if this is a way you want to go. At least you should be able to understand why you are 'stuck' otherwise from what I found out.
With the coupling of ARG_MAX to ulim -s / 4 came the introduction of MAX_ARG_STRLEN as max. length of an argument:
/*
* linux/fs/exec.c
*
* Copyright (C) 1991, 1992 Linus Torvalds
*/
...
#ifdef CONFIG_MMU
/*
* The nascent bprm->mm is not visible until exec_mmap() but it can
* use a lot of memory, account these pages in current->mm temporary
* for oom_badness()->get_mm_rss(). Once exec succeeds or fails, we
* change the counter back via acct_arg_size(0).
*/
...
static bool valid_arg_len(struct linux_binprm *bprm, long len)
{
return len <= MAX_ARG_STRLEN;
}
...
#else
...
static bool valid_arg_len(struct linux_binprm *bprm, long len)
{
return len <= bprm->p;
}
#endif /* CONFIG_MMU */
...
static int copy_strings(int argc, struct user_arg_ptr argv,
struct linux_binprm *bprm)
{
...
str = get_user_arg_ptr(argv, argc);
...
len = strnlen_user(str, MAX_ARG_STRLEN);
if (!len)
goto out;
ret = -E2BIG;
if (!valid_arg_len(bprm, len))
goto out;
...
}
...
MAX_ARG_STRLEN is defined as 32 times the page size in linux/include/uapi/linux/binfmts.h:
...
/*
* These are the maximum length and maximum number of strings passed to the
* execve() system call. MAX_ARG_STRLEN is essentially random but serves to
* prevent the kernel from being unduly impacted by misaddressed pointers.
* MAX_ARG_STRINGS is chosen to fit in a signed 32-bit integer.
*/
#define MAX_ARG_STRLEN (PAGE_SIZE * 32)
#define MAX_ARG_STRINGS 0x7FFFFFFF
...
The default page size is 4 KB so you cannot pass arguments longer than 128 KB.
I can't try it now but maybe switching to huge page mode (page size 4 MB) if possible on your system solves this problem.
For more detailed information and references see this answer to a similar question on Unix & Linux SE.
edits:
(1)
According to this answer one can change the page size of x86_64 Linux to 1 MB by enabling CONFIG_TRANSPARENT_HUGEPAGE and setting CONFIG_TRANSPARENT_HUGEPAGE_MADVISE to n in the kernel config.
(2)
After recompiling my kernel with the above configuration changes getconf PAGESIZE still returns 4096.
According to this answer CONFIG_HUGETLB_PAGE is also needed which I could pull in via CONFIG_HUGETLBFS. I am recompiling now and will test again.
(3)
I recompiled my kernel with CONFIG_HUGETLBFS enabled and now /proc/meminfo contains the corresponding HugePages_* entries mentioned in the corresponding section of the kernel documentation.
However, the page size according to getconf PAGESIZE is still unchanged. So while I should be able now to request huge pages via mmap calls, the kernel's default page size determining MAX_ARG_STRLEN is still fixed at 4 KB.
(4)
I modified linux/include/uapi/linux/binfmts.h to #define MAX_ARG_STRLEN (PAGE_SIZE * 64), recompiled my kernel and now your code produces:
...
117037
123196
123196
129680
129680
136505
143689
151251
159211
...
227982
227982
239981
239981
252611
252611
265906
./testCL: line 11: ./foo: Argument list too long
279901
./testCL: line 11: ./foo: Argument list too long
294632
./testCL: line 11: ./foo: Argument list too long
So now the limit moved from 128 KB to 256 KB as expected.
I don't know about potential side effects though.
As far as I can tell, my system seems to run just fine.
Just put the arguments into some file, and modify your program to accept "arguments" from a file. A common convention (notably used by GCC and several other GNU programs) is that an argument like #/tmp/arglist.txt asks your program to read arguments from file /tmp/arglist.txt, often one line per argument
You might perhaps pass some data thru long environment variables, but they also are limited (and what is limited by the kernel in fact is the size of main's initial stack, containing both program arguments and the environment)
Alternatively, modify your program to be configurable thru some configuration file which would contain the information you want to pass thru arguments.
(If you can recompile your kernel, you might try to increase -to a bigger power of two much smaller than your available RAM, e.g. to 2097152- the ARG_MAX which is #define-d in linux-4.*/include/uapi/linux/limits.h before recompiling your kernel)
In other ways, there is no way to circumvent that limitation (see execve(2) man page and its Limits on size of arguments and environment section) - once you have raised your stack limit (using setrlimit(2) with RLIMIT_STACK, generally with ulimit builtin in the parent shell). You need to deal with it otherwise.
I have to pass 256Kb of text as an argument to the "aws sqs" command but am running into a limit in the command-line at around 140Kb. This has been discussed in many places that it been solved in the Linux kernel as of 2.6.23 kernel.
But cannot get it to work. I am using 3.14.48-33.39.amzn1.x86_64
Here's a simple example to test:
#!/bin/bash
SIZE=1000
while [ $SIZE -lt 300000 ]
do
echo "$SIZE"
VAR="`head -c $SIZE < /dev/zero | tr '\0' 'a'`"
./foo "$VAR"
let SIZE="( $SIZE * 20 ) / 19"
done
And the foo script is just:
#!/bin/bash
echo -n "$1" | wc -c
And the output for me is:
117037
123196
123196
129680
129680
136505
./testCL: line 11: ./foo: Argument list too long
143689
./testCL: line 11: ./foo: Argument list too long
151251
./testCL: line 11: ./foo: Argument list too long
159211
So, the question how do I modify the testCL script is it can pass 256Kb of data? Btw, I have tried adding ulimit -s 65536 to the script and it didn't help.
And if this is plain impossible I can deal with that but can you shed light on this quote from my link above
"While Linux is not Plan 9, in 2.6.23 Linux is adding variable
argument length. Theoretically you shouldn't hit frequently "argument
list too long" errors again, but this patch also limits the maximum
argument length to 25% of the maximum stack limit (ulimit -s)."
edit:
I was finally able to pass <= 256 KB as a single command line argument (see edit (4) in the bottom). However, please read carefully how I did it and decide for yourself if this is a way you want to go. At least you should be able to understand why you are 'stuck' otherwise from what I found out.
With the coupling of ARG_MAX to ulim -s / 4 came the introduction of MAX_ARG_STRLEN as max. length of an argument:
/*
* linux/fs/exec.c
*
* Copyright (C) 1991, 1992 Linus Torvalds
*/
...
#ifdef CONFIG_MMU
/*
* The nascent bprm->mm is not visible until exec_mmap() but it can
* use a lot of memory, account these pages in current->mm temporary
* for oom_badness()->get_mm_rss(). Once exec succeeds or fails, we
* change the counter back via acct_arg_size(0).
*/
...
static bool valid_arg_len(struct linux_binprm *bprm, long len)
{
return len <= MAX_ARG_STRLEN;
}
...
#else
...
static bool valid_arg_len(struct linux_binprm *bprm, long len)
{
return len <= bprm->p;
}
#endif /* CONFIG_MMU */
...
static int copy_strings(int argc, struct user_arg_ptr argv,
struct linux_binprm *bprm)
{
...
str = get_user_arg_ptr(argv, argc);
...
len = strnlen_user(str, MAX_ARG_STRLEN);
if (!len)
goto out;
ret = -E2BIG;
if (!valid_arg_len(bprm, len))
goto out;
...
}
...
MAX_ARG_STRLEN is defined as 32 times the page size in linux/include/uapi/linux/binfmts.h:
...
/*
* These are the maximum length and maximum number of strings passed to the
* execve() system call. MAX_ARG_STRLEN is essentially random but serves to
* prevent the kernel from being unduly impacted by misaddressed pointers.
* MAX_ARG_STRINGS is chosen to fit in a signed 32-bit integer.
*/
#define MAX_ARG_STRLEN (PAGE_SIZE * 32)
#define MAX_ARG_STRINGS 0x7FFFFFFF
...
The default page size is 4 KB so you cannot pass arguments longer than 128 KB.
I can't try it now but maybe switching to huge page mode (page size 4 MB) if possible on your system solves this problem.
For more detailed information and references see this answer to a similar question on Unix & Linux SE.
edits:
(1)
According to this answer one can change the page size of x86_64 Linux to 1 MB by enabling CONFIG_TRANSPARENT_HUGEPAGE and setting CONFIG_TRANSPARENT_HUGEPAGE_MADVISE to n in the kernel config.
(2)
After recompiling my kernel with the above configuration changes getconf PAGESIZE still returns 4096.
According to this answer CONFIG_HUGETLB_PAGE is also needed which I could pull in via CONFIG_HUGETLBFS. I am recompiling now and will test again.
(3)
I recompiled my kernel with CONFIG_HUGETLBFS enabled and now /proc/meminfo contains the corresponding HugePages_* entries mentioned in the corresponding section of the kernel documentation.
However, the page size according to getconf PAGESIZE is still unchanged. So while I should be able now to request huge pages via mmap calls, the kernel's default page size determining MAX_ARG_STRLEN is still fixed at 4 KB.
(4)
I modified linux/include/uapi/linux/binfmts.h to #define MAX_ARG_STRLEN (PAGE_SIZE * 64), recompiled my kernel and now your code produces:
...
117037
123196
123196
129680
129680
136505
143689
151251
159211
...
227982
227982
239981
239981
252611
252611
265906
./testCL: line 11: ./foo: Argument list too long
279901
./testCL: line 11: ./foo: Argument list too long
294632
./testCL: line 11: ./foo: Argument list too long
So now the limit moved from 128 KB to 256 KB as expected.
I don't know about potential side effects though.
As far as I can tell, my system seems to run just fine.
Just put the arguments into some file, and modify your program to accept "arguments" from a file. A common convention (notably used by GCC and several other GNU programs) is that an argument like #/tmp/arglist.txt asks your program to read arguments from file /tmp/arglist.txt, often one line per argument
You might perhaps pass some data thru long environment variables, but they also are limited (and what is limited by the kernel in fact is the size of main's initial stack, containing both program arguments and the environment)
Alternatively, modify your program to be configurable thru some configuration file which would contain the information you want to pass thru arguments.
(If you can recompile your kernel, you might try to increase -to a bigger power of two much smaller than your available RAM, e.g. to 2097152- the ARG_MAX which is #define-d in linux-4.*/include/uapi/linux/limits.h before recompiling your kernel)
In other ways, there is no way to circumvent that limitation (see execve(2) man page and its Limits on size of arguments and environment section) - once you have raised your stack limit (using setrlimit(2) with RLIMIT_STACK, generally with ulimit builtin in the parent shell). You need to deal with it otherwise.
I am looking at the man-page for read(int fd, void *buf, size_t count)
http://man7.org/linux/man-pages/man2/read.2.html
where I need some more explanation on the words "On files that support seeking, the read operation commences at the current file offset, and the file offset is incremented by the number of bytes read. "
1) If I want to read a file not from the beginning, say at offset 100 (bytes) to read 1 byte, is the offset 100 added to the fd, i.e, read(fd+100, buf, 1)? If not, how can I specify the offset in the code?
2) How do I know if "files support seeking"? I "opened" the FPGA as a spi device through spi bus, to get the fd. I am using read() to read registers of FPGA. In this case, is the file support seeking?
Thanks!
You need to first move the file pointer (current file offset) to 100, via a read or seek call.
Alternatively you might be interested in pread, depending on what you are doing. pread is equivalent of atomically (1) saving the current offset, (2) read the offset you want to read, and (3) restoring the original offset.
On your second question you will know if it isn't a seekable device because your call to lseek will fail. I don't know of any reliable way to know in advance.
You use lseek to move the possition in the file -- so you would do something like
lseek(fd, 100, SEEK_SET);
read(fd, buffer, 1);
to read one byte at position 100.
However, while this is a valid example, I would advice not to read individual bytes in a file this way, as it is very slow/expensive.
If you want to make random io getting individual bytes in a file at scale, you may be better of usin mmap rather than lseek/read
I want to share memory between two process.
After mmap(), I get a address mapStart, then I add offset to mapStart and get mapAddr, and make sure mapAddr will not exceed maped PAGE_SIZE.
When I write to mapAddr by
memcpy((void *)mapAddr, data, size);
everything is OK.
But when I read from mapAddr by
memcpy( &data, (void *)mapAddr, size);`
that will case system crash.
Who know Why?
The similar problem is here
Add some Info: #Tony Delroy, #J-16 SDiZ
mmap function is:
mapStart = (void volatile *)mmap(0, PAGE_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_LOCKED, memfd, pa_base);
system crash: have no any OS error message, Console print some MCA info
the detail described in here
Just some idea.
Is your mmap() spanning over memory regions with different attribute? This is illegal.
Older kernel (you said 2.6.18) allowed this, but crash when you write to some of it.
See this post for some starting point. If it is possible, try a newer kernel.
There are at least two possible issues:
After mmap(), I get a address mapStart, then I add offset to mapStart and get mapAddr, and make sure mapAddr will not exceed maped PAGE_SIZE.
Not mapAddr must be made sure not to exceed the mapped size, but mapAddr+size. You are trying to touch size bytes, not just one.
memcpy((void *)mapAddr, data, size);
memcpy( &data, (void *)mapAddr, size);
Assuming data is not a array (which is a plausible assumption since you use it without address operator in the first line), the second line copies not from the location pointed to by data, but starting with data. This is quite possibly some unallocated memory, or some location on the stack, or whatever. If there is not a lot on the stack, it might as well read beyond the end of the stack into the text segment, or... something else.
(If data is indeed an array, it is of course equivalent, but then your code style would be inconsistent.)
I am new in vc++... I have one doubt in vc++. what is the size of GetTickCount() function.The return type of GetTickCount() is DWORD. Please anyone Answer for my question.
Thanks in Advance
The size of a function means the number of bytes occupied by the code that belongs to the function. You can find this out using a debugger like Windbg. But this is not useful information in most cases. To get the size of a data type, you can use the sizeof operator. Since the return type of GetTickCount is DWORD (4 bytes), you can either do sizeof(DWORD) or sizeof(GetTickCount()) to get its size. There is also a function by the name GetTickCount64 which returns ULONGLONG which is a 64-bit unsigned value (8 bytes).
GetTickCount() returns a DWORD which is 4 bytes. The function itself can be represented using its start address (function pointer) which will have size equal to size of void* which is 4 bytes on 32-bit systems and 8 bytes on 64-bit systems. Finding the size of code that the function occupies can be problematic and is rarely needed.