Finding out if your script has a significant memory leak might be of interest even before you run into serious trouble. Unfortunately, I was not able to find out how to measure the current stack/heap size or "string table" size (see http://www.smartdxl.com/content/?p=481). Can anybody help?
Related question: Predicting DXL Memory and CPU Usage
The biggest memory "leak" would be open modules that are no longer being used. So of course you should be closing those.
Next you want to keep the production of new strings to a minimum since each new one creates an entry in a string table. You can find an excellent dissertation on that by Mathias Mamsch here: https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014886977&ps=25
Finally, data types that have create/delete methods can eat up memory if they are not being deleted. To find unreleased instances, I use some memory functions originally created by Mathias Mamsch. The link I have back to his post no longer works, but here are the functions that I use:
//< Memory Functions [Memory.inc]
/*
Code adapted from forum post by Mathias Mamsch:
https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014830975
*/
int *::+(int *ptr, int ofs)
{
int *rtn = ptr
rtn += ofs
return(rtn)
}
int *::#(int *ptr, int ofs)
{
int adr = *(ptr + ofs)
int *rtn = addr_(adr)
return(rtn)
}
int *mbn(int *ptr)
{
return(ptr # 0x74)
}
int *nnn(int *ptr)
{
return(ptr # 8)
}
int *ccp()
{
DB db = create("")
int *ptr = addr_(db)
int *rtn = ptr # 48
destroy(db)
return(rtn)
}
int allocatedObjects()
{
int cnt = 0
int *mb = mbn(ccp())
while(!null mb) { mb = nnn(mb) ; cnt++ }
return(cnt)
}
I'm pretty sure I changed the function and variable names from the original posted code, so be aware of that if you ever come across his original code. And don't ask me about the hard-coded numbers... Mathias explained them in the post and I don't recall the explanation.
Here's how you would use the code:
//< Test of Memory.inc
/*
*/
pragma encoding, "UTF-8"
#include <stdlib/Memory.inc>
pragma runLim, 0
int numallobj = allocatedObjects()
print numallobj "\n"
Skip skp = null Skip
numallobj = allocatedObjects()
print numallobj "\n"
skp = create()
numallobj = allocatedObjects()
print numallobj "\n"
delete(skp)
numallobj = allocatedObjects()
print numallobj "\n"
/*
Output should be:
0
0
1
0
*/
I found a solution for the "string table" part of the question. Mathias Mamsch provides a file "stringTable.inc" at https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014886977 which defines the function printStringTable(). Apparently, it outputs information about the table size and can be easily patched to provide an approximation of the string table bytesize.
Related
I am trying to write a program that calls upon an [external library (?)] (I'm not sure that I'm using the right terminology here) that I am also writing to clean up a provided string. For example, if my main.c program were to be provided with a string such as:
asdfFAweWFwseFL Wefawf JAWEFfja FAWSEF
it would call upon a function in externalLibrary.c (lets call it externalLibrary_Clean for now) that would take in the string, and return all characters in upper case without spaces:
ASDFFAWEWFWSEFLWEFAWFJAWEFFJAFAWSEF
The crazy part is that I have this working... so long as my string doesn't exceed 26 characters in length. As soon as I add a 27th character, I end up with an error that says
malloc(): corrupted top size.
Here is externalLibrary.c:
#include "externalLibrary.h"
#include <ctype.h>
#include <malloc.h>
#include <assert.h>
#include <string.h>
char * restrict externalLibrary_Clean(const char* restrict input) {
// first we define the return value as a pointer and initialize
// an integer to count the length of the string
char * returnVal = malloc(sizeof(input));
char * initialReturnVal = returnVal; //point to the start location
// until we hit the end of the string, we use this while loop to
// iterate through it
while (*input != '\0') {
if (isalpha(*input)) { // if we encounter an alphabet character (a-z/A-Z)
// then we convert it to an uppercase value and point our return value at it
*returnVal = toupper(*input);
returnVal++; //we use this to move our return value to the next location in memory
}
input++; // we move to the next memory location on the provided character pointer
}
*returnVal = '\0'; //once we have exhausted the input character pointer, we terminate our return value
return initialReturnVal;
}
int * restrict externalLibrary_getFrequencies(char * ar, int length){
static int freq[26];
for (int i = 0; i < length; i++){
freq[(ar[i]-65)]++;
}
return freq;
}
the header file for it (externalLibrary.h):
#ifndef LEARNINGC_EXTERNALLIBRARY_H
#define LEARNINGC_EXTERNALLIBRARY_H
#ifdef __cplusplus
extern "C" {
#endif
char * restrict externalLibrary_Clean(const char* restrict input);
int * restrict externalLibrary_getFrequencies(char * ar, int length);
#ifdef __cplusplus
}
#endif
#endif //LEARNINGC_EXTERNALLIBRARY_H
my main.c file from where all the action is happening:
#include <stdio.h>
#include "externalLibrary.h"
int main() {
char * unfilteredString = "ASDFOIWEGOASDGLKASJGISUAAAA";//if this exceeds 26 characters, the program breaks
char * cleanString = externalLibrary_Clean(unfilteredString);
//int * charDist = externalLibrary_getFrequencies(cleanString, 25); //this works just fine... for now
printf("\nOutput: %s\n", unfilteredString);
printf("\nCleaned Output: %s\n", cleanString);
/*for(int i = 0; i < 26; i++){
if(charDist[i] == 0){
}
else {
printf("%c: %d \n", (i + 65), charDist[i]);
}
}*/
return 0;
}
I'm extremely well versed in Java programming and I'm trying to translate my knowledge over to C as I wish to learn how my computer works in more detail (and have finer control over things such as memory).
If I were solving this problem in Java, it would be as simple as creating two class files: one called main.java and one called externalLibrary.java, where I would have static String Clean(string input) and then call upon it in main.java with String cleanString = externalLibrary.Clean(unfilteredString).
Clearly this isn't how C works, but I want to learn how (and why my code is crashing with corrupted top size)
The bug is this line:
char * returnVal = malloc(sizeof(input));
The reason it is a bug is that it requests an allocation large enough space to store a pointer, meaning 8 bytes in a 64-bit program. What you want to do is to allocate enough space to store the modified string, which you can do with the following line:
char *returnVal = malloc(strlen(input) + 1);
So the other part of your question is why the program doesn't crash when your string is less than 26 characters. The reason is that malloc is allowed to give the caller slightly more than the caller requested.
In your case, the message "malloc(): corrupted top size" suggests that you are using libc malloc, which is the default on Linux. That variant of malloc, in a 64-bit process, would always give you at least 0x18 (24) bytes (minimum chunk size 0x20 - 8 bytes for the size/status). In the specific case that the allocation immediately precedes the "top" allocation, writing past the end of the allocation will clobber the "top" size.
If your string is larger than 23 (0x17) you will start to clobber the size/status of the subsequent allocation because you also need 1 byte to store the trailing NULL. However, any string 23 characters or shorter will not cause a problem.
As to why you didn't get an error with a string with 26 characters, to answer that one would have to see that exact program with the string of 26 characters that does not crash to give a more precise answer. For example, if the program provided a 26-character input that contained 3 blanks, this would would require only 26 + 1 - 3 = 24 bytes in the allocation, which would fit.
If you are not interested in that level of detail, fixing the malloc call to request the proper amount will fix your crash.
i am trying to use dgels function of lapacke:
when i use it with malloc fucntion. it doesnot give correct value.
can anybody tell me please what is the mistake when i use malloc and create a matrix?
thankyou
/* Calling DGELS using row-major order */
#include <stdio.h>
#include <lapacke.h>
#include <conio.h>
#include <malloc.h>
int main ()
{
double a[3][2] = {{1,0},{1,1},{1,2}};
double **outputArray;
int designs=3;
int i,j,d,i_mal;
lapack_int info,m,n,lda,ldb,nrhs;
double outputArray[3][1] = {{6},{0},{0}};*/
outputArray = (double**) malloc(3* sizeof(double*));
for(i_mal=0;i_mal<3;i_mal++)
{
outputArray[i_mal] = (double*) malloc(1* sizeof(double));
}
for (i=0;i<designs;i++)
{
printf("put first value");
scanf("%lf",&outputArray[i][0]);
}
m = 3;
n = 2;
nrhs = 1;
lda = 2;
ldb = 1;
info = LAPACKE_dgels(LAPACK_ROW_MAJOR,'N',m,n,nrhs,*a,lda,*outputArray,ldb);
for(i=0;i<m;i++)
{
for(j=0;j<nrhs;j++)
{
printf("%lf ",outputArray[i][j]);
}
printf("\n");
}
getch();
return (info);
}
The problem may come from outputArray not being contiguous in memory. You may use something like this instead :
outputArray = (double**) malloc(3* sizeof(double*));
outputArray[0]=(double*) malloc(3* sizeof(double));
for (i=0;i<designs;i++){
outputArray[i]=&outputArray[0][i];
}
Don't forget to free the memory !
free(outputArray[0]);
free(outputArray);
Edit : Contiguous means that you have to allocate the memory for all values at once. See http://www.fftw.org/doc/Dynamic-Arrays-in-C_002dThe-Wrong-Way.html#Dynamic-Arrays-in-C_002dThe-Wrong-Way : some packages, like fftw or lapack require this feature for optimization. As you were calling malloc three times, you created three parts and things went wrong.
If you have a single right hand side, there is no need for a 2D array (double**). outputArray[i] is a double*, that is, the start of the i-th row ( row major). The right line may be outputArray[i]=&outputArray[0][i*nrhs]; if you have many RHS.
By doing this in your code, you are building a 3 rows, one column, that is one RHS. The solution, is of size n=2. It should be outputArray[0][0] , outputArray[1][0]. I hope i am not too wrong, check this on simple cases !
Bye,
Suppose I have
void * space= malloc(500) //alloc a block of memory
and I need to write two strings and an int: "Hello world", "Goodbye friend",5 in memory address 50,150,380, correspondance to space.I tried the approach:
int * insert = (int *)(space +380);
*insert = 5;
char * insert2 = (char *)(space+50);
*insert2 = "Hello world";
char * insert3 = (char *)(space + 150);
strcpy(insert3,"Goodbye friend");
the int has no problem, but the two string throws error messages. so what's the correct method to do this? Also, how can you check if there is an overlap in memory between those inputs (the string inputed can be arbitarily long?
Looking at this, I'm pretty sure that you trying to store "H" (which is a constant in C) that is of type int into insert2. In other words, your trying to store a type int (sizeof("H") == 2 on my compiler) into a char (sizeof(char) == 1). From there, it is up to your compiler to allow this behavior or throw an error. Plus "Hello world" is an immutable string that is located only in ROM, so you can't directly modified the string anyways.
To demonstrate, I ran:
printf("val:%d\n", "H");
printf("val:%d\n", "He");
printf("val:%d\n", "He");\\note 2nd instance the same
printf("val:%d\n", "Hel");
printf("val:%d\n", "Hell");
printf("val:%d\n", "Hello");
printf("val:%d\n", "Hello ");
printf("val:%d\n", "Hello W");
printf("val:%d\n", "Hello Wo");
printf("val:%d\n", "Hello Wor");
printf("val:%d\n", "Hello Worl");
printf("val:%d\n", "Hello World");
Yields:
val:4196404
val:4196406
val:4196406\\note 2nd instance the same
val:4196409
val:4196413
val:4196418
val:4196424
val:4196431
val:4196439
val:4196448
val:4196458
val:4196469
As #Rasmus was saying:
char * insert2 = (char *)(space+50);
strcpy(insert2 , "Hello world\0");// the '\0' if changing strings
Should resolve the issue.
BUT, I do not see much practical use for this, besides being vague/indecipherable, and not to mention wasteful of memory (unless you calculate it all out), so I recommend just using a struct:
struct blockOfMem{
int num;
char* str1;
char* str2;
};
....
struct blockOfMem space;
space.num = 5;
strcpy(space.str1, "Hello world\0");// the '\0' if changing strings
strcpy(space.str2, "Goodbye friend\0");// the '\0' if changing strings
This makes you code much more readable and practical to use.
Firstly, using strcpy i could make your code compile.
void * space= malloc(500); //alloc a block of memory
int * insert = (int *)(space +380);
*insert = 5;
char * insert2 = (char *)(space+50);
strcpy(insert2 , "Hello world");
char * insert3 = (char *)(space + 150);
strcpy(insert3,"Goodbye friend");
I would use strlen to get the length of the string and then sizeof(char). To get how much to allocate:
malloc(sizeof(int)+sizeof(char)*(1+strlen(string1))); // 1+ since you need to count the \0.
http://www.cplusplus.com/reference/cstring/strlen/
I'm trying to write a kernel module which writes some data to a proc file. I'm trying to write something like 5000 characters but when I say $>cat /proc/myentry I can read only 1000 characters.
int procfile_read(char *buffer, char **buffer_location, off_t offset, int buffer_length, int *eof, void *data){
int ret;
static char my_buffer[4096];
if (offset > 0) {
ret = 0;
} else {
ret = sprintf(my_buffer, LARGE STRING HERE);
}
*buffer_location=my_buffer;
return ret;
}
This is my code. Thanks in advance.
I had exactly this problem.
One issue in the original post, the if (offset>0) is used many times in examples of small proc files. The read is called multiple times until we return a 0 to indicate that there is no more data. So the if (offset>0) means we return (length of the buffer) as 0.
There are 3 ways to return data with this function. Look at the source code comments, line 75 onwards :
For large files (method 2 from comments), I did the following :-
For each lump of your large data, copy 'buffer_length' of data to 'buffer'.
Set '*start' (or in your case *buffer_location) to 'buffer'.
return the amount of data you wrote (typically 'buffer_length')
Finally, all data will be written and you return 0.
This worked for me with several Meg of data.
I am not a kernel expert, but in linux-3.1.6/fs/proc/task_mmu.c I see some code like
seq_printf(m,
"VmPeak:\t%8lu kB\n"
"VmSize:\t%8lu kB\n"
"VmLck:\t%8lu kB\n"
"VmHWM:\t%8lu kB\n"
"VmRSS:\t%8lu kB\n"
"VmData:\t%8lu kB\n"
"VmStk:\t%8lu kB\n"
so this suggests that you might want to use seq_printf not sprintf .... The m is a
struct seq_file * pointer.
As a general rule, you'll learn a lot by studying the free software source code which you are extending. In your case, it is the Linux kernel source code
I need help getting started with libsndfile.
I have four .wav files (all the same sample rate).
I want to join the first two together,then the next two together,
Then mix the resulting 2 .wav files into one.
For mixing two wav files.
#include <cstdio>
#include <sndfile.h>
#include <windows.h>
#include <cstdlib>
#include <cmath>
#define BUFFER_LEN 1024
#define MAX_CHANNELS 6
static void data_processing (double *data, int count, int channels) ;
int main (void) {
static double data [BUFFER_LEN] ;
static double data2 [BUFFER_LEN] ;
static double outdata [BUFFER_LEN] ;
SNDFILE *infile, *outfile, *infile2 ;
SF_INFO sfinfo ;
int readcount ;
SF_INFO sfinfo2 ;
int readcount2 ;
const char *infilename = "inputOne.wav" ;
const char *infilename2 = "inputTwo.wav" ;
const char *outfilename = "output.wav" ;
if (! (infile = sf_open (infilename, SFM_READ, &sfinfo))) {
/* Open failed so print an error message. */
printf ("Not able to open input file %s.\n", infilename) ;
/* Print the error message from libsndfile. */
puts (sf_strerror (NULL)) ;
return 1 ;
} ;
if (sfinfo.channels > MAX_CHANNELS) {
printf ("Not able to process more than %d channels\n", MAX_CHANNELS) ;
return 1 ;
} ;
if (! (infile2 = sf_open (infilename2, SFM_READ, &sfinfo2))) {
/* Open failed so print an error message. */
printf ("Not able to open input file %s.\n", infilename2) ;
/* Print the error message from libsndfile. */
puts (sf_strerror (NULL)) ;
return 1 ;
} ;
if (sfinfo2.channels > MAX_CHANNELS) {
printf ("Not able to process more than %d channels\n", MAX_CHANNELS) ;
return 1 ;
} ;
/* Open the output file. */
if (! (outfile = sf_open (outfilename, SFM_WRITE, &sfinfo))) {
printf ("Not able to open output file %s.\n", outfilename) ;
puts (sf_strerror (NULL)) ;
return 1 ;
} ;
while ((readcount = sf_read_double (infile, data, BUFFER_LEN)) &&
(readcount2 = sf_read_double (infile2, data2, BUFFER_LEN))) {
data_processing (data, readcount, sfinfo.channels) ;
data_processing(data2, readcount2, sfinfo2.channels) ;
for(int i=0; i < 1024;++i) {
outdata[i] = (data[i] + data2[i]) -(data[i])*(data2[i])/65535;
}
sf_write_double (outfile, outdata , readcount) ;
} ;
/* Close input and output files. */
sf_close (infile) ;
sf_close (infile2) ;
sf_close (outfile) ;
system("PAUSE");
return 0 ;
} /* main */
static void data_processing(double *data, int count, int channels) {
double channel_gain [MAX_CHANNELS] = { 0.5, 0.8, 0.1, 0.4, 0.4, 0.9 } ;
int k, chan ;
for (chan = 0 ; chan < channels ; chan ++)
for (k = chan ; k < count ; k+= channels)
data [k] *= channel_gain [chan] ;
return ;
} /* data_processing */
This is how i am mixing two simple wav files which are 16 bit signals.
First of all audio mixing is not easy as one might think. There might be many ambiguities after joining two signals. In my case it is working just fine as much as i need it, but if you need exact results you might need to google more for exact superimposing signals over each other.
For joining two wav files you can just read your first wav file copy the data in the result wav file and finally append the data of the second wav file to the result wav file.
This link might also be useful for you
http://www.vttoth.com/digimix.htm
This is old, but I'm reading it, so somebody else inevitably will do so.
I agree in general with nishantmishra regarding the use of libsndfile, but this mixing algorithm would cause a certain amount of distortion if it was doing as the author expects:
outdata[i] = (data[i] + data2[i]) -(data[i])*(data2[i])/65535;
(In reality, that last term only adds a little bit of low-level noise...before reading V-Toth's article I thought it was an interesting form of dither),
Assuming this was applied in a way that works as intended (floating point audio ranges from -1.0 to 1.0, so dividing by 65535 reduces the product by 96 dB, which makes it inaudible for 16-bit audio). If you actually want to implement this method, then read on in V Toth's posting about doing this for signed data.
Whether signed or unsigned, you will be adding inter-modulation distortion (even if it isn't nasty audible, it will be there). In other words, this would work great for voice, or low bit-rate (telephone) audio where the distortion in the transmission channel far exceeds the intermodulation distortion added by product of channels.
If you are just processing two files, and not doing so in real-time (play-back as you read blocks from the file or stream), you can normalize both files, apply mix gain such that gain1+gain2 = 1.0, and sum them together. These low resolution challenges as mentioned by V Toth are not a large concern with 32-bit float or 64-bit double.
Finally, if you are concerned about one source being too quiet while the other is silent, then you can apply a dynamic range compressor cross-linked to the other channel. Another tactic is to apply the same algorithm, but to the envelopes of the audio, and not to individual samples:
outEnvelope[i] = (envelope1[i] + envelope2[i]) \
-(envelope1[i])*(envelope2[i]);
outdata[i]=outEnvelope[i]*(data[i] + data2[i]);
Where envelope =
envelope1[i]=sqrt(lowPassFilter(data[i]*data[i]));//moving RMS representation
And the low pas filter cutoff frequency is something on the order of <10Hz to minimize harmonic distortion.
Really, I think all you want to do most times is to drop that last term and use this line:
outdata[i] = (data[i] + data2[i]);
As long as the sum of channel_gain[] = 1.0, you will get a good result. The code by nishantmishra works well because that last distortion-adding term is reduced to the noise floor, so you may as well save CPU time and eliminate it.