When we call a function, its stack is something like:
LOW MEMORY ADDRESS
localvariables
saved frame pointer
return address
....
HIGH MEMORY ADDRESS
Why does it fill data into a buffer the direction is from low to high memory address?
Many people tell me: "because this is how it works", but I think someone in some book or other has written why we have this behavior but I'm unable to find a good resource about.
I think you are misunderstanding or confusing a few things.
in you example you seem to mix up, operation system functionality with program and compiler operation.
If you allocate multiple memory addresses there is always a lower address and a higher address. You can only change that by writing everything to the same address which might result in a very limited or useless program.
there are many buffer implementations, depending on your programming language, framework, ...
which one you choose is up to you, if you use a buffer that is already implemented in a library, of course you have the follow the rules this buffer adds data, because that is how THIS specific buffer works. If you are not happy by how this is done, you need to change the chosen buffer or even the whole library or in extreme cases write your own buffer.
how to add data to a buffer
some buffers allow you to add data anywhere in the buffer, at the cost of e.g. performance or reliability. If you wish to do it that way its up to you.
Related
I am confused about using Buffer.allocUnsafe() and Buffer.alloc() , I know that Buffer.allocUnsafe() creates a buffer with pre-filled data or old buffers, but why do i need such thing if Buffer.alloc() creates a buffer with zero filled data
In Node.js Buffer is an abstraction over RAM, therefore if you allocate it in an unsafe way, there is a high risk of having even some source code in the buffer instance. Try running console.log(Buffer.allocUnsafe(10000).toString('utf-8')) and I guarantee that you will see some code in your stdout.
Allocation is a synchronous operation and we know that single threaded Node.js doesn't really feel good about synchronous stuff. Unsafe allocation is much faster than safe, because the buffer santarization step takes time. Safe allocation is, well, safe, but there is a performance trade off.
I'd suggest sticking to safe allocation first and if you end up with low performance, you can think of ways to implement unsafe allocation, without exposing private stuff. Just keep in mind that allocUnsafe method has the word unsafe for a reason. E.g, if you are going to pass some compliance certification like PCI DSS, I'm pretty sure QSA will notice that and will have a lot of questions.
Buffer.alloc(size, fill, encoding) -> returns a new initialized Buffer
of the specified size. This method is slower than Buffer.allocUnsafe(size) but guarantees that newly created Buffer instances never contain old data that is potentially sensitive.
Buffer.allocUnsafe(size) -> the Buffer is uninitialized, the allocated
segment of memory might contain old data that is potentially
sensitive. Using a Buffer created by Buffer.allocUnsafe() without completely overwriting the memory can allow this old data to be leaked when the Buffer memory is read.
Note: While there are clear performance advantages to using Buffer.allocUnsafe(), extra care must be taken in order to avoid introducing security vulnerabilities into an application
What is the purpose of struct iov_iter ? This structure is being used in Linux kernel instead of struct iovec. There is no any good documentation for iter interface. I had found one document on LWN but I am not able to understand that. Could anyone please help me to understand the iter interface which is being used in Linux kernel ?
One purpose of iovec, which the LWN article states up front, is to process data in multiple chunks.
If you have a number of discrete buffers, chained with pointers, and want to read/write them in one go, you could simply replace this with several read/write ops, but in some cases semantics are associated with read/write boundaries - so ops can't simply be split without changing the meaning. An alternative is to copy all the data in and out of a contiguous buffer, which is wasteful and we want to avoid at all costs.
Using the POSIX readv/writev or, in our case the iov_iter API, reduces the number of system calls, and hence the overhead involved. While in the kernel this doesn't translate to expensive ops like context switches, it is still a minor concern. Drivers also might handle larger chunks of data more efficiently than they would lots of smaller chunks when they have no way to know if there's more to come in the near future - this is especially true with network drivers, although I'm not aware of iov_iter being used there atm.
Another instance of the same situation is I/O to raw disk
devices, which only allow I/O to start and end of block
boundaries. A user might occasionally want to perform random access or overwrite a small piece of the buffer at, say, the start of a block and/or zero the rest.
Scenarios like that is exactly what iovec aimed to address; you can construct an iovec which enables you to do a whole block operation spread over several discrete buffers, which might even include a "scratch" buffer for dumping the parts of a block you read and don't care about processing, and a pre-zeroed buffer for chaining at the end of writev to zero out the rest of a block. Again, I should point out you can use a contiguous buffer with associated copying and/or zeroing, but the iov_iter API provides an alternative abstraction with less overhead, and perhaps easier to reason with when reading the code.
The term for operations like these in vector processing, or parallel computing, is "scatter/gather processing".
So I'm writing a gameboy emulator, and I'm not 100% sure why other projects took the time to break up the memory into proper categories. I don't know if there is a major technical dilemma I'm missing (maybe handling illegal parameters in instructions?), but it seems like the only thing that matters is that the address given by a write instruction is retrievable by the proper read instruction. So for a sub question, if I'm working under the assumption that the assembly is perfectly legal (meaning nothing is trying to read/write where it can't), can I just make a big array and read and write to it?
Note that this is a conceptual question and that I am aware a big array would be a memory hog, I'm not necessarily looking for the best way to do it, simply trying to learn how it works and why other emulator developers did it the way they did.
You are going to have read only memory, read/write memory and memory mapped I/O (peripherals etc). So you need to decode the address to some extent to break it into the major categories, then for the peripherals you have to emulate all of those so you have to get very detailed in your address decoding.
For the peripherals you will need to detect a read/write to some address which you cannot do by simply landing the writes in an array (two writes of the same value for example make a difference, you cant just scan some array to look for changes you have to trigger on reads and writes and perform the hardware action).
If you wish to be cycle accurate you will also need to know the timings for the rams and roms in order to mimic those, depending on how many banks of each or if timing is dependent on that you will need to decode the address further.
Hardware decodes these addresses to the same level so if you are emulating hardware then you need to...emulate hardware...and do the same amount of address decoding.
I'm going to be gameboy specific here. Look at gameboy's address space map. The address space itself is divided, it's not that emulators do it. Hardware itself operates that way.
Here's some of the regions that can't be implemented as just an array:
0x0000-0x3FFF. First bank of a ROM. It's read-only but not quite. Read the next one
0x4000-0x7FFF. Switchable ROM bank, it's also not quite read-only. Cartridges that don't fit into gameboy's address space contain memory bank controller. ROM will write to some specific read-only ROM regions to actually select which ROM bank is mapped into 0x4000-0x7FFF address range. So you have to detect these writes and then redirect reads into the selected ROM bank.
0xA000-0xBFFF. Switchable RAM bank. Same thing as with switchable ROM banks but now for RAM. Cartridges may contain additional RAM that's being mapped into gameboy's address space. Which bank of the RAM is mapped is controlled, again, by writes to specific read-only regions.
0xFF00-0xFF4B. IO ports. Here you have hardware registers mapped into address space. Gameboy has several hardware components each with it's own registers and even memory (display controller, sound processor, timers etc). To control that hardware ROM reads and writes into the IO ports. You obviously have to detect these writes so you can emulate the hardware they correspond to. It's not just CPU and memory you have to emulate. I would even say that the least part of it and the easy one. For example, it much harder to get display controller and sound channels right. They have complicated logic, bugs and very tricky behaviour that's not documented very well but is crucial to achieve accurate emulation. Wave channel in particular gave me a hard time.
We need to store a large 1GB of contiguous bytes in memory for long periods of time (weeks to months), and are trying to choose a Vector/Array library. I had two concerns that I can't find the answer to.
Vector.Unboxed seems to store the underlying bytes on the heap, which can be moved around at will by the GC.... Periodically moving 1GB of data would be something I would like to avoid.
Vector.Storable solves this problem by storing the underlying bytes in the c heap. But everything I've read seems to indicate that this is really only to be used for communicating with other languages (primarily c). Is there some reason that I should avoid using Vector.Storable for internal Haskell usage.
I'm open to a third option if it makes sense!
My first thought was the mmap package, which allows you to "memory-map" a file into memory, using the virtual memory system to manage paging. I don't know if this is appropriate for your use case (in particular, I don't know if you're loading or computing this 1GB of data), but it may be worth looking at.
In particular, I think this prevents the GC moving the data around (since it's not on the Haskell heap, it's managed by the OS virtual memory subsystem). On the other hand, this interface handles only raw bytes; you couldn't have, say, an array of Customer objects or something.
I am writing a simple VM and I have a question on implementing object and structure member access.
Since the begin address of a program is arbitrary on each run, and subsequently the address of each and every of its objects is arbitrary too.
Thus the only way I can think of to access an object or its member object is by accessing an offset from the "base" pointer, which means there is an arithmetic operation needed to access anything in a program structure.
My question is whether this is the way it is done in professional compilers, because obviously this approach adds some overhead to the runtime, and I myself can't think of any way to offload this process from the runtime because of the lack of guarantees for consistency of memory allocation and its address?
Most computers for many decades provide addressing modes that let you specify the address as a combination of a base and an offset, and the actual calculation is carried out in the hardware for no additional cost in CPU clock cycles.
More recent (past few decades) computers offer hardware for virtualizing memory layout, meaning that even through the physical address of an item is different on every run, its address in the virtual address space remains the same. Again, there is no additional cost for using the base address, because the calculations are performed implicitly and invisibly to the executing binary code of a program.