Hey I'm currently trying to emulate an old gaming console. I'm run into a little problem that I do not know how to fix.
More specific, I'm doing a emulator for a GBA. But to make it a little more challenging, I have decided to emulate the games as external devices with memory, possible extra CPUs and such. Just like they are in the real world. To do this I need to create a self contained executable file, where there is allocated some fix size spare memory to save data like saved games.
Is this possible? I know this is uncommon and bad habit. I don't care, it is like people saying DON'T USE GOTO in C, the instruction are there to be used, and I use it with good effect an no headache, and no weird behaviour, it just takes a moment of planning, and maybe a course in compiler implementation to know how.
The language are not important, C will do, but if .net supported it I'll prefer that.
You can take the approach used by some self-extracting archives. Append a marker GUID and a data block after the compiled executable
(executable file) + (marker GUID) + (data area)
In Windows for example, using a command box:
type "c:\vsprojects\my-gba\release\my-gba.exe" "c:\data-files\guid.txt" "c:\data-files\save-data-area.txt" > "c:\data-files\gba-combo.exe"
In Linux you'd cat the 3 pieces together into a new file to form your executable-with-data.
To find the start of the data area, you search for the GUID in the EXE and then seek past it.
There is a very slight chance the GUID could occur naturally in your code, but the odds of that become extremely low the longer the GUID gets. You can use an online GUID generator and combine multiple strings in the GUID text file for example
731056dd-1dd7-4f46-b90b-2ad623198404 a2d0cd76-f8fc-4bcf-8a15-80ca4d9b205f
You do need to make sure you don't write your code in a way that embeds the GUID as a string constant earlier in your code, before the appended marker.
BAD:
const char* foo = "731056dd-1dd7-4f46-b90b-2ad623198404 a2d0cd76-f8fc-4bcf-8a15-80ca4d9b205f"
-- code that seeks for foo ---
OK:
const char* arr[] = { "731056dd-1dd7-4f46-b90b-2ad623198404", "BREAK-MATCH", "a2d0cd76-f8fc-4bcf-8a15-80ca4d9b205f" };
-- code sets foo from arr [0], " ", [2] then seeks for foo ---
Related
In C, I can define many structures and structure of structures.
From a buffer, I can just set the pointer at the beginning of this structure to say this buffer represents this structure.
Of course, I do not want to copy anything, just mapping, otherwise I loose the benefit of the speed.
Is it possible in NodeJs ? How can I do ? How can I be sure it's a mapping and not creating a new object and copy information inside ?
Example:
struct House = {
uint8 door,
uint16BE kitchen,
etc...
}
var mybuff = Buffer.allocate(10, 0)
var MyHouse = new House(mybuff) // same as `House* MyHouse = (House*) mybuff`
console.log(MyHouse.door) // will display the value of door
console.log(MyHouse.kitchen) // will display the value of kitchen with BE function.
This is wrong but explain well what I am looking for.
This without copying anything.
And if I do MyHouse.door=56, mybuff contains know the 56. I consider mybuff as a pointer.
Edit after question update below
Opposed to C/C++, javascript uses pionters by default, so you don't have to do anything. It's the other way around, actually: You have to put some effort in if you want a copy of the current object.
In C, a struct is nothing more than a compile-time reference to different parts of data in the struct. So:
struct X {
int foo;
int bar;
}
is nothing more than saying: if you want bar from a variable with type X, just add the length of foo (length of int) to the base pointer.
In Javascript, we do not even have such a type. We can just say:
var x = {
foo: 1,
bar: 2
}
The lookup of bar will automatically be a pointer (we call them references in javascript) lookup. Because javascript does not have types, you can view an object as a map/dictionary with pointers to mixed types.
If you, for any reason, want to create a copy of a datastructure, you would have to iterate through the entire datastructure (recursively) and create a copy of the datastructure manually. The basic types are not pointer based. These include number (Javascript automatically differentiates between int and float under the hood), string and boolean.
Edit after question update
Although I am not an expert on this area, I do not think it is possible. The problem is, the underlying data representation (as in how the data is represented as bytes in memory) is different, because javascript does not have compile-time information about data structures. As I said before, javascript doesn't have classes/structs, just objects with fields, which basically behave (and may be implemented as) maps/dictionaries.
There are, however, some third party libraries to cope with these problems. There are two general approaches:
Unpack everything to javascript objects. The data will be copied, but you can work with it as normal javascript objects. You should use this if you read/write the data intensively, because the performance increase you get when working with normal javascript objects outweighs the advantage of not having to unpack the data. Link to example library
Leave all data in the buffer. When you need some of the data, compute the location of the data in the buffer at runtime, and read/write at this location accordingly. Because the struct data location computations are done in runtime, you should use this only when you have loads of data and only a few reads/writes to it. In this case the performance decrease of unpacking all data outweighs the few runtime computations that have to be done. Link to example library
As a side-note, if the amount of data you have to process isn't that much, I'd recommend to just unpack the data. It saves you the headache of having to use the library as interface to your data. Computers are fast enough nowadays to copy/process some amount of data in memory. Also, these third party libraries are just some examples. I recommend you do a little more research for libraries to decide which one suits your needs.
I am in a process of designing a program that runs end-user scripts written in Lua. The program defines an interface that tells how many parameters are being passed to which function, their order and so on. At some point in the future, the interface may change so I would like to pass a version information of the host program to a script. My common practice is to use MAJOR.MINOR.PATCH format.
The goal is to keep the representation simple as much as possible, so that the information can be processed using arithmetic operations, like so:
if program.version < SCRIPT_SUPPORTS_VERSION then
error ("This version is not supported, please upgrade...")
end
Common sense dictates to use integers written in base of sixteen, where each byte represents a part in the version format. Like so:
-- Version 1.3.2
-- Compatible with versions >=1.2.0
--
local program = {}
program.version = 0x010302
local SCRIPT_SUPPORTS_VERSION = 0x010200
As I mentioned before, this is the first thing that came to my head when thinking about the problem and it seems to work. On other hand, hexadecimal numbers may seem scary to some.
I would like to know your opinion, please propose alternative solutions.
For fun I'm implementing an NES emulator. I'm currently reading through documentation for the 6502 CPU and I'm a little confused.
I've seen documentation stating because the 6502 is little-endian so when using absolute addressing mode you need to swap the bytes. I'm writing this on an x86 machine which is also little-endian, so I don't understand why I couldn't simply cast to a uint16_t*, dereference that, and let the compiler work out the details.
I've written some simple tests in google test and they seem to agree with me.
// implementation of READ16
#define READ16(addr) (*(uint16_t*)addr)
TEST(MemMacro, READ16) {
uint8_t arr[] = {0xFF,0xCC};
uint8_t *mem = (&arr[0]);
EXPECT_EQ(0xCCFF, READ16(mem));
}
This passes, so it appears my supposition is correct, but I thought I'd ask someone with more experience than I.
Is this correct for pulling out the operand in 6502 absolute addressing mode? Am I possibly missing something?
It will work for simple cases on little-endian systems, but tying your implementation to those feels unnecessary when the corresponding portable implementation is simple. Sticking to the macro, you could do this instead:
#define READ16(addr) (addr[0] + (addr[1] << 8))
(Just to be pedantic, you should also make sure that addr[1] can't be out-of-bounds, and would need to add some more parentheses if addr could be a complex expression.)
However, as you keep developing your emulator, you will find that it's most natural to use a pair of general-purpose read_mem() and write_mem() functions that operate on single bytes. Remember that the address space is split up into multiple regions (RAM, ROM, and memory-mapped registers from the PPU and APU), so having e.g. a single array that you index into won't work well. The fact that memory regions can be remapped by mappers also complicates things. (You won't have to worry about that for simple games though -- I recommend starting with Donkey Kong.)
What you need to do is to figure out what region or memory-mapped register the address belongs to inside your read_mem() and write_mem() functions (this is called address decoding), and do the right thing for the address.
Returning to the original question, the fact that you'll end up using read_mem() to read the individual bytes of the address anyway means that the uint16_t casting trickery is even less likely to be useful. This is the simplest and most robust approach w.r.t. handling corner cases, and what every emulator I've seen does in practice (Nestopia, Nintendulator, and FCEUX).
In case you've missed it, the #nesdev channel on EFNet is very active and a good resource by the way. I assume you're already familiar with the NESDev wiki. :)
I've also been working on an emulator which can be found here.
As a hobby project called 'beercan', I'm reverse-engineering the resource files of the Torchlight games. Using an okay-ish hex editor, I try to guess the structure of the files, and then I model my ideas, use cereal to write Getters (and later some Putters), and try to decode every file in an application of the library.
I've just started on Torchlight's compiled layout files (*.LAYOUT in TL1, *.LAYOUT.cmp in TL2). The format turns out to be a little trickier than the dat files, but I think I figured out the basic structure, and how they are encoded in the TL2 files. so I'm trying to make a map of file versions, tag numbers, and guessed data types.
To do so, I wrote an application that flattens the data structure, leaving only the guessed type of the values of the leaves, each annotated with the file version and the node and leaf tag numbers. I turn this into a map from the file version and tag numbers to a set of the guessed types. For every file, I'd expect this Map to maybe take twice the file size in memory. (Not sure, though.) Then, I merge these maps, and I print the map.
For some reason, even if I only take 20MB worth of files (100 files), memory usage increases linearly to about 200MB, then decreases to the final size of the resulting map, and then deflates rapidly as I print it.
I wouldn't expect this memory usage. Does anyone know how I could fix it? I've tried to force values after decoding them (using deepseq), I've tried adding bangs to data types, but this hasn't really helped. I've tried copying all bytestrings I keep in the file structure, which brought down the memory usage a bit, but it's still unacceptably high, especially when I want to analyze the entire dataset (200MB+ of original files).
-edit- I've pushed a (not very S)SCCE to demonstrate the performance issue, (accidentally) along with my profiling results.
Clone the repository.
cabal configure, with flags to enable profiling (is it normal to need --enable-library-profiling --enable-executable-profiling --ghc-options="-rtsopts -prof"?)
cabal build
cd test, and run StressTest.sh.
This script tries to load a regular TL2 layout file 100 times. On my machine, top says it takes about 500MB of memory, and the profiling results are consistent with my description above.
I totally agree with #petrpudlak, we would need actual code to make any meaningful comments to the question "why does my code use so much memory?" :) (sorry, you did offer code), however, some of the patterns you describe are pretty typical in Haskell and some generic discussion is possible.
First of all, note that native Haskell types use a lot more memory than you might guess. Take a look at the ghc memory footprint page at http://www.haskell.org/haskellwiki/GHC/Memory_Footprint. Note that even a simple Char will take a full 16 bytes of memory! Add to that pointers for linked list items in a String, and you will easily use more than an order of magnitude greater memory than you might have guessed. If memory is important, you should use another data type, like Data.Text or Data.ByteString, which store Strings internally more like c would (as a block of bytes in memory, with 1-4 bytes per char, depending on encoding and what char is used). If data other than Strings are the problem, you can use unboxed arrays for arbitrary data types.
Second of all, if possible, you can cut down memory usage by processing items in series (where the memory will be garbage collected right away). Haskell laziness often does this for you automatically, for instance, try to run the following program
import Data.Char
main = interact $ map toUpper
As you type, the output will appear continuously (your OS, not Haskell, may buffer full lines, so you may need to hit 'enter' before seeing anything, but you will see output update for each 'enter'). Rather than loading the whole input into memory and then processing all at once, Char memory is being created and garbage collected Char by Char.
Of course this isn't always possible (ie- if you have to process the data in a very nonlocal way), but most of the time at least parts of the code can be refactored this way to cut down total memory usage.
Edit- Sorry, I just realized that you did post a link to the code, and you are using ByteString..... So some of what I wrote isn't valid. But I do still see boxed lists and unpacking of the ByteString, so I will leave the answer as it is.
The memory usage pattern sounds like your application is building up a lot of unnecessary thunks and then memory consumption starts going down when those thunks get evaluated. I only glanced at your code quickly but one simple change you could try is to replace all imports of Data.Map with Data.Map.Strict. This is especially important if you are doing a lot of updates on the values inside a Map without forcing evaluation in between.
Another things you should be aware of is that replicateM is quite inefficient with larger numbers in a strict monad (see e.g. this answer). I'm not sure what kinds of counts you are usually dealing with in your application, but it's good to keep in mind.
It might also help to use strict fields in simple container data types like your LeafValue type and compile with -funbox-strict-fields (and -O2 of course).
What are the main existing approaches to hide the value of literals in code, so that they are not easily traced with just an hexdumper or a decompiler?
For example, instead of coding this:
static final int MY_VALUE = 100;
We could have:
static final int MY_VALUE = myFunction1();
private int myFunction1(){
int i = 23;
i += 8 << 4;
for(int j = 0; j < 3; j++){
i-= (j<<1);
}
return myFunction2(i);
}
private int myFunction2(int i){
return i + 19;
}
That was just an example of what we're trying to do. (Yes, I know, the compiler may optimize it and precalculate the constant).
Disclaimer: I know this will not provide any aditional security at all, but it makes the code more obscure (or interesting) to reverse-engineer. The purpose of this is just to force the attacker to debug the program, and waste time on it. Keep in mind that we're doing it just for fun.
Since you're trying to hide text, which will be visible in the simple dump of the program, you can use some kind of simple encryption to obfuscate your program and hide that text from prying eyes.
Detailed instuctions:
Visit ROT47.com and encode your text online. You can also use this web site for a more generic ROTn encoding.
Replace contents of your string constants with the encoded text.
Use the decoder in your code to transform the text back into its original form when you need it. ROT13 Wikipedia article contains some notes about implementation, and here is Javascript implementation of ROTn on StackOverflow. It is trivial to adapt it to whatever language you're using.
Why use ROT47 which is notoriously weak encryption?
In the end, your code will look something like this:
decryptedData = decryptStr(MY_ENCRYPTED_CONSTANT)
useDecrypted(decryptedData)
No matter how strong your cypher, anybody equipped with a debugger can set a breakpoint on useDecrypted() and recover the plaintext. So, strength of the cypher does not matter. However, using something like Rot47 has two distinct advantages:
You can encode your text online, no need to write a specialized program to encode your text.
Decryption is very easy to implement, so you don't waste your time on something that does not add any value to your customers.
Anybody reading your code (your coworker or yourself after 5 years) will know immediately this is not a real security, but security by obscurity.
Your text will still appear as gibberish to anyone just prying inside your compiled program, so mission accomplished.
Run some game of life variant for a large number of iterations, and then make control flow decisions based on the final state vector.
If your program is meant to actually do something useful, you could have your desired branches planned ahead of time and choose bits of the state vector to suit ("I want a true here, bit 17 is on, so make that the condition..")
You could also use some part of compiled code as data, then modify it a little. This would be hard to do in a program executed by virtual machine, but is doable in languages like asm or c.