A process ran somewhere and initialized an lmdb environment with some max number of named databases, say 10, via mdb_env_set_maxdbs(). Is it something final about an instantiated lmdb environment, or can the max number of named databases, be re-set afterwards? Say, 20 or 5 in this case?
From the documentation
This function may only be called after mdb_env_create() and before mdb_env_open().
This is because the mdb_env_open function uses this parameter to allocate various arrays so trying to change it afterwards would not have an effect (and in fact errors with EINVAL).
You should just set this number higher then you think you could possibly need (within reason). There is almost no overhead in having this be, for example, 128.
Related
I am using the MSIX manager tool to convert a *.msix (an application installer) to a *.vhdx so that it can be mounted in an Azure virtual machine. One of the flags that the tool requires is -vhdSize, which is in megabytes. This has proven to be problematic because I have to guess what the size should be based off the MSIX. I have ran into numerous creation errors due to too small of a vhdSize.
I could set it to an arbitrarily high value in order to get around these failures, but that is not ideal. Alternatively, guessing the correct size is an imprecise science and a chore to do repeatedly.
Is there a way to have the tool dynamically set the vhdSize, or am I stuck guessing a value that is both large enough to accommodate the file, but not too large as to waste disk space? Or, is there a better way to create a *.vhdx file?
https://techcommunity.microsoft.com/t5/windows-virtual-desktop/simplify-msix-image-creation-with-the-msixmgr-tool/m-p/2118585
There is an MSIX Hero app that could select a size for you, it will automatically check how big the uncompressed files are, add an extra buffer for safety (currently double the original size), and round it to the next 10MB. Reference from https://msixhero.net/documentation/creating-vhd-for-msix-app-attach/
I have implemented several counters in my project using the new increment API. There are two functions whose implementation is a simple one line:
exports.MY_FUNC = functions.firestore.document('/PATH_TO_DOC/{id}').onCreate((snapshot, context) => {
return admin.firestore().collection('COUNTERS').doc('DOC').update({COUNTER: admin.firestore.FieldValue.increment(1)});
});
(everything in CAPS is a generic replacement for the actual values which I have).
This function works perfectly well. However, each day, there are instances when it fails with "Error: memory limit exceeded. Function invocation was interrupted." It puzzles me as it works most of the time correctly.
I do know how to increase the memory limit, and I am going to increase it to 512MB right now. However, I don't like the unpredictability of this setup. Undoubtedly, the same function operating a counter on the same doc should either work correctly with 256MB or should fail every time.
I am also quite stunned that an operation which fits into one line may require more than 256MB. I have other functions which perform 3-4 different steps (reads, writes) and they work correctly with 256MB.
Given the unpredictability, I now worry that 512MB may also not be enough. Unfortunately, it doesn't seem that docs mention the memory requirements for increment API, so it looks like my only option is to deploy and see if the new memory limit is sufficient.
I would appreciate if anyone had any suggestions on any points below:
Is there a way to enforce consistent behaviour? I want the function to either fail every time if 256MB is not enough or succeed every time if it is enough.
How can developers estimate in advance the memory requirement of a particular function? I honestly do not think that it is OK to have a working function with 256MB during debugging only to find out that it may exceed memory limit in production.
Thanks for any input.
I have a class that does some parsing of two large (~90K rows, 11 columns in the first and around ~20K, 5 columns in the second) CSV files. According to the specification I'm working with the CSV files can be externally changed (removing/adding of new rows; columns remain constant as well as the paths). Such updates can happen at any time (though highly unlikely that an update will be launched in time intervals shorter than a couple of minutes) and an update of any of the two files has to terminate the current processing of all that data (CSV, XML from an HTTP GET request, UDP telegrams), followed by re-parsing the content of each of the two (or just one if only one has changed).
I keep the CSV data (quite reduced since I apply multiple filters to remove unwanted entries) in memory to speed working with it and also to avoid unnecessary IO operations (opening, reading, closing file).
Right now I'm looking into the QFileSystemWatcher, which seems to be exactly what I need. However I'm unable to find any information on how it actually works internally.
Since all I need is to monitor 2 files for changes the number of files shouldn't be an issue. Do I need to run it in a separate thread (since the watcher is part of the same class where the CSV parsing happens) or is it safe to say that it can run without too much fuss (that is it works asynchronously like the QNetworkAccessManager)? My dev environment for now is a 64bit Ubuntu VM (VirtualBox) on a relatively powerful host (a HP Z240 workstation) however the target system is an embedded one. While the whole parsing of the CSV files takes just 2-3 seconds at the most I don't know how much performance impact there will be once the application gets deployed so additional overhead is something of a concern of mine.
Inside a particular quarantine is all of the stuff one needs to run an application (bin, share, lib, etc.). Ideally, the quarantine has no leaks, which means it's not relying on any code outside of itself on the system. A quarantine can be defined as a set of executables (and some environment settings needed to make them run).
I think it will be beneficial to separate the built packages enough such that upgrading to a newer version of the quarantine won't require rebuilding the whole thing. I'll be able to update just a few packages, and then the new quarantine can use some of old parts and some of the new parts.
One issue I'm wondering about is the environment variables I'll be setting up to use a particular quarantines.
Is there a hard limit on how big PATH can be? (either in number of characters, or in the number of directories it contains) Does path length affect performance?
There's a hard limit. It's something like 32MB.
Yes, you can get it long enough to affect performance easily. Number of entries is the primary limiting factor, followed by number of / characters (this should not show itself unless the path depth exceeds some outrageous number like 30).
I have made a Linux CUI app that communicates between processes via Message-quene.
There is no problem with it as long as it is a single instance. However when there are multiple instances of the same app, the messages in the quene get sent to the wrong instance.
I understand this can be avoided by giving the msgget function a unique key.
Using ftok() to create a key, but since the variables are the same they result in
identical keys.
Can someone guide me how to have a unique key for each instance?
The only idea I have now is to randamize the variable given to ftok, and
I know that cant be right.
Be careful with ftok!
This will only be unique for a given file system and only if then if
the file system is not heavily used.
fttok is driven by the file entry number in the file system.
This used to be a pretty good way of getting unique values but time
and Moores law caught up with it a few years ago. It works on the
lower 8 bits of the file number but the actual file number is now 32 bits
and numbering starts again for each file system.
Process id is a pretty good choice, they do get re-cycled but not as
long as the process is still alive.
You could try using the process id. My google foo got this
Looking globally unique ids usually called Guid or Uuid. There must be a library you can use to generate them. They are unique strings made from your nic address, the current time, and a random number.
How about the clock? WikiPedia say's it's better than RDTSC (and SMP safe).
"Under Linux, similar functionality is provided by reading the value of CLOCK_MONOTONIC clock using POSIX clock_gettime function."