I’m trying to find the least painful way of transporting config files between different environments and I have found many things that can break the system after the transport. I got a script that will keep the values for attributes correct for the ones that are depended on the environment but here is the list of couple of things that I’m not sure about. Maybe someone can shed some light on them.
What I want to do is to simply transport xml file based on the steps from the book for openAM 9 (simply export/import using ssoadm to xml file) but by analyzing the file in depth I find many differences that might break the system, so any help is appreciated.
In every xml file we have sections for ‘iplanet-am-auth-ldap-bind-passwd’ with hash value under it but in one xml file we’re missing one line with hash. I was wondering if we add that line with the correct hash value will it break the system or it won’t matter as long as the hash matches target environment?
Does the size of the ‘iplanet-am-logging-buffer-size’ has to match what was originally setup in the target environment or it will be ok if we overwrite the value from the source xml file?
For some reason we have different links in delegation-rules with the same name, for example:
# environment1 - sms://dc=test-domain,dc=net/sunEntitlementService/1.0/application/ws/1/entitlement/entitlements
# environment2 - sms://dc=test-domain,dc=net/sunEntitlementService/1.0/application/ws/1/entitlement/decision
# environment3 - sms://*dc=test-domain,dc=net/sunIdentityRepositoryService/1.0/application/agent
It could be due the way the server was setup long time ago or due to development processes over time ( I don’t know) but my question is:
If the rule names are the same but some(or all) options/values are different between environments and we overwrite them with the source file from different environment, will this break things or it won’t matter ?
Related
We are trying to migrate our old issue "tracking" system to GitLab.
For legacy reasons the issues have relatively large numbers 800 and above and they are not consecutive.
However for backreference it would be great if we could have one number for each issue and not an "old" and a "new" number, as in some contexts issues are referred to by number (e.g. external parties who in the future also will use GitLab)
I found this Set Minimum Issue Number in Gitlab where issues were created to "fill" the gaps. However this creates a lot of clutter (especially E-Mails Gitlab API - Create issue quietly?).
Any ideas how to solve this?
The ideal flow would be:
use gitlab-api to create issues we have and
add a parameter so set the number of the issue.
When using gitlab, the numbers are filled up by new issues over time,
or they count up from the highest issue number currently in the
project.
If I could actually set the issue number afterwards in the database (as was hinted in the linked question above), how will gitlab handle this? (I don't even know where to start looking in the gitlab code base, any hints on that might also answer this question).
Thanks in advance for any advice on how to tackle this.
I found a trick to do that.
There is exporting to .csv from the GitLab and .csv is an editable format. It can be edited in any excel-like program, or even in a notepad.
Make a first issue, fill it with dummy data (I suggest filling as that's a way to spot which field is responsible for what).
Then fill your data by any way suitable, preferably mass-exported from the previous issue tracker and import back to the GitLab.
If done correctly, this should be possible to be done partly automatically.
I'm a bit curious about Terraform's modules. What exactly are their points? I keep seeing it is supposed to save me time, no copy paste, and such, but I think I might be missing the point because I don't see why I would use them?
I have used them only once so my experience is limited. I have to create 2 different environments and I'm not sure why I would go for it. Maybe I would have a different opinion if I had to do 10 environments.
The environments will be in different accounts, using different vpcs, different IAM, ... Which led me to think I could create basically 2 folders and reference some variables.
Regarding the "no copy-paste", it seems to me you do a file which then refers to the modules and the variables. At some point, you still need to write all the ressources (int-sg, test-sg for example). so why not write it directly into the right folder?
I hope this question makes sense, I would appreciate any constructive opinion
modules are indented to group resources that are required multiple times.
Let's say you have the three environments dev, staging and prod.
You would want to keep them as equal as possible, especially staging and prod.
So, if you want to create a server with an EIP, certain security groups and whatever it needs to be accessible, you can create a module to group these resources.
The module can then easily be included in your .tf files for the respective environments with different parameters for each env.
So you can keep the resources very generic, and also have the avantage that Terraform will automatically update all environments equally in case you change resources in the module.
You'll thus have a single point where to apply modifications, instead of editing all changed resources in all env files.
Now only imagine you have more than 2 or 3 environments (or use cases for the same resource groups) and you'll see the advantage of modules ;)
It appears that the time zone database files used by Nodatime are named by year, with releases within the same year incrementing by a
letter - i.e., "tzdb2019a.nzd" is current as I write this, the next release
will be "tzdb2019b.nzd", and some of the previous versions may have been
"tzdb2018a.nzd", "tzdb2018b.nzd", "tzdb2018c.nzd", etc.
However, I have not been able to find this naming convention formally documented anywhere, and assumptions make me nervous.
I expect the time zone data to change more often than my application
is updated, so the application periodically checks for the latest data file at
https://nodatime.org/tzdb/latest.txt, and downloads a new file if the one in
use is different. Eventually there will be several files locally available.
I want to know that I can sort these by name and be assured that I can
identify the most recent from among those that have already been
downloaded.
That's what I anticipate, certainly. We use the versioning from the IANA time zone page, just with a tzdb prefix and a .nzd suffix. So far, that's been enough, and it has maintained the sort order.
It's possible that we might want to provide other files at some point, e.g. if there's no IANA changes for a long time (as if!) but the CLDR Windows mapping files change significantly. I don't have any concrete plans for what I'd do in that case, but I can imagine something like tzdb2019-2.nzd etc.
It's hard to suggest specific mitigations against this without knowing the exact reason for providing other files, but you could potentially only download files if they match a regex of tzdb\d{4}[a-z]+.nzd.
I'd certainly communicate on the Noda Time discussion group before doing anything like this, so if you subscribe there you should get advance warning.
Another nasty possibility that we might need more than 26 releases in a single calendar year... IANA says that would go 2020a...2020z, then 2020za...2020zz etc. The above regex handles that situation, and it stays sortable in the normal way.
Another option I could provide is an XML or JSON format for "all releases" - so just like there's https://nodatime.org/tzdb/index.txt that just lists the files, I could provide https://nodatime.org/tzdb/index.json that lists the files and release dates. If you kept hold of that file along with the data, you'd always have more information. Let me know if that's of interest to you and I'll look into implementing it.
I am trying to extract some features from an audio sample using OpenSMILE, but I'm realizing how difficult it is to set up a config file.
The documentation is not very helpful. The best I could do was run some of the sample config files that are provided, see what came out, and then go into the config file and try to determine where the feature was specified. Here's what I did:
I used the default feature set used from The INTERSPEECH 2010 Paralinguistic Challenge (IS10_paraling.conf).
I ran it over a sample audiofile.
I looked at what came out. Then I read the config file in depth, trying to find out where the feature was specified.
Here's a little markdown table showing the results of my exploration:
| Feature generated | instruction in the conf file |
|-------------------|---------------------------------------------------------|
| pcm_loudness | I see: 'loudness=1' |
| mfcc | I see a section: [mfcc:cMfcc] |
| lspFreq | no matches for the text 'lspFreq' anywhere |
| F0finEnv | I seeF0finalEnv = 1 under [pitchSmooth:cPitchSmoother] |
What I see, is 4 different features, all generated by a different instruction in the config file. Well, for one of them, there was no disconcernable instruction in the config file that I could find. With no pattern or intuitive syntax or apparent system, I have no idea how I can eventually figure out how to specify my own features I want to generate.
There are no tutorials, no YouTube videos, no StackOverflow question and no blog posts out there talking about how this could be done. Which is really surprising since this is obviously a huge part of using OpenSMILE.
If anyone finds this, please, can you advise me on how to create custom config files of OpenSMILE? Thanks!
thanks for your interest in openSMILE and your eagerness to build your own configuration files.
Most users in the scientific community actually use openSMILE for its pre-defined config files for the baseline feature sets, which in version 2.3 are even more flexible to use (more commandline options to output to different file formats etc.).
I admit that the documentation provided is not as good as it could be. However, openSMILE is a very complex piece of Software with a lot of functionality, of which only the most important parts are currently well documented.
The best starting point would be to read the openSMILE book and the SIG'MM tutorials all referenced at http://opensmile.audeering.com/ . It contains a section on how to write configuration files. The next important element is the online help of the binary:
SMILExtract -L lists the available components
SMILExtract -H cComponentName lists all options which a given component supports (and thus also features it can extract) with a short description for each
SMILExtract -configDflt cComponentName gives you a template configuration section for the component with all options listed and defaults set
Due to the architecture of openSMILE, which is centered on incremental processing of all audio features, there is (at least not yet) no easy syntax to define the features you want. Rather, you define the processing chain by adding components:
data sources will read in data (from audio files, csv files, or microphone, for example),
data processors will do signal processing and feature extraction in individual steps (windowing, window function, FFT, magnitudes, mel-spectrum, cepstral coefficients (MFCC), for example for extracting MFCC); for each step there is a data processor.
data sinks will write data to output files or send results to a server etc.
You connect the components via the "reader.dmLevel" and "writer.dmLevel" options. These define a name of a data memory level that the components use to exchange data. Only one component may write to one level, i.e. writer.dmLevel=levelName defines the level and may appear only once. Multiple components can read from this level by setting reader.dmLevel=levelName.
In each component you then set the options to enable computation of features and set parameters for this. To answer your question about lspFreq: This is probably enabled by default in the cLsp component, so you don't see an explicit option for it. For future versions of openSMILE the practice of setting all options explicitly will and should be followed more tightly.
The names of the features in the output will be automatically defined by the components. Often each component adds a part the the name, so you can infer from the name the full chain of processing. The options nameAppend and copyInputName (available to most data processors) control this behaviour, although some components might internally override them or change the behaviour a bit.
To see the names (and other info) for each data memory level, including e.g. which features a component in the configuration produces, you can set the option "printLevelStats=5" in the section of componentInstances:cComponentManager.
As everyhting in openSMILE is built for real-time incremental processing, each data memory level has a buffer, which by default is a ring buffer to keep memory footprint constant when the application runs for a longer time.
Sometimes you might want to summarise features over a window of a given length (e.g. with the cFunctionals component). In this case you must ensure that the buffer size of the input level to this component is large enough to hold the full window. You do this via the following options:
writer.levelconf.isRb = 1/0 : sets type of buffer to ringbuffer (1) or fixed size buffer
writer.levelconf.growDyn = 1/0 : sets the buffer to dynamically grow if more data is written to it (1)
writer.levelconf.nT = sets the size of the buffer in frames. Alternatively you can use bufferSizeSec=x to set the size size in seconds and convert to frames automatically.
In most cases the sizes will be set correctly automatically. Subsequent levels also inherit the configuration from the previous levels. Exceptions are when you set a cFunctionals component to read the full input (e.g. only produce one feature at the end of the file), the you must use growDyn=1 on the level that the functionals component reads from, or if you use a variable framing mode (see below).
The cFunctionals component provides frameMode, frameSize, and frameStep options. Where frameMode can be full* (one vector produced at end of input/file), **list (specify a list of frames), var (receive messages, e.g. from a cTurnDetector component, that define frames on-the-fly), or fix (fixed length window). Only in the case of fix the options frameSize set the size of this window, and frameStep the rate at which the window is shifted forward. In case of fix the buffer size of the input level is set correctly automatically, in the other cases you have to set it manually.
I hope this helps you to get started! With every new openSMILE release we at audEERING are trying to document things a bit better and unify things through various components.
We also welcome contributions from the community (e.g. anybody willing to write a graphical configuration file editor where you drag/drop components and connect them graphically? ;)) - although we know that more documentation will make this easier. Until then, you always have to source code to read ;)
Cheers,
Florian
I'm working on an instant messaging app, where users can receive files from their friends.
The names of the files received are set by the sender of the file, and multiple files can be sent together with the possibility of subdirectories. For example, two files sent together might be "1" and "sub/2" such that the downloaded results should be like "downloads/1" and "downloads/sub/2".
I'm worried about the security implications of this. Right off the top of my head, two potentially dangerous filnames would be something like "../../../somethingNasty" or "~/somethingNasty" for Unix-like users. Other potential issues that cross my mind are filenames with characters that are unsupported on the target filesystem, but that seems much harder and may just be better to ignore?
I'm considering stripping received filenames for ".." and "~" but this type of blacklist approach where I individually think of problem cases hardly seems like the recipe for good security. What's the recommended way to sanitize filenames to ensure nothing sinister happens?
If it makes a difference, my app is running on C++ with the QT framework.
It's wiser to replace ".." with say XXX and ~ with say YYY. This way you convert any invalid path to a perfectly valid path. I.e. if the user wants to upload "../../../somethingNasty" - no problems, let him upload the file and store it in XXX/XXX/XXX/somethingNasty.
Or even better, you can encode all nonalphanumeric characters (except slashes) with %XY where XY is hexidecimal code of the character. This way you would have %2E%2E/%2E%2E/%2E%2E/SomethingNasty