I have a config file. It has variables stored in the following manner.
[general]
webapp=/var/www
data=/home/data
[env]
WEBAPP_DEPLOY=${general:webapp}/storage/deploy
SYSTEM_DEPLOY=${general:data}/deploy
As you can see it has 2 sections general and env. Section env uses the variables from section general.
So I want to read this file into a variable. Let's say config. Here's I want config object to look like:
{
general: {
webapp: '/var/www',
data: '/home/data'
},
env: {
WEBAPP_DEPLOY: '/var/www/storage/deploy',
SYSTEM_DEPLOY: '/home/data/deploy'
}
}
I general I am looking for a config parser for nodejs that supports string interpolation.
I would assume most ini libraries don't include the variable expansion functionality, but with lodash primitives a generic "deep object replacer" isn't too complex.
I've switched the : delimiter for . so has and get can lookup values directly.
const { get, has, isPlainObject, reduce } = require('lodash')
// Match all tokens like `${a.b}` and capture the variable path inside the parens
const re_token = /\${([\w$][\w\.$]*?)}/g
// If a string includes a token and the token exists in the object, replace it
function tokenReplace(value, key, object){
if (!value || !value.replace) return value
return value.replace(re_token, (match_string, token_path) => {
if (has(object, token_path)) return get(object, token_path)
return match_string
})
}
// Deep clone any plain objects and strings, replacing tokens
function plainObjectReplacer(node, object = node){
return reduce(node, (result, value, key) => {
result[key] = (isPlainObject(value))
? plainObjectReplacer(value, object)
: tokenReplace(value, key, object)
return result
}, {})
}
> plainObjectReplacer({ a: { b: { c: 1 }}, d: 'wat', e: '${d}${a.b.c}' })
{ a: { b: { c: 1 } }, d: 'wat', e: 'wat1' }
You'll find most config management tools (like ansible) can do this sort of variable expansion for you before app runtime, at deployment.
Related
I'm trying to create a config file for my java app that can have lazy evaluation, for instance, contain an e-mail template with current date time. I use ConfigSlurper to parse the config file.
For some reason I can't find at https://groovy-lang.org/closures.html the description of the commonly used form (see build.gradle): prop1 { ... } prop2 { ... }. Am I looking at the wrong place?
My tests show that such closures are evaluated immediately and the example above can be rewritten as: prop1 = { ... }() prop2 = { ... }() (notice the parentheses, without them the evaluation is lazy).
I also don't understand why using dots in property without the equal sign cause MissingMethodException: No signature of method: groovy.util.ConfigObject.URL() if I use: server.URL { System.out.println('ccc'); 'asd' } in the example below.
It's disappointing that the choice is completely transparent for Groovy, but you have to check that a value is instance of Closure or GString when using java.
I used this article as the starting point https://blog.mrhaki.com/2009/08/grassroots-groovy-configuration-with.html
// We can group settings.
// The following settings are created:
// app.version, app.setting1, app.setting2, app.setting3, app.date, app.active
app {
System.out.println('aaa')
version = "1.0"
// We can write code to set up settings.
[1, 2, 3].each {
this."setting${it}" = it * 10
}
// We can use Java objects
date = new Date()
active = true
}
server.URL = "http://default"
// Environment specific settings override settings with the
// same name.
environments {
development {
server = { System.out.println('ccc'); 'asd' }
// server.URL = "${-> System.out.println('bbb');new Date()}http://localhost"
}
test {
server.URL = 'http://test:9080'
}
integrationtest {
server.URL = 'http://integrationtest/url'
}
production {
server.URL = 'http://prod/url'
}
}
My tests show that such closures are evaluated immediately and the
example above can be rewritten as: prop1 = { ... }()
Closures are not evaluated immediately. The parens above are causing the closure to be evaluated.
For some reason I can't find at https://groovy-lang.org/closures.html
the description of the commonly used form (see build.gradle): prop1 {
... } prop2 { ... }.
prop1 {} is invoking a method named prop1 and passing a closure as a parameter That code is equivalent to this:
def myClosure = {}
prop1(myClosure)
I also don't understand why using dots in property without the equal
sign cause MissingMethodException: No signature of method:
groovy.util.ConfigObject.URL()
Something like server.URL = 'http://test:9080' is assigning a value (http://test:9080) to a property (server.URL). If you remove the equal sign you are changing the expression fundamentally and then would be invoking a method named URL on an object named server and passing http://test:9080 as a parameter.
The plugin gulp-pug allows to pass global variables to pug files via data property.
What if we don't need full data set in each .pug file? To implement conditional data injection, we need to access to current vinyl file instance inside pipe(this.gulpPlugins.pug({}) or at least to know the source file absolute path. Possible?
const dataSetForTopPage = {
foo: "alpha",
bar: "bravo"
};
const dataSetForAboutPage = {
baz: "charlie",
hoge: "delta"
};
gulp.src(sourceFileGlobsOrAbsolutePath)
.pipe(gulpPlugins.pug({
data: /*
if path is 'top.pug' -> 'dataSetForTopPage',
else if path is 'about.pug' -> 'dataSetForAboutPage'
else -> empty object*/
}))
.pipe(Gulp.dest("output"));
I am using gulp-intercept plugin. But how to synchronize it with gulpPlugins.pug?
gulp.src(sourceFileGlobsOrAbsolutePath)
.pipe(this.gulpPlugins.intercept(vinylFile => {
// I can compute conditional data set here
// but how to execute gulpPlugins.pug() here?
}))
// ...
It was just one example, but we will deal with same problem when need to conditional plugins options for other gulp plugins, too. E. g:
.pipe(gulpPlugins.htmlPrettify({
indent_char: " ",
indent_size: // if source file in 'admin/**' -> 2, else if in 'auth/**' -> 3 else 4
}))
You'll need to modify the stream manually - through2 is probably the most used package for this purpose. Once in the through2 callback, you can pass the stream to your gulp plugins (as long as their transform functions are exposed) and conditionally pass them options. For example, here is a task:
pugtest = () => {
const dataSet = {
'top.pug': {
foo: "alpha",
bar: "bravo"
},
'about.pug': {
foo: "charlie",
bar: "delta"
}
};
return gulp.src('src/**/*.pug')
.pipe(through2.obj((file, enc, next) =>
gulpPlugins.pug({
// Grab the filename, and set pug data to the value found in dataSet by that name
data: dataSet[file.basename] || {}
})._transform(file, enc, next)
))
.pipe(through2.obj((file, enc, next) => {
const options = {
indent_char: ' ',
indent_size: 4
};
if(file.relative.match(/admin\//)) {
options.indent_size = 2;
} else if(file.relative.match(/auth\//)) {
options.indent_size = 3;
}
file.contents = new Buffer.from(html.prettyPrint(String(file.contents), options), enc);
next(null, file);
}))
.pipe(gulp.dest('output'));
}
For the pug step, we call through2.obj and create the pug plugin, passing it data grabbed from our object literal, indexed by filename in this example. So now the data passed into the compiler comes from that object literal.
For the html step you mention, gulp-html-prettify doesn't expose its transform function, so we can't reach into it and pass the transform back to the stream. But in this case that's OK, if you look at the source it's just a wrapper to prettyPrint in the html package. That's quite literally all it is doing. So we can just rig up our step using through2 to do the same thing, but changing our options based on the vinyl file's relative path.
That's it! For a working example see this repo: https://github.com/joshdavenport/stack-overflow-61314141-gulp-pug-conditional
Using grpc with Node, Enums in responses to my queries are resolving as integer values. However, when I make the same queries with BloomRPC, the Enums resolve as Integer values.
Is there a parameter or option to force these Enums to be resolved as String using Node grpc?
In our project, we use enum to help us ensure the integrity of a finite set of possibilities by eliminating human error. Why should we need to remember what the string value is when we have the protocol buffer enum so handy? Thus, we use the .proto as the source of truth; that's our rule.
To do that, follow these steps, which are written for ES6+ code.
Define your gRPC/Protobuf enum in a .proto file.
// life.proto
syntax = 'proto3';
package life;
enum Choices
{
EAT = 0;
DRINK = 1;
SLEEP = 2;
CODE = 3;
SKI = 4;
}
Install #grpc/proto-loader and #grpc/grpc-js.
$ npm i -s #grpc/proto-loader #grpc/grpc-js
Import the modules that pay the bills, so to speak. Load the .proto file directly into memory (don't compile).
// myNodeApp.js
import * as grpc from '#grpc/grpc-js'
import * as protoLoader from '#grpc/proto-loader'
import path from 'path'
// these options help make definitions usable in our code
const protoOptions = {
keepCase: true,
longs: String,
enums: String,
defaults: true,
oneofs: true
}
// this allows us to prepare the path to the current dir
const dir = path.dirname(new URL(import.meta.url).pathname)
// this loads the .proto file
const lifeDef = protoLoader.loadSync(path.join(dir, 'life.proto'), protoOptions)
// this loads the package 'life' from the .proto file
const life = grpc.loadPackageDefinition(lifeDef).life
Take a peek at the enum Choices definition (in the same file).
// myNodeApp.js (cont'd)
console.log(life.Choices)
/* stdout */
{
format: 'Protocol Buffer 3 EnumDescriptorProto',
type: {
value: [ [Object], [Object], [Object], [Object], [Object] ],
name: 'Choices',
options: null
},
fileDescriptorProtos: [
<Buffer 0a ... 328 more bytes>
]
}
...look deeper...
console.log(life.Choices.value)
/* stdout */
{
value: [
{ name: 'EAT', number: 0, options: null },
{ name: 'DRINK', number: 1, options: null },
{ name: 'SLEEP', number: 2, options: null },
{ name: 'CODE', number: 3, options: null },
{ name: 'SKI', number: 4, options: null }
],
name: 'Choices',
options: null
}
Use the enum.
// myNodeApp.js
const myDay = { // plain JSON (or define a gRPC message, same same)
dawn: life.Choices.type.value[1].name,
morning: life.Choices.type.value[0].name,
afternoon: life.Choices.type.value[4].name,
evening: life.Choices.type.value[3].name,
night: life.Choices.type.value[2].name
}
You could write an accessor or utility function to manage the key lookup (by passing the imported grpc enum and index), like so:
export const getEnumByName = function (protoEnum, needle) {
return protoEnum.type.value.find(p => {
return p.name === needle
})
}
export const getEnumByNum = function (protoEnum, needle) {
return protoEnum.type.value.filter(p => {
return p.number = needle
})
}
export const getEnumKeys = function (protoEnum, key = 'name') {
return protoEnum.type.value.map(p => {
return p[key]
})
}
Inverting and assigning a value to a Message is what's already covered in other answers, just set the enum field to the string value using, you guessed it, the string that represents the enum name, which you access using the code above.
This is along the lines of how we do it. Clean and simple, just a touch obscure until you look "under the hood" one day.
Learn more about #grpc/proto-loader and #grpc/grpc-js. Hope this helps someone out there in the wild. :)
If you are using the #grpc/proto-loader library, you can set the option enums to the value String (not the string "String", the constructor function String). Then all enum values will be represented by their name strings.
I know this is a general question but I have exhausted google and tried many approaches.Any feedback is appreciated.
The HTTPClient is Angular 5+ so it returns an object created from the response JSON data. I get a massive JSON response from an endpoint I have no control over and I want to use about 20% of the response in my app and ignore the rest.
I am really trying hard to avoid using a series of templates or export objects or whatever and trying to force this massive untyped Observable into a typed object with hundreds of fields many being Arrays. All I need for the app is just a Array of very small objects with 3 fields per object. The 3 fields are all over within the JSON response and I want to map them to my object .map only seems to work when you are using the full response object and I can't find an example where .map does custom work besides in the case where you are mapping a few fields to 1 object and I am trying to map to an Array of my small objects.
UPDATED
Basically I want this service to return an object of Type DislayData to the module that subscribes to it but I get just an Object back. This is not what I ultimately need to do but if I can prove I can map the body of the response to my needed return type I can then start to break down the response body and return an Array of the Type I really need based on my silly DisplayData object. Thanks again!
export interface DislayData {
body: any;
}
...
export class DataService {
constructor(private http: HttpClient) { }
/** GET data from the black box */
getData(): Observable<DislayData> {
return this.http.get<HttpResponse<any>>(searchUrl, { observe: 'response' })
.pipe(
map(res => {
return res.body as DislayData;
}
tap(res => console.log(//do stuff with entire respoonse also)),
catchError(err => this.handleError(err)));
}
private handleError(error: HttpErrorResponse) {
...
Do you know the structure of the answering object?
If yes, you can do something like this:
item$ = new BehaviorSubject<any>({});
item = {
foo: 'a',
bar: 'b',
iton: [1, 2, 3],
boo: {
far: 'c'
}
};
logNewItem() {
this.item$
.pipe(
map(response => {
if (response.foo
&& response.iton
&& response.iton.length >= 3
&& response.boo
&& response.boo.far) {
let newItem = {
foo: response.foo,
iton2: response.iton[2],
far: response.boo.far
};
console.log(newItem); // output: Object { foo: "a", iton2: 3, far: "c" }
}
})
)
.subscribe();
this.item$.next(this.item);
}
Basically, you can simply make sure the properties exist, call them directly and map them to a better fitting object.
I heavily recommend creating an interface for the object you're receiving and an interface or class for the object you're mapping to. In that case you can also write the code more compact like this:
[...]
map(response: MyAPIResponse => {
let newItem = new NewItem(response);
console.log(newItem); // output: Object { foo: "a", iton2: 3, far: "c" }
}
})
[...]
class NewItem {
foo: string;
iton2: string;
far: string;
constructor(apiResponse: MyAPIResponse) {
//Validate parameter first
this.foo = apiResponse.foo;
this.iton2 = apiResponse.iton[2];
this.far = apiResponse.boo.far;
and make your code a lot more readable.
I'm trying to select certain keys from an JSON array, and filter the rest.
var json = JSON.stringify(body);
which is:
{
"FirstName":"foo",
"typeform_form_submits":{
"foo":true,
"bar":true,
"baz":true
},
"more keys": "foo",
"unwanted key": "foo"
}
Want I want:
{
"FirstName":"foo",
"typeform_form_submits":{
"foo":true,
"bar":true,
"baz":true
}
}
I've checked out How to filter JSON data in node.js?, but I'm looking to do this without any packages.
Now you can use Object.fromEntries like so:
Object.fromEntries(Object.entries(raw).filter(([key]) => wantedKeys.includes(key)))
You need to filter your obj before passing it to json stringify:
const rawJson = {
"FirstName":"foo",
"typeform_form_submits":{
"foo":true,
"bar":true,
"baz":true
},
"more keys": "foo",
"unwanted key": "foo"
};
// This array will serve as a whitelist to select keys you want to keep in rawJson
const filterArray = [
"FirstName",
"typeform_form_submits",
];
// this function filters source keys (one level deep) according to whitelist
function filterObj(source, whiteList) {
const res = {};
// iterate over each keys of source
Object.keys(source).forEach((key) => {
// if whiteList contains the current key, add this key to res
if (whiteList.indexOf(key) !== -1) {
res[key] = source[key];
}
});
return res;
}
// outputs the desired result
console.log(JSON.stringify(filterObj(rawJson, filterArray)));
var raw = {
"FirstName":"foo",
"typeform_form_submits":{
"foo":true,
"bar":true,
"baz":true
},
"more keys": "foo",
"unwanted key": "foo"
}
var wantedKeys =["FirstName","typeform_form_submits" ]
var opObj = {}
Object.keys(raw).forEach( key => {
if(wantedKeys.includes(key)){
opObj[key] = raw[key]
}
})
console.log(JSON.stringify(opObj))
I know this question was asked aways back, but I wanted to just toss out there, since nobody else did:
If you're bound and determined to do this with stringify, one of its less-well-known capabilities involves replacer, it's second parameter. For example:
// Creating a demo data set
let dataToReduce = {a:1, b:2, c:3, d:4, e:5};
console.log('Demo data:', dataToReduce);
// Providing an array to reduce the results down to only those specified.
let reducedData = JSON.stringify(dataToReduce, ['a','c','e']);
console.log('Using [reducer] as an array of IDs:', reducedData);
// Running a function against the key/value pairs to reduce the results down to those desired.
let processedData = JSON.stringify(dataToReduce, (key, value) => (value%2 === 0) ? undefined: value);
console.log('Using [reducer] as an operation on the values:', processedData);
// And, of course, restoring them back to their original object format:
console.log('Restoration of the results:', '\nreducedData:', JSON.parse(reducedData), '\nprocessedData:', JSON.parse(processedData));
In the above code snippet, the key value pairs are filtered using stringify exclusively:
In the first case, by providing an array of strings, representing the keys you wish to preserve (as you were requesting)
In the second, by running a function against the values, and dynamically determining those to keep (which you didn't request, but is part of the same property, and may help someone else)
In the third, their respective conversions back to JSON (using .parse()).
Now, I want to stress that I'm not advocating this as the appropriate method to reduce an object (though it will make a clean SHALLOW copy of said object, and is actually surprisingly performant), if only from an obscurity/readability standpoint, but it IS a totally-effective (and mainstream; that is: it's built into the language, not a hack) option/tool to add to the arsenal.