I am facing trouble where datadog RESOURCE column is not giving the correct value as shown in the image. Really need some help here.
My assumption is that, it is happening because http tags are not appearing correctly. I think datadog itself add the http tags and it's value.
The http.path_group & http.route should have this value "/api-pim2/v1/attribute/search" but for some reason it's not coming correctly.
I am using this library dd-trace at backend. The tracers options which i provided are these
{"logInjection":true,"logLevel":"debug","runtimeMetrics":true,"analytics":true,"debug":true,"startupLogs":true,"tags":{"env":"dev02","region":"us-east-1","service":"fabric-gateway-pim-ecs"}}
The initialising code looks like this which ran at the start of my app
app/lib/tracer.js:
const config = require('config')
const tracerOptions = config.get('dataDog.tracer.options')
const logger = require('app/lib/logger')
const tracer = require('dd-trace').init({
...tracerOptions,
enabled: true,
logger
})
module.exports = tracer
I also tried to set the http.path_group & http.route tag manually but still it's not updating the values. Though i can add the new tags like http.test which has the same value which i was trying to override in http.path_group & http.route
const addTagsToRootSpan = tags => {
const span = tracer.scope().active()
if (span) {
const root = span.context()._trace.started[0]
for (const tag of tags) {
root.setTag(tag.key, tag.value)
}
log.debug('Tags added')
} else {
log.debug('Trace span could not be found')
}
}
...
const tags = [
{ key: 'http.path_group', value: request.originalUrl },
{ key: 'http.route', value: request.originalUrl },
{ key: 'http.test', value: request.originalUrl }
addTagsToRootSpan(tags)
...
I was requiring tracer.js file at the start of my app where server was listening.
require('app/lib/tracer')
app.listen(port, err => {
if (err) {
log.error(err)
return err
}
log.info(`Your server is ready for ${config.get('stage')} on PORT ${port}!`)
})
By enabling the debug option in datadog tracer init function. I can see the tracer logs and what values are passed by the library for http.route and resource.
I was confused by this line according to the data dog tracers doc you should init first before importing any instrumented module.
// This line must come before importing any instrumented module.
const tracer = require('dd-trace').init();
But for me http.route & resource value get correct if i initialise it on my routing file. They start giving me complete route "/api-pim2/v1/attribute/search" instead of only "/api-pim2"
routes/index.js:
const router = require('express').Router()
require('app/lib/tracer')
const attributeRouter = require('app/routes/v1/attribute')
router.use('/v1/attribute', attributeRouter)
module.exports = router
I am not accepting this answer yet because i am still confused where to initialise the tracer. Maybe someone can explain better. I am just writing this answer if someone else facing the issue can try this. Might just resolve their problem.
Related
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 months ago.
Improve this question
I was wondering if any of the following is possible to implement using vite build tool.
Consider that I have files in directory matching the pattern: /content/file-[id].md
/content/file-1.md
/content/file-2.md
Every time I serve the SPA app with vite command or building an app with vite build I would like to
grab all the files /content/file-[id].md and transform them into /content_parsed/file-[id].html
/content_parsed/file-1.html
/content_parsed/file-2.html
grab all files /content_parsed/file-[id].html and generated a manifest file /files.manifest containing all paths of files.
/files.manifest
This has to be done automatically in watch mode, when the app is served (vite command) and on-demand when app is built (vite build).
I am pretty sure this is possible to be done with a manual script that I could run with node ./prepareFiles.js && vite, but in this case I am loosing the reactivity when serving the app (i.e. the watch-mode).. so a direct integration into vite would be a step-up in terms of usability and testability (I think).
Given the above use-case - can vite do this? Do I need to write a custom plugin for that? or do you recommend creating a separate watch-files/watch-directory script for that?
I have been able to partially accomplish what I wanted. The only issue right now is the hot reload functionality.
if you import the manifest as
import doc from 'docs.json'
then the page will be auto-reloaded if the module is updated.
On the other had, if you want to dynamically load the data with fetch API:
fetch('docs.json')
.then(r => r.json())
.then(json => {
//...
})
Then the only way to refresh page contents is by manual refresh.. If anyone has a suggestion how to trigger reload from within vite plugin context please let me know.. I will update the post once I figure it out.
Also I should mention that I have decided not to pre-generate the html pages so this functionality is missing from the plugin but could easily be extended with marked, markdown-it remarked etc..
Plugin: generateFilesManifest.ts
import {PluginOption} from "vite";
import fs from "fs";
import path from 'path'
const matter = require('front-matter');
const chokidar = require('chokidar');
import {FSWatcher} from "chokidar";
export type GenerateFilesManifestConfigType = {
watchDirectory: string,
output: string
}
export type MatterOutputType<T> = {
attributes: T,
body: string,
bodyBegin: number,
frontmatter: string,
path: string,
filename: string,
filenameNoExt: string,
}
export default function generateFilesManifest(userConfig: GenerateFilesManifestConfigType): PluginOption {
let config: GenerateFilesManifestConfigType = userConfig
let rootDir: string
let publicDir: string
let command: string
function generateManifest() {
const watchDirFullPath = path.join(rootDir, config.watchDirectory)
const files = fs.readdirSync(watchDirFullPath);
// regenerate manifest
const manifest: any[] = []
files.forEach(fileName => {
const fileFullPath = path.join(watchDirFullPath, fileName)
// get front matter data
const fileContents = fs.readFileSync(fileFullPath).toString()
//const frontMatter = matter.read(fileFullPath)
const frontMatter = matter(fileContents)
//console.log(frontMatter);
// get file path relative to public directory
//const basename = path.basename(__dirname)
const fileRelativePath = path.relative(publicDir, fileFullPath);
const fileInfo = JSON.parse(JSON.stringify(frontMatter)) as MatterOutputType<any>;
fileInfo.path = fileRelativePath
fileInfo.filename = fileName
fileInfo.filenameNoExt = fileName.substring(0, fileName.lastIndexOf('.'));
fileInfo.frontmatter = ''
manifest.push(fileInfo);
});
const outputString = JSON.stringify(manifest, null, 2);
fs.writeFileSync(config.output, outputString, {encoding: 'utf8', flag: 'w'})
console.log('Auto-generated file updated')
}
let watcher: FSWatcher | undefined = undefined;
return {
name: 'generate-files-manifest',
configResolved(resolvedConfig) {
publicDir = resolvedConfig.publicDir
rootDir = resolvedConfig.root
command = resolvedConfig.command
},
buildStart(options: NormalizedInputOptions) {
generateManifest();
if (command === 'serve') {
const watchDirFullPath = path.join(rootDir, config.watchDirectory)
watcher = chokidar.watch(watchDirFullPath,
{
ignoreInitial: true
}
);
watcher
.on('add', function (path) {
//console.log('File', path, 'has been added');
generateManifest();
})
.on('change', function (path) {
//console.log('File', path, 'has been changed');
generateManifest();
})
.on('unlink', function (path) {
//console.log('File', path, 'has been removed');
generateManifest();
})
.on('error', function (error) {
console.error('Error happened', error);
})
}
},
buildEnd(err?: Error) {
console.log('build end')
watcher?.close();
}
}
}
in vite.config.ts, use as
export default defineConfig({
plugins: [
vue(),
generateFilesManifest({
watchDirectory: '/public/docs',
output: './public/docs.json'
})
]
})
you might want to cover such as edge-cases as watch directory not present etc...
front-matter is the library that parses markdown files. Alternative is gray-matter
EDIT: thanks to #flydev response I was able to dig some more examples on page reload functionality. Here's the experimental functionality that you could add:
function generateManifest() {
// ...
ws?.send({type: 'full-reload', path: '*'})
}
let ws: WebSocketServer | undefined = undefined;
return {
name: 'generate-files-manifest',
//...
configureServer(server: ViteDevServer) {
ws = server.ws
}
// ...
}
Currently the whole page is reloaded regardless of the path.. Not sure if there is a way to make it smart enough to just reload pages that loaded the manifest file.. I guess it's currently limited by my own ability to write a better code :)
I'm facing a problem I'm not able to resolve on my own, maybe some of you faced the same problem.
Let me show you what I'm trying to do, here is the mock:
let mockConfig = {name: 'dude'};
jest.mock('../../../configManager', () => mockConfig);
configManager is a dependency of the function I'm trying to test.
It works well but I want to change the returning object of configManager in another test so the tested function behaves differently.
Let me show you, here is the function I'm testing:
const config = require('../../../configManager');
module.exports = () => {
if (config.name === 'dude') {
do stuff;
}
if (config.name === 'dudette') {
do something else;
}
So, typically, I want to change the config.name to 'dudette' to be able to test the second part of my function.
Naturally, when I want to do this with an imported function, I just do:
let mockJsonQueryResult = { value: 'stuff' };
jest.mock('json-query', () => jest.fn(() => mockJsonQueryResult));
and then in the test, I directly set another value to mockJsonQueryResult:
mockJsonQueryResult = { value: 'hotterStuff' };
But I don't find any way of doing this with a dependency that returns an object, with a dependency returning a function, no problem.
Is there even any way of doing this?
Thanks in advance!
Edit: this is not the same as how to change jest mock function return value in each test? as #Dor Shinar suggested because his problem is to mock a function, even if it is inside a returning object it is still a function, I just want to change a value inside the returned object.
So, I found a solution I'm not completely satisfied with but it works:
I simply set the original full object and then for my tests, change the value of specific properties by setting them directly before calling the function I want to test.
example:
let mockConfig = { person: { name: 'dude', origin: {country: 'France'} } };
jest.mock('../../../configManager', () => mockConfig);
mockConfig.person = {};
mockConfig.person.name = 'dudette';
You don't need to mock the module at all.
If your module export is just an object with property values then just change the properties as needed.
Here is a simple working example to demonstrate:
configManager.js
module.exports = {
name: 'original'
}
code.js
const config = require('./configManager');
module.exports = () => `name: ${config.name}`;
code.test.js
const config = require('./configManager');
const func = require('./code');
test('func', () => {
expect(func()).toBe('name: original'); // Success!
config.name = 'dude';
expect(func()).toBe('name: dude'); // Success!
config.name = 'dudette';
expect(func()).toBe('name: dudette'); // Success!
})
Details
A module binding can't be directly changed to something else:
const config = require('./configManager');
config = { name: 'mock' }; // <= this doesn't work
...but you can change the properties of an object representing a module binding:
const config = require('./configManager');
config.name = 'mock'; // <= this works!
...and any code using the module will automatically see the changes.
I apologise for the phrasing of the question - it's a bit difficult to sum up as a question - please feel free to edit it if you can clarify. Also, as this quite a complex and long query - thank you to all those who are putting in the time to read through it!
I have 4 files (listed with directory tree from project root) as part of a project I'm building which aims to scrape blockchains and take advantage of multiple cores do get the job done:
./main.js
./scraper.js
./api/api.js
./api/litecoin_api.js
main.js
const { scraper } = require('./scraper.js')
const blockchainCli = process.env.BLOCKSCRAPECLI || 'litecoin-cli'
const client = (args) => {
// create child process which returns a promise which resolves after
// data has finished buffering from locally hosted node using cli
let child = spawn(`${blockchainCli} ${args.join(' ')}`, {
shell: true
})
// ... wrap command in a promise here, etc
}
const main = () => {
// count cores, spawn a worker per core using node cluster, add
// message handlers, then begin scraping blockchain with each core...
scraper(blockHeight)
}
main()
module.exports = {
client,
blockchainCli
}
scraper.js
const api = require('./api/api.js')
const scraper = async (blockHeight) => {
try {
let blockHash = await api.getBlockHashByHeight(blockHeight)
let block = await api.getBlock(blockHash)
// ... etc, scraper tested and working, writes to shared writeStream
}
module.exports = {
scraper
}
api.js
const { client, blockchainCli } = require('../main.js')
const litecoin = require('./litecoin_api')
let blockchain = undefined
if (blockchainCli === 'litecoin-cli' || blockchainCli === 'bitcoin-cli') {
blockchain = litecoin
}
// PROBLEM HERE: blockchainCli (and client) are both undefined if and
// only if running scraper from main.js (but not if running scraper
// from scraper.js)
const decodeRawTransaction = (txHash) => {
return client([blockchain.decodeRawTransaction, txHash])
}
const getBlock = (blockhash) => {
return client([blockchain.getBlock, blockhash])
}
const getBlockHashByHeight = (height) => {
return client([blockchain.getBlockHash, height])
}
const getInfo = () => {
return client([blockchain.getInfo])
}
const getRawTransaction = (txHash, verbose = true) => {
return client([blockchain.getRawTransaction, txHash, verbose])
}
module.exports = {
decodeRawTransaction,
getBlock,
getBlockHashByHeight,
getInfo,
getRawTransaction
}
So, I've taken out most the noise in the files which I don't think is necessary but it's open source so if you need more take a look here.
The problem is that, if I start the scraper from inside scraper.js by doing, say, something like this: scraper(1234567) it works like a charm and outputs the expected data to a csv file.
However if I start the scraper from inside the main.js file, I get this error:
Cannot read property 'getBlockHash' of undefined
at Object.getBlockHashByHeight (/home/grayedfox/github/blockscrape/api/api.js:19:29)
at scraper (/home/grayedfox/github/blockscrape/scraper.js:53:31)
at Worker.messageHandler (/home/grayedfox/github/blockscrape/main.js:81:5)
I don't know why, when launching the scraper from main.js, the blockchain is undefined. I thought it might be from the destructuring, but removing the curly braces from around the first line in the example main.js file doesn't change anything (same error).
Things are a bit messy at the moment (in the middle of developing this branch) - but the essential problem now is that it's not clear to me why the require would fail (cannot see variables inside main.js) if it's used in the following way:
main.js (execute scraper()) > scraper.js > api.js
But not fail (can see variables inside main.js) if it's run like this:
scraper.js (execute scraper()) > api.js
Thank you very much for your time!
You have a circular dependency between main and api, each requiring in the other. main requires api through scraper and api directly requires main. That causes things not to work.
You have to remove the circular dependency by putting common shared code into its own module that can be included by both, but doesn't include others that include it. It just needs better modularity.
I'd like to override some values at test-time, specifically setting my retries for an http service to 1 (immediate failure, no retries). Our project uses node-config. According to the docs I can override with NODE_CONFIG env variable:
node myapp.js --NODE_CONFIG='{"Customer":{"dbConfig":{"host":"customerdb.prod"}}}'
Well I would prefer to do this in my test, but not for all tests. The code says that you can allow config mutations by setting ALLOW_CONFIG_MUTATIONS.
process.env.ALLOW_CONFIG_MUTATIONS = "true";
const importFresh = require('import-fresh');
importFresh("config");
process.env.NODE_CONFIG = JSON.stringify({httpServices:{integration:{enrich: {retryInterval: 1, retries: 1}}}});
expect(process.env.NODE_CONFIG, 'NODE_CONFIG not set').to.exist();
expect(process.env.NODE_CONFIG, 'NODE_CONFIG not set').to.match(/retryInterval/);
expect(process.env.ALLOW_CONFIG_MUTATIONS, 'ALLOW_CONFIG_MUTATIONS not set').to.equal("true");
const testConfig = require("config");
console.dir(testConfig.get("httpServices.integration.enrich"));
expect(testConfig.get("httpServices.integration.enrich.retryInterval"), 'config value not set to 1').to.equal(1);
Result:
{ url: 'https://internal-**********',
retryInterval: 5000,
retries: 5 }
`Error: config value not set to 1: Expected 5000 to equal specified value: 1`
How do I get this override to work?
(expect is from Hapi.js Code library)
I'm one of the maintainers of node-config. Your bug is that you used require the second time when you should have used importFresh again.
Your first use of "importFresh()" does nothing different than require() would, because it is the first use of require().
After setting some variables, you call require(), which will return the copy of config already generated and cached, ignoring the effects of the environment variables set.
You only needed to use importFresh() once, where you currently use require(). This will cause a "fresh" copy of the config object to be returned, as you expected.
Simply changing config's property worked for me.
For example:
const config = require( 'config' );
config.httpServices.integration.enrich.retryInterval = 1;
// Do your tests...
UPD: Make sure that overrides are done before anyone calls the first config.get(), because the config object is made immutable as soon as any client uses the values via get().
Joining late, but other answers did not fit with the testing standard in my project, so here is what I came up with
TL;DR
Use mocks..
Detailed Answer
node-config uses a function get to get the configuration values.
By mocking the function get you can easily modify any configuration you see fit..
My personal favorite library is sinon
Here is an implementation of a mock with sinon
const config = require('config');
const sinon = require('sinon');
class MockConfig {
constructor () {
this.params = {};
this.sandbox = sinon.sandbox.create();
}
withConfValue (confKey, confValue) {
this.params.confValues[confKey] = confValue;
return this;
}
reset () {
this.params.confValues: {};
return this;
}
restore() {
this.sandbox.restore();
}
apply () {
this.restore(); // avoid duplicate wrapping
this.sandbox.stub(config, 'get').callsFake((configKey) => {
if (this.params.confValues.hasOwnProperty(configKey)) {
return this.params.confValues[configKey];
}
// not ideal.. however `wrappedMethod` approach did not work for me
// https://stackoverflow.com/a/57017971/1068746
return configKey
.split('.')
.reduce((result, item) => result[item], config)
});
}
}
const instance = new MockConfig();
MockConfig.instance = () => instance;
module.exports = MockConfig;
Usage would be
const mockConfig = require('./mock_config').instance();
...
beforeEach(function () {
mockConfig.reset().apply();
})
afterEach(function () {
mockConfig.reset().clear();
})
it('should do something') {
mockConfig.withConfValue('some_topic.some_field.property', someValue);
... rest of the test ...
}
Assumptions
The only assumption this approach makes is that you adhere to node-config way of reading the configuration (using the get function) and not bypass it by accessing fields directly.
It's better to create a development.json, production.json et test.json in your config folder node-config will use it your app configuration.
you just net to set your NODE_ENV to use the specific file.
Hope it helps :)
I'm trying to use NeDB as storage for my data in node-webkit application. I have the single collection named config.db:
var Datastore = require('nedb')
, path = require('path')
, db = new Datastore({ filename: path.join(require('nw.gui').App.dataPath, 'config.db') });
When user opens node-webkit application first time my config.db should have default data like:
{
color: "red",
font: 'bold'
...
}
Does NeDB have option for providing default data if there are no yet? Or What it the best way to save it if config.db is empty (in case if user opens node-webkit application first time)?
As far as I know NeDB does not have an option to create initial data.
I think the easiest way to achieve this is to simply query whether there is data. If counting documents returns 0, obviously the initial data have not yet been saved, so you should do this now.
If you include this check in the startup code of your application, it will automatically initialize the data on first run, and afterwards simply do nothing.
I came across this question while looking for a similar solution. I thought I'd share what I ended up with (this is a module):
var fs = require("fs");
module.exports = function (app) {
var customizationService = app.service("customization");
fs.readFile("./db/customization", "utf8", function (err, data) {
if (err) {
return console.log(err);
}
if (data) {
// Sweet, carry on
} else {
var customOptions = {
SiteTitle: "VendoMarket",
SiteTagline: "The freshest eCommerce platform around"
};
// Save data to the locations service
customizationService.create(customOptions);
}
});
};
And then in my app.js file:
//--------------------------------------
// Initialize
//--------------------------------------
var vendoInit = require("./src/init");
vendoInit(app);
(My app.js file is at the base of my project, src is a folder next to it)