Mustache does not render values, when deployed - node.js

Node version: v8.9.0;
Im using moustache to render some html from templates as follows:
static createHtml(title, variables, template) {
let header = fs.readFileSync(headerPath);
let content = fs.readFileSync(contentPath);
let footer = fs.readFileSync(footerPath);
var headerVars = {
title: title
};
console.log('createHtml: ', variables); // <- variables are as expected
try {
header = Mustache.render(header.toString(), headerVars);
content = Mustache.render(content.toString(), variables);
} catch (e) {
console.log('Moustache error:\n', e); // <- no error is thrown
}
const html = header + content + footer;
console.log('content', content); // <- content does not include filled variables when deployed
return html;
}
When I run it on my local machine everything works just perfectly, however when it is deployed the variables are not injected into the template. I had several assumptions and tried to figure out every possible way the environment is different then the local, but nothing worked so far. Its is deployed on EC2 to an AMI Linux instance.
Any idea, how could I figure out what's the difference? According to the console logs it must be something inside Mustache.render().

Seems that the deployed instance on AMI Linux did not really like the syncronous fs.readFileSync operation after one point (what was the point and why is still and open question), though the solution was very simple. Just refactor the code to async as follows:
static createHtml(title, variables, template) {
fs.readFile(headerPath, (err, header) => {
fs.readFile(contentPath, (err, content) => {
fs.readFile(footerPath, (err, footer) => {
var headerVars = { title: title };
header = Mustache.render(header.toString(), headerVars);
content = Mustache.render(content.toString(), variables);
const html = header + content + footer.toString();
return html;
});
});
});
}

Related

Dynamic Slash Command Options List via Database Query?

Background:
I am building a discord bot that operates as a Dungeons & Dragons DM of sorts. We want to store game data in a database and during the execution of certain commands, query data from said database for use in the game.
All of the connections between our Discord server, our VPS, and the VPS' backend are functional and we are now implementing slash commands since traditional ! commands are being removed from support in April.
We are running into problems making the slash commands though. We want to set them up to be as efficient as possible which means no hard-coded choices for options. We want to build those choice lists via data from the database.
The problem we are running into is that we can't figure out the proper way to implement the fetch to the database within the SlashCommandBuilder.
Here is what we currently have:
const {SlashCommandBuilder} = require('#discordjs/builders');
const fetch = require('node-fetch');
const {REST} = require('#discordjs/rest');
const test = require('../commonFunctions/test.js');
var options = async function getOptions(){
let x = await test.getClasses();
console.log(x);
return ['test','test2'];
}
module.exports = {
data: new SlashCommandBuilder()
.setName('get-test-data')
.setDescription('Return Class and Race data from database')
.addStringOption(option =>{
option.setName('class')
.setDescription('Select a class for your character')
.setRequired(true)
for(let op of options()){
//option.addChoice(op,op);
}
return option
}
),
async execute(interaction){
},
};
This code produces the following error when start the npm for our bot on our server:
options is not a function or its return value is not iterable
I thought that maybe the function wasn't properly defined, so I replaced the contents of it with just a simple array return and the npm started without errors and the values I had passed showed up in the server.
This leads me to think that the function call in the modules.exports block is immediatly attempting to get the return value of the function and as the function is async, it isn't yet ready and is either returning undefined or a promise or something else not iteratable.
Is there a proper way to implement the code as shown? Or is this way too complex for discord.js to handle?
Is there a proper way to implement the idea at all? Like creating a json object that contains the option data which is built and saved to a file at some point prior to this command being registered and then having the code above just pull in that file for the option choices?
Alright, I found a way. Ian Malcom would be proud (LMAO).
Here is what I had to do for those with a similar issues:
I had to basically re-write our entire application. It sucks, I know, but it works so who cares?
When you run your index file for your npm, make sure that you do the following things.
Note: you can structure this however you want, this is just how I set up my js files.
Setup a function that will setup the data you need, it needs to be an async function as does everything downstream from this point on relating to the creation and registration of the slash commands.
Create a js file to act as your application setup "module". "Module" because we're faking a real module by just using the module.exports method. No package.jsons needed.
In the setup file, you will need two requires. The first is a, as of yet, non-existent data manager file; we'll do that next. The second is a require for node:fs.
Create an async function in your setup file called setup and add it to your module.exports like so:
module.exports = { setup }
In your async setup function or in a function that it calls, make a call to the function in your still as of yet non-existent data manager file. Use await so that the application doesn't proceed until something is returned. Here is what mine looks like, note that I am writing my data to a file to read in later because of my use case, you may or may not have to do the same for yours:
async function setup(){
console.log('test');
//build option choice lists
let listsBuilt = await buildChoiceLists();
if (listsBuilt){
return true;
} else {
return false;
}
}
async function buildChoiceLists(){
let classListBuilt = await buildClassList();
return true;
}
async function buildClassList(){
let classData = await classDataManager.getClassData();
console.log(classData);
classList = classData;
await writeFiles();
return true;
}
async function writeFiles(){
fs.writeFileSync('./CommandData/classList.json', JSON.stringify(classList));
}
Before we finish off this file, if you want to store anything as a property in this file and then get it later on, you can do so. In order for the data to return properly though, you will need to define a getter function in your exports. Here is an example:
var classList;
module.exports={
getClassList: () => classList,
setup
};
So, with everything above you should have something that looks like this:
const classDataManager = require('./DataManagers/ClassData.js')
const fs = require('node:fs');
var classList;
async function setup(){
console.log('test');
//build option choice lists
let listsBuilt = await buildChoiceLists();
if (listsBuilt){
return true;
} else {
return false;
}
}
async function buildChoiceLists(){
let classListBuilt = await buildClassList();
return true;
}
async function buildClassList(){
let classData = await classDataManager.getClassData();
console.log(classData);
classList = classData;
await writeFiles();
return true;
}
async function writeFiles(){
fs.writeFileSync('./CommandData/classList.json', JSON.stringify(classList));
}
module.exports={
getClassList: () => classList,
setup
};
Next that pesky non-existent DataManager file. For mine, each data type will have its own, but you might want to just combine them all into a single .js file for yours.
Same with the folder name, I called mine DataManagers, if you're combining them all into one, you could just call the file DataManager and leave it in the same folder as your appSetup.js file.
For the data manager file all we really need is a function to get our data and then return it in the format we want it to be in. I am using node-fetch. If you are using some other module for data requests, write your code as needed.
Instead of explaining everything, here is the contents of my file, not much has to be explained here:
const fetch = require('node-fetch');
async function getClassData(){
return new Promise((resolve) => {
let data = "action=GetTestData";
fetch('http://xxx.xxx.xxx.xx/backend/characterHandler.php', {
method: 'post',
headers: { 'Content-Type':'application/x-www-form-urlencoded'},
body: data
}).then(response => {
response.json().then(res => {
let status = res.status;
let clsData = res.classes;
let rcData = res.races;
if (status == "Success"){
let text = '';
let classes = [];
let races = [];
if (Object.keys(clsData).length > 0){
for (let key of Object.keys(clsData)){
let cls = clsData[key];
classes.push({
"name": key,
"code": key.toLowerCase()
});
}
}
if (Object.keys(rcData).length > 0){
for (let key of Object.keys(rcData)){
let rc = rcData[key];
races.push({
"name": key,
"desc": rc.Desc
});
}
}
resolve(classes);
}
});
});
});
}
module.exports = {
getClassData
};
This file contacts our backend php and requests data from it. It queries the data then returns it. Then we format it into an JSON structure for use later on with option choices for the slash command.
Once all of your appSetup and data manager files are complete, we still need to create the commands and register them with the server. So, in your index file add something similar to the following:
async function getCommands(){
let cmds = await comCreator.appSetup();
console.log(cmds);
client.commands = cmds;
}
getCommands();
This should go at or near the top of your index.js file. Note that comCreator refers to a file we haven't created yet; you can name this require const whatever you wish. That's it for this file.
Now, the "comCreator" file. I named mine deploy-commands.js, but you can name it whatever. Once again, here is the full file contents. I will explain anything that needs to be explained after:
const {Collection} = require('discord.js');
const {REST} = require('#discordjs/rest');
const {Routes} = require('discord-api-types/v9');
const app = require('./appSetup.js');
const fs = require('node:fs');
const config = require('./config.json');
async function appSetup(){
console.log('test2');
let setupDone = await app.setup();
console.log(setupDone);
console.log(app.getClassList());
return new Promise((resolve) => {
const cmds = [];
const cmdFiles = fs.readdirSync('./commands').filter(f => f.endsWith('.js'));
for (let file of cmdFiles){
let cmd = require('./commands/' + file);
console.log(file + ' added to commands!');
cmds.push(cmd.data.toJSON());
}
const rest = new REST({version: '9'}).setToken(config.token);
rest.put(Routes.applicationGuildCommands(config.clientId, config.guildId), {body: cmds})
.then(() => console.log('Successfully registered application commands.'))
.catch(console.error);
let commands = new Collection();
for (let file of cmdFiles){
let cmd = require('./commands/' + file);
commands.set(cmd.data.name, cmd);
}
resolve(commands);
});
}
module.exports = {
appSetup
};
Most of this is boiler plate for slash command creation though I did combine the creation and registering of the commands into the same process. As you can see, we are grabbing our command files, processing them into a collection, registering that collection, and then resolving the promise with that variable.
You might have noticed that property, was used to then set the client commands in the index.js file.
Config just contains your connection details for your discord server app.
Finally, how I accessed the data we wrote for the SlashCommandBuilder:
data: new SlashCommandBuilder()
.setName('get-test-data')
.setDescription('Return Class and Race data from database')
.addStringOption(option =>{
option.setName('class')
.setDescription('Select a class for your character')
.setRequired(true)
let ops = [];
let data = fs.readFileSync('./CommandData/classList.json','utf-8');
ops = JSON.parse(data);
console.log('test data class options: ' + ops);
for(let op of ops){
option.addChoice(op.name,op.code);
}
return option
}
),
Hopefully this helps someone in the future!

react-beautiful-dnd and mysql state management

okay. I'm confused as to the best way to do this:
the following pieces are in play: a node js server, a client-side react(with redux), a MYSql DB.
in the client app I have lists (many but for this issue, assume one), that I want to be able to reorder by drag and drop.
in the mysql DB the times are stored to represent a linked list (with a nextKey, lastKey, and productionKey(primary), along with the data fields),
//mysql column [productionKey, lastKey,nextKey, ...(other data)]
the current issue I'm having is a render issue. it stutters after every change.
I'm using these two function to get the initial order and to reorder
function SortLinkedList(linkedList)
{
var sortedList = [];
var map = new Map();
var currentID = null;
for(var i = 0; i < linkedList.length; i++)
{
var item = linkedList[i];
if(item?.lastKey === null)
{
currentID = item?.productionKey;
sortedList.push(item);
}
else
{
map.set(item?.lastKey, i);
}
}
while(sortedList.length < linkedList.length)
{
var nextItem = linkedList[map.get(currentID)];
sortedList.push(nextItem);
currentID = nextItem?.productionKey;
}
const filteredSafe=sortedList.filter(x=>x!==undefined)
//undefined appear because server has not fully updated yet, so linked list is broken
//nothing will render without this
return filteredSafe
;
}
const reorder = (list, startIndex, endIndex) => {
const result = Array.from(list);
const [removed] = result.splice(startIndex, 1);
result.splice(endIndex, 0, removed);
const adjustedResult = result.map((x,i,arr)=>{
if(i==0){
x.lastKey=null;
}else{
x.lastKey=arr[i-1].productionKey;
}
if(i==arr.length-1){
x.nextKey=null;
}else{
x.nextKey=arr[i+1].productionKey;
}
return x;
})
return adjustedResult;
};
I've got this function to get the items
const getItems = (list,jobList) =>
{
return list.map((x,i)=>{
const jobName=jobList.find(y=>y.jobsessionkey==x.attachedJobKey)?.JobName;
return {
id:`ProductionCardM${x.machineID}ID${x.productionKey}`,
attachedJobKey: x.attachedJobKey,
lastKey: x.lastKey,
machineID: x.machineID,
nextKey: x.nextKey,
productionKey: x.productionKey,
content:jobName
}
})
}
my onDragEnd
const onDragEnd=(result)=> {
if (!result.destination) {
return;
}
// dropped outside the list
const items = reorder(
state.items,
result.source.index,
result.destination.index,
);
dispatch(sendAdjustments(items));
//sends update to server
//server updates mysql
//server sends back update events from mysql in packets
//props sent to DnD component are updated
}
so the actual bug looks like the graphics are glitching - as things get temporarily filtered in the sortLinkedList function - resulting in jumpy divs. is there a smoother way to handle this client->server->DB->server->client dataflow that results in a consistent handling in DnD?
UPDATE:
still trying to solve this. currently implemented a lock pattern.
useEffect(()=>{
if(productionLock){
setState({
items: SortLinkedList(getItems(data,jobList)),
droppables: [{ id: "Original: not Dynamic" }]
})
setLoading(false);
}else{
console.log("locking first");
setLoading(true);
}
},[productionLock])
where production lock is set to true and false from triggers on the server...
basically: the app sends the data to the server, the server processes the request, then sends new data back, when it's finished the server sends the unlock signal.
which should trigger this update happening once, but it does not, it still re-renders on each state update to the app from the server.
What’s the code for sendAdjustments()?
You should update locally first, otherwise DnD pulls it back to its original position while you wait for backend to finish. This makes it appear glitchy. E.g:
Set the newly reordered list locally as your state
Send network request
If it fails, reverse local list state back to the original list

chrome.devtools.inspectedWindow.eval - frameURL

I am trying to get the selected element to the sidebar pane in my chrome extension.
It's working fine if the page has no frames when the element is in the frame, it's not working.
As per the document I have to pass the frameURL, but how do I get the frame or Iframe URL?
Thank you.
Note: This issue is duplicate that was opened in 3 years ago, but still no solution there, so re-opening it again.
In devtools.js
chrome.devtools.panels.elements.createSidebarPane(name, (panel) => {
// listen for the elements changes
function updatePanel() {
chrome.devtools.inspectedWindow.eval("parseDOM($0)", {
frameURL: // how to pass dynamic
useContentScriptContext: true
}, (result, exceptipon) => {
if (result) {
console.log(result)
}
if (exceptipon) {
console.log(exceptipon)
}
});
}
chrome.devtools.panels.elements.onSelectionChanged.addListener(updatePanel);
});
I ran into this as well. I ended up needing to add a content_script on each page/iframe and a background page to help pass messages between devtools and content scripts.
The key bit is that in the devtools page, we should ask the content_scripts to send back what their current url is. For every content script that was registered, we can then call chrome.devtools.inspectedWindow.eval("setSelectedElement($0)", { useContentScriptContext: true, frameURL: msg.iframe } );
Or in full:
chrome.devtools.panels.elements.createSidebarPane( "example", function( sidebar ) {
const port = chrome.extension.connect({ name: "example-name" });
// announce to content scripts that they should message back with their frame urls
port.postMessage( 'SIDEBAR_INIT' );
port.onMessage.addListener(function ( msg) {
if ( msg.iframe ) {
// register with the correct frame url
chrome.devtools.panels.elements.onSelectionChanged.addListener(
() => {
chrome.devtools.inspectedWindow.eval("setSelectedElement($0)", { useContentScriptContext: true, frameURL: msg.iframe } );
}
);
} else {
// otherwise assume other messages from content scripts should update the sidebar
sidebar.setObject( msg );
}
} );
}
);
Then in the content_script, we should only process the event if we notice that the last selected element ($0) is different, since each frame on the page will also handle this.
let lastElement;
function setSelectedElement( element ) {
// if the selected element is the same, let handlers in other iframe contexts handle it instead.
if ( element !== lastElement ) {
lastElement = element;
// Pass back the object we'd like to set on the sidebar
chrome.extension.sendMessage( nextSidebarObject( element ) );
}
}
There's a bit of setup, including manifest changes, so see this PR for a full example:
https://github.com/gwwar/z-context/pull/21
You can found url of the frame this way:
document.querySelectorAll('iframe')[0].src
Assuming there is at lease one iframe.
Please note, you cannot use useContentScriptContext: true, as it will make the script execute as a context page (per documentation) and it will be in a separate sandboxed environment.
I had a slightly different problem, but it might be helpful for your case too, I was dynamically inserting an iframe to a page, and then tried to eval a script in it. Here the code that worked:
let win = chrome.devtools.inspectedWindow
let code = `
(function () {
let doc = window.document
let insertFrm = doc.createElement('IFRAME')
insertFrm.src = 'about:runner'
body.appendChild(insertFrm)
})()`
win.eval(code, function (result, error) {
if (error) {
console.log('Eror in insertFrame(), result:', result)
console.error(error)
} else {
let code = `
(function () {
let doc = window.document
let sc = doc.createElement('script')
sc.src = '${chrome.runtime.getURL('views/index.js')}'
doc.head.appendChild(sc)
})()`
win.eval(code, { frameURL: 'about:bela-runner' }, function (result, error) {
if (error) {
console.log('Eror in insertFrame(), result:', result)
console.error(error)
}
})
}
})

Calling external function from within Phantomjs+node.js

I'm going to be honest. I'm way in over my head here.
I need to scrape data from a dynamic site for my employer. Before the data is visible on the page, there are some clicks and waits necessary. Simple PHP scraping won't do. So I found out about this NodeJS + PhantomJS combo. Quite a pain to set up, but I did manage to load a site, run some code and get a result.
I wrote a piece of jQuery which uses timeout loops to wait for some data to be loaded. Eventually I get a js object that I want to write to a file (JSON).
The issue I'm facing.
I build up the the js object inside the PhantomJS .evaluate scope, which runs in a headerless browser, so not directly in my Node.JS server scope. How do I send the variable I built up inside evaluate back to my server so I can write it to my file?
Some example code (I know it's ugly, but it's for illustrative purposes). I use node-phantom-simple as a bridge between Phantom and Node
var phantom = require('node-phantom-simple'),
fs = require('fs'),
webPage = 'https://www.imagemedia.com/printing/business-card-printing/'
phantom.create(function(err, ph) {
return ph.createPage(function(err, page) {
return page.open(webPage, function(err, status) {
page.onConsoleMessage = function(msg) {
console.log(msg);
};
console.log("opened site? ", status);
page.evaluate(function() {
setTimeout(function() {
$('.price-select-cnt').eq(0).find('select').val('1266').change()
timeOutLoop()
function timeOutLoop() {
console.log('looping')
setTimeout(function() {
if ($('#ajax_price_tool div').length != 6) {
timeOutLoop()
} else {
$('.price-select-cnt').eq(1).find('select').val('25')
$('.price-select-cnt').eq(2).find('select').val('Premium Card Stock')
$('.price-select-cnt').eq(3).find('select').val('Standard').change()
timeOutLoop2()
}
}, 100)
}
function timeOutLoop2() {
console.log('looping2')
setTimeout(function() {
if ($('.pricing-cost-cnt').text() == '$0' || $('.pricing-cost-cnt').text() == '') {
timeOutLoop2()
} else {
var price = $('.pricing-cost-cnt').text()
console.log(price)
}
}, 100)
}
}, 4000)
});
});
});
});
function writeJSON(plsWrite) {
var key = 'file'
fs.writeFile('./results/' + key + '.json', plsWrite, 'utf8', function() {
console.log('The JSON file is saved as');
console.log('results/' + key + '.json');
});
}
So do do I write the price this code takes from the website, get it out of the evaluate scope and write it to a file?

Loading a texture in THREE.js using Node

I am trying to build a messenger bot which does some image processing in 3d and returns a brand new image. I use THREE.CanvasRenderer and my app is hosted on Heroku.
When a user /POSTs an attachment to my webhook, I want to take the URL of the newly created image and insert it into my 3d Scene.
This is my code (using the node-canvas library):
const addTexture = (imageUrl) => {
request({
uri: imageUrl,
method: 'GET'
}, (err, res, body) => {
let image = new Canvas.Image();
image.src = body;
mesh.material.map = new THREE.Texture(image);
mesh.material.needsUpdate = true;
});
}
The callback gets run and I can actually console.log() the image's contents, but nothing shows up in the scene - the plane I am supposed to render to just gets black without any errors... What am I missing here?
I also tried several other ways, without any success...
I tried using THREE.TextureLoader and patch it with jsdom (mock document, window) and node-xmlhttprequest, but then there is error with my load event (event.target is not defined...) Just like in this example
How should one approach this problem? I have a url generated by Facebook, I want to download the image from it and place it in my scene?
Okay, I figured it out after half a day and am posting it for future generations:
Here is the code that did the trick for me:
const addTexture = (imageUrl) => {
request.get(imageUrl, (err, res, data) => {
if (!err && res.statusCode == 200) {
data = "data:" + res.headers["content-type"] + ";base64," + new Buffer(data).toString('base64');
let image = new Canvas.Image();
image.onload = () => {
mesh.material.map = new THREE.Texture(image);
mesh.material.map.needsUpdate = true;
}
image.src = data;
}
});
}
This way, I still get an error: document is not defined, because it seems under the hood three.js writes the texture to a separate canvas.
So the quick and ugly way is to do what I did:
document.createElement = (el) => {
if (el === 'canvas') {
return new Canvas()
}
}
I really hope this helps somebody out there, because besides all the hurdles, rendering WebGL on the server is pure awesomeness.

Resources