I'm trying to process images from the WebCam using Posenet on the server-side, but I'm not sure how to pass the image data to the estimateSinglePose.
Below is the simplified version of the code;
CLIENT
const imageData = context.getImageData(0, 0, 320, 180);
const buffer = imageData.data.buffer;
socket.emit("signal", buffer); //Pass it to the server through websocket
BACKEND
socket.on("signal", (data)=> {
const buffer = new Uint8Array(data);
const image = ts.tensor(data).reshape([180, 320, -1]);
// this where I'm stuck, I don't know how to pass the image to the estimateSinglePose
})
EDIT 1
Passing it to the estimateSinglePose resulted in an error.
Error: Invalid TF_Status: 3
Message: Incompatible shapes: [193,257,4] vs. [3]
at NodeJSKernelBackend.executeSingleOutput (/Users/xxx/app/server/node_modules/#tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:209:43)
at Object.kernelFunc (/Users/xxx/app/server/node_modules/#tensorflow/tfjs-node/dist/kernels/Add.js:28:24)
at kernelFunc (/Users/xxx/app/server/node_modules/#tensorflow/tfjs-core/dist/tf-core.node.js:3139:32)
at /Users/xxx/app/server/node_modules/#tensorflow/tfjs-core/dist/tf-core.node.js:3200:27
at Engine.scopedRun (/Users/xxx/app/server/node_modules/#tensorflow/tfjs-core/dist/tf-core.node.js:3012:23)
at Engine.runKernelFunc (/Users/xxx/app/server/node_modules/#tensorflow/tfjs-core/dist/tf-core.node.js:3196:14)
at Engine.runKernel (/Users/xxx/app/server/node_modules/#tensorflow/tfjs-core/dist/tf-core.node.js:3068:21)
at add_ (/Users/xxx/app/server/node_modules/#tensorflow/tfjs-core/dist/tf-core.node.js:8969:19)
at Object.add__op [as add] (/Users/xxx/app/server/node_modules/#tensorflow/tfjs-core/dist/tf-core.node.js:3986:29)
at ResNet.preprocessInput (/Users/xxx/app/server/node_modules/#tensorflow-models/posenet/dist/resnet.js:41:19)
estimateSinglePose takes as parameter an HTMLImageElement or HTMLVideoElement. Server side with nodejs, you can use the package canvas to have the same behavior as the canvas in the browser
const posenet = require('#tensorflow-models/posenet');
const {Image, createCanvas} = require('canvas');
const canvas = createCanvas(img.width,img.height); // 180, 320
const ctx = canvas.getContext('2d');
const net = await posenet.load();
socket.on("signal", async (data)=> {
ctx.putImageData(data, 0, 0)
const pose = await net.estimateSinglePose(canvas, {
flipHorizontal: false
});
// you can now use pose
})
Related
I have been trying to fix this thing for over a week now, searching in google and youtube for this damn error and no solution have been found. I know Im at first days of node js so my code might be not perfect, but the canvas one were taken from people that did tutorials and I have try as many of them but the result is always the same error no matter what. Apparently the error going away if I remove every related displayAvatarURL code, which sucks because I can't use the user avatar in my welcome image. I have try to change formats, changing code, changing background image with a black one made with Gimp (not that matter because the problem seem is avatar)and removed background to check again. Nothing work. The bot will crash ALWAYS as soon a real user join probably because the avatar image and it DOESN'T crash when invite a bot for testing (because it doesnt have custom avatars apparently?). Thank you for the help.
Error:
node:events:505
throw er; // Unhandled 'error' event
^
Error: Unsupported image type
at setSource (C:**\Desktop\lolinya_bot_js\node_modules\canvas\lib\image.js:91:13)
at C:**\Desktop\lolinya_bot_js\node_modules\canvas\lib\image.js:59:11
at C:**\Desktop\lolinya_bot_js\node_modules\simple-get\index.js:97:7
at IncomingMessage. (C:**\Desktop\lolinya_bot_js\node_modules\simple-concat\index.js:88:13)
at Object.onceWrapper (node:events:641:28)
at IncomingMessage.emit (node:events:539:35)
at endReadableNT (node:internal/streams/readable:1345:12)
at processTicksAndRejections (node:internal/process/task_queues:83:21)
Emitted 'error' event on Client instance at:
at emitUnhandledRejectionOrErr (node:events:384:10)
at processTicksAndRejections (node:internal/process/task_queues:85:21)
Index.js:
const {
Client,
GatewayIntentBits,
Routes,
// InteractionType,
} = require('discord.js');
const Discord = require('discord.js');
const { REST } = require('#discordjs/rest');
const fs = require('node:fs');
// const { Console } = require('node:console');
const generateImage = require('../generateImage.js');
dotenv.config();
// - CONFIG TOKEN, CLIENT AND GUILD ID
const TOKEN = process.env.TOKEN;
const CLIENT_ID = process.env.CLIENT_ID;
const GUILD_ID = process.env.GUILD_ID;
const WELCOME_ID = process.env.WELCOME_ID;
const client = new Client({
intents: [
GatewayIntentBits.Guilds,
GatewayIntentBits.GuildMessages,
GatewayIntentBits.GuildMembers,
GatewayIntentBits.MessageContent,
],
});
// - CONFIG SLASH COMMANDS -
const commands = [];
const commandFiles = fs.readdirSync('./src/commands')
.filter(file => file.endsWith('js'));
client.slashcommands = new Discord.Collection();
for (const file of commandFiles) {
const command = require(`./commands/${file}`);
client.slashcommands.set(command.data.name, command);
commands.push(command.data.toJSON());
}
const rest = new REST({ version: '10' }).setToken(TOKEN);
// - CLIENT EMITTERS -
client.on('ready', () => {
console.log(`${client.user.tag} has logged in.`);
});
client.on('guildMemberAdd', async (member) => {
const img = await generateImage(member);
member.guild.channels.cache.get(WELCOME_ID).send({
content: `<#${member.id}> Welcome to the server!`,
files: [img]
});
});
client.on('interactionCreate', async (interaction) => {
if (!interaction.isChatInputCommand()) return;
const slashcmd = client.slashcommands.get(interaction.commandName);
await slashcmd.run({ client, interaction });
});
// - INITIALIZE THE BOT AND ALSO REFRESH SLASH COMMANDS LIST -
(async () => {
try {
console.log('Started refreshing application (/) commands.');
await rest.put(Routes.applicationGuildCommands(CLIENT_ID, GUILD_ID), {
body: commands,
});
console.log('Successfully reloaded application (/) commands.');
client.login(TOKEN);
} catch (err) {
console.error(err);
}
})();
generateImage.js:
const { createCanvas, loadImage, registerFont } = require('canvas');
registerFont('./font/Cat paw.ttf', {family: 'neko'});
const Discord = require('discord.js');
const { AttachmentBuilder } = require('discord.js');
const background = "https://i.imgur.com/VZblp7S.jpg";
const dim = {
height: 675,
width: 1200,
margin: 50,
}
const av = {
size: 256,
x: 480,
y: 170,
}
const generateImage = async (member) => {
let canvas = await createCanvas(dim.width, dim.height);
let ctx = await canvas.getContext('2d');
let username = member.user.username;
let discrim = member.user.discriminator;
// let avatarURL = member.displayAvatarURL({format: 'jpg', dynamic: true, size: av.size})
// Draw the canvas for our image
// const canvas = Canvas.createCanvas(dim.width, dim.height);
// const ctx = canvas.getContext('2d');
const backimg = await loadImage(background);
let x = 0 //canvas.width / 2 - backimg.width / 2;
let y = 0 //canvas.height / 2 - backimg.height / 2;
ctx.drawImage(backimg, x, y);
// Draw a semi-transparent box for text readability
ctx.fillStyle = "rgba(0,0,0,0.7)"
ctx.fillRect(
dim.margin,
dim.margin,
dim.width - 2 * dim.margin,
dim.height - 2 * dim.margin
); //fillRect(posX,posY, width, height)
ctx.save();
ctx.beginPath();
ctx.arc(
av.x + av.size / 2,
av.y + av.size / 2,
av.size / 2, 0,
Math.PI * 2,
false
); //arc(centerX, centerY, radius, startAngle, endAngle, clockwise)
ctx.clip();
let avimg = await loadImage(member.displayAvatarURL({ format: 'png' }));
ctx.drawImage(avimg, av.x, av.y);
ctx.restore();
// Config our welcome text
ctx.fillStyle = 'pink';
ctx.textAlign = 'center';
// Draw the welcome text
ctx.font = '80px Cat paw';
ctx.fillText("Welcome to the Server", dim.width / 2, dim.margin + 70)
// Draw the username text
ctx.font = '100px Cat paw';
ctx.fillText(`${username} #${discrim}`, dim.width / 2, dim.height - dim.margin - 125);
// Draw the server name text
ctx.font = '40px Cat paw';
ctx.fillText(`You are the member #${member.guild.memberCount}!`, dim.width / 2, dim.height - dim.margin - 50);
let attachment = new AttachmentBuilder(canvas.toBuffer(), { name: 'welcome.png' })
return attachment;
}
module.exports = generateImage ```
If you get the unsupported image type error with the avatar URL -> On discord.js v14 format was replaced with extension so you'll have to change that
I have the same issue, it comes from the fact that
member.displayAvatarURL({ format: 'png' })
returns a .webp instead of returning a .png. It's strange, maybe it's bug from Discord.js v14.
I have found an ugly fix:
let avimg = await loadImage("https://cdn.discordapp.com/avatars/" + member.id + "/" > + member.avatar + ".png");
I keep getting this error when importing an image and using canvas with discord.js:
C:\Users\Travi\OneDrive\Documents\GitHub\re\src\img\licenseTemp.png:1
�PNG
SyntaxError: Invalid or unexpected token
Here's my code:
I'm also using module Alias so that's why it's const licenseTemp = require('#img')
const BaseCommand = require('../../utils/structures/BaseCommand');
const Canvas = require('canvas');
const { MessageAttachment } = require('discord.js');
const licenseTemp = require('#img');
module.exports = class RankCommand extends BaseCommand {
constructor() {
super('rank', 'Information', []);
}
async run(client, message, args) {
const canvas = Canvas.createCanvas(449, 292);
const ctx = canvas.getContext('2d');
const background = await Canvas.loadImage(licenseTemp);
ctx.drawImage(background, 0, 0, canvas.width, canvas.height);
const attachment = new MessageAttachment(canvas.toBuffer(), 'license.png');
message.channel.send(attachment);
}
}
Do not require('#img'), this will try to make your image's content read as text be part of the script.
Instead let node-canvas's loadImage handle the fetching of that resource: make licenseTemp be the URL to that image directly.
My server receives a list base64 image eg [img1,img2....] at an endpoint.
I have function getPoses(img) that takes an image and returns an object back.
I want to be able to map the images to getPoses() and get the object back for each image in Parallel.
I have tried Async.each function but I notice it's not really Parallel. Probably because it runs in a single thread/core.
I have also tried to use cluster, so that I can utilize all the cores to run getPoses() in multiple cores. But I feel stuck because, It's not clear to me how the master can obtain the poses found by workers.
I would be super gratefully for any help from the community.
Aync code:
//END POINT
app.post('/postImage' ,async function(req,res){
const imgBase64List = req.body.imgBase64.split('next');
const imgBase64IndexList = []
imgBase64List.map((imgBase64,index)=>{
imgBase64IndexList.push([index,imgBase64])
})
async.each(imgBase64IndexList , getPoseDetection , function(err){
if(err){
console.log("Error occured when feteching one of the pose")
console.log(err)
}else{
console.log('Sending Poses!')
res.send(JSON.stringify(poseDetectionResults))
}
})
});//End of app.post and asyn function
const getPoseDetection = async (imgBase64Index, callback) => {
//console.log('start');
const img = new Image();
img.src = imgBase64Index[1];
const imgIndex = imgBase64Index[0]
const canvas = createCanvas(img.width, img.height);
const ctx = canvas.getContext('2d');
ctx.drawImage(img, 0, 0);
const input = tf.browser.fromPixels(canvas);
const imageScaleFactor = 0.3;
const flipHorizontal = false;
const outputStride = 8;
const poses = await net.estimateMultiplePoses(input, {
flipHorizontal: false,
maxDetections: 2,
minPoseConfidence: 0.15,
minPartConfidence:0.1,
nmsRadius:20,
});
boundingBoxes = [] //reset the boundingBoxese
poses.forEach(pose => {
var box = posenet.getBoundingBoxPoints(pose.keypoints);
boundingBoxes.push(box)
});
poseDetectionResults[imgIndex] = {detectionList:poses}
//console.log(poses)
}
Thank You
I'm trying to create and image on the Google Firebase server with node-canvas and store it in Firebase Storage.
const functions = require('firebase-functions');
const admin = require('firebase-admin');
const gcs = require('#google-cloud/storage')();
const path = require('path');
const Canvas = require('canvas-prebuilt');
const env = require('dotenv').config();
try {admin.initializeApp(functions.config().firebase);} catch(e) {}
//Trigger on creation of a new post
exports.default = functions.database.ref('/posts/{postId}').onCreate(event => {
//Get the postID
const postId = event.params.postId;
console.log('We have a new post ' + postId);
//Get data from the postid
return admin.database().ref('/posts/' + postId).once('value').then(function(snapshot) {
const text = snapshot.val().text;
const canvas = new Canvas(1024, 512);
const ctx = canvas.getContext('2d');
//Draw Background
ctx.fillStyle = '#000';
ctx.fillRect(0, 0, 1024 , 512);
//Draw Text
ctx.font = 'bold 66px Arial';
ctx.textAlign = 'center';
ctx.fillStyle = '#fff';
ctx.fillText(text, 120, 256, 784);
// Use the postID to name the file Ex : -L1rVJUAtSbc_FampT0D.png
var filename = postId + '.png';
// Create the file metadata
var metadata = {
contentType: 'image/png'
};
const bucket = gcs.bucket('images');
const filePath = 'images/' + filename;
return canvas.toDataURL('image/png', function(err, png){
//Save on Firebase Storage
return bucket.upload(png, {
destination: filePath,
metadata: metadata
}).then(() => {
console.log('Image uploaded to Storage at ', filePath);
});
});
});
});
But, when I try to save it with toDataURL I get this error :
ENAMETOOLONG: name too long, stat 'data:image/png;base64,iVBORw0 ...'
And when I try with toBuffer I get this one :
TypeError: Path must be a string. Received
at assertPath (path.js:7:11)
at Object.basename (path.js:1362:5)
at Bucket.upload (/user_code/node_modules/#google-cloud/storage/src/bucket.js:2259:43)
at /user_code/node_modules/#google-cloud/storage/node_modules/#google-cloud/common/src/util.js:777:22
at Bucket.wrapper [as upload] (/user_code/node_modules/#google-cloud/storage/node_modules/#google-cloud/common/src/util.js:761:12)
at /user_code/sendTweet.js:107:21
I also try toBlob but the function doesn't exist server side with node-canvas.
Anyone know how I should save the image server side before transfer it to Firebase Storage?
Thanks!
I am trying to convert multiple svg strings to png's so I can render them onto a pdf using PdfMake in nodejs. This works fine for one svg, but when I add multiple svg strings, they get overwritten by the last one. With this example code, it renders two images of png2 (svg2).
const promises = [svg1,svg2].map(str => {
const stream = new Readable();
stream.push(str);
stream.push(null);
return svgPromise(stream);
});
const result = await Promise.all(promises);
const png1 = result[0].content;
const png2 = result[1].content;
function svgPromise(stream) {
return new Promise((resolve, reject) => {
const svg = new Rsvg();
stream.pipe(svg);
svg.on("finish", function() {
const buffer = svg.render({
format: "png",
width: width * 2,
height: height * 2
}).data;
const png = datauri.format(".png", buffer);
resolve(png);
});
});
}
Not sure if this error is related to stream or my promise logic. Any ideas?
Dependencies:
"librsvg": "0.7.0"
"pdfmake": "0.1.35"
"datauri": "1.0.5"
It pays to list all the used modules. Assuming you used datauri, it seems you need to initialize a new instance for every call:
svg.on("finish", function() {
const datauri = new Datauri();
const buffer = svg.render({
format: "png",
width: 16,
height: 16
}).data;
const png = datauri.format(".png", buffer);
resolve(png);
});