MongoDB misses saving data in a long period of time - node.js

Good afternoon, I have a script which is receiving data from Binance API every 30 minutes and saves it in MongoDB specific database in different collections.
My MongoDB is installed on VPS. I'm connecting to it from my local computer.
Problem is that I leave my code to work constantly for ± 3 days and I receive that sometimes data did not save on specific collection or missed, etc. How I can configure my code or what can I do to create a perfect connection to save all data correctly.
Problem Explanation:
I have an array with
symbols=["ada","ae","kava","eth","etc","zrx","xzc","faq","gas","vfg","req"];
And when I leave the code for let's say 25 hours. I suppose to see 50 documents on every collection name.
But I receive that some collections got 48 documents saved instead of 50, some of them got 49, some of them got 50 like sometimes collections didn't save d properly.
FULLCODE HERE:
const { MongoClient } = require('mongodb');
const schedule = require('node-schedule');
const fetch = require("node-fetch");
const symbols=["ada","ae","kava","eth","etc","zrx","xzc","faq","gas","vfg","req"];
//a descriptive name helps your future self and others understand code easier
const getBTCData = async symbol => { //make this function accept the current symbol
//async/await lets us write this much nicer and with less nested indents
let data = await fetch(`https://api.binance.com/api/v3/klines?symbol=${symbol}&interval=30m&limit=1`).then(res => res.json());
const btcusdtdata = data.map(d => {
return {
Open: parseFloat(d[1]),
High: parseFloat(d[2]),
Low: parseFloat(d[3]),
Close: parseFloat(d[4]),
Volume: parseFloat(d[5]),
Timespan: 30,
}
});
console.log(btcusdtdata);
saveToDatebase(symbol, btcusdtdata);
//recursive functions are complicated, we can get rid of it here
//by moving the responsibility to the caller
};
//helper function for an awaitable timeout
const sleep = ms => new Promise(res => setTimeout(res, ms));
const j = schedule.scheduleJob('*/30 * * * *', async() => {
//expand this function to be responsible for looping the data
for (let symbol of symbols) {
//we can pass symbol to getBTCData instead of making it
//responsible for figuring out which symbol it should get
await getBTCData(symbol);
await sleep(8000);
}
});
//make this a helper function so `saveToDatabase()` isn't also responsible for it
const getDateTime = () => {
let today = new Date();
let date = today.getFullYear() + '-' + (today.getMonth() + 1) + '-' + today.getDate();
let time = today.getHours() + ":" + today.getMinutes() + ":" + today.getSeconds();
return date + ' ' + time;
};
const saveToDatebase = async(symbol, BTCdata) => {
const url = 'mongodb://username:password#server:port/dbname?retryWrites=true&w=majority';
let dateTime = getDateTime();
//use await here and below to vastly simplify this function
let db = await MongoClient.connect(url, { useUnifiedTopology: true });
const dbo = db.db('Crypto');
const myobj = { Name: symbol, Array: BTCdata, Date: dateTime };
await dbo.collection(symbol).insertOne(myobj);
console.log('1 document inserted');
db.close();
};
Goal: Solve the saving issue, I think it also can be possible something with VPS response itself. How can I maybe update my code that it looks cleaner and faster if possible?
Any suggestions or help really appreciated.

Related

How do I get data from mongodb, manipulate it and print it

I want to connect to MongoDB and query a collection based on the filter 'category'. From the results, I want to randomly select one of entries, concatenate with another string and print to console. I am a novice at this and can't figure out how to get the results of my query and randomly select one of the entries from the query results.
//Connecting To DB
const mongoose = require('mongoose');
//const { makeCard } = require('./utils/card');
const db = mongoose.connection;
const host = process.env.host;
console.log(host)
const dbupdate = {
useNewUrlParser : true,
useUnifiedTopology : true
};
mongoose.connect(host, dbupdate);
db.on('error', (err) => console.log('Error, DB not connected'));
db.on('connected', () => console.log('connected to mongo'));
db.on('disconnected', () => console.log('Mongo is disconnected'));
db.on('open', () => console.log('Connection Made!'));
const Schema= mongoose.Schema;
const cardSchema = new Schema({
word: String,
visual : String,
category : String
});
const Card = mongoose.model('expressions', cardSchema );
--I want this function to return the results of the query, but it doesn't.
function getWords(ctgry ) {
const result = Card.find({"category" : ctgry },(error,results) => {
return results;
});
};
function getRandomInt(min, max) {
min = Math.ceil(min);
max = Math.floor(max);
return Math.floor(Math.random() * (max - min) + min);
};
function getWord( ctgry ) {
const result = getWords( ctgry );
const num = getRandomInt(0,result.length); --this doesn't work
return result[num];
};
console.log("Sample text: " + getWord());
What you seem to be doing here is that you're fetching all the documents from the Card model for a particular category and picking a random one from it, which I wouldn't recommend. Not sure what data you're storing here, but what if your app grows to a point where there are a million documents in your cards collection? You wouldn't fetch them all in memory and pick a random one.
A good practice while using any database is to try to only fetch as much data is required. There's no way to fetch a random document from a query in mongodb from what I know, but there's a way to do this (inspired from this answer which I recommend you read).
async function getRandomWord(ctgry) {
const count = await User.count({
"category": ctgry,
}).exec();
const random = Math.floor(Math.random() * count);
return User.findOne({
"category": ctgry,
}).skip(random).exec();
}

Is it possible to fetch the AWS S3 Uploaded Image File Data and pass that file data through Twitter API from Node JS?

I have a Scheduler where I can post scheduled tweets and threads and it will post through my twitter account the whole backend is based on Node JS.
So right now for Posting scheduled Tweets and Tweet threads with images I'm using mongoDB, storing the base64 string data, but I don't want to store such a big data to mongo, is it possible to upload images to AWS S3 first and then when time matches as Scheduled time that Image will be fetched again to my Node Server and pass it through the Twitter API?
In this way I will have to just store the AWS S3 image link to my MongoDB and get rid of a big string data.
API
// THREAD SCHEDULER API
app.post('/TweetThread/:id', async (req, res) => {
const ID = req.params.id;
const { texts, dateTimeGMT } = req.body;
const filter = { _id: ObjectId(ID) };
const findUser = await userCollection.findOne(filter);
// GET TIMEZONE FROM INSTANT REQ
const timeZone = Intl.DateTimeFormat().resolvedOptions().timeZone;
const threadSchData = {
texts,
date: new Date(dateTimeGMT).toLocaleDateString(),
time: new Date(dateTimeGMT).toLocaleTimeString(),
timezone: timeZone,
token: findUser.token,
token_secret: findUser.token_secret,
screen_name: findUser.screen_api
}
const result = await threadDataCollection.insertOne(threadSchData);
res.json(result);
});
Scheduler
// TWITTER THREAD SCHEDULER
const threadSch = cron.schedule('*/1 * * * *', async () => {
const cursor = threadDataCollection.find({});
const threadsArray = await cursor.toArray();
// console.log(threadsArray);
threadsArray.forEach(thread => {
// Twitter Thread CONFIG
const configThread = {
consumer_key: process.env.CONSUMER_KEY,
consumer_secret: process.env.CONSUMER_SECRET,
access_token: thread.token,
access_token_secret: thread.token_secret
};
// INSERTING THREAD DATA TO THREAD FUNCTION
async function tweetThread() {
const t = new TwitThread(configThread);
await t.tweetThread(thread.texts);
}
// Calculating The Time According to Time ZONE
let scheduledTime = new Date(thread.date + ' ' + thread.time);
let now = new Date();
let year = now.getFullYear();
let month = now.getMonth() + 1;
let day = now.getDate();
let hour = now.getHours();
let minute = now.getMinutes();
let new_date = new Date(year + '-' + month + '-' + day + ' ' + hour + ':' + minute + ':' + '00');
const countingTime = scheduledTime.toLocaleString("en-US", { timeZone: thread.timezone });
const serverTime = new_date.toLocaleString("en-US", { timeZone: thread.timezone });
// POSTING THREAD WHEN TIME MATCHED
if (countingTime === serverTime) {
tweetThread();
console.log('Posted Thread Once');
console.log('TRUE');
}
});
});
threadSch.start();
This is the request body where in media_data field I'm passing the string data from front-end
you need credentials to connect to aws from command line. Then you have only to run from the image path aws s3 cp your_file_name s3:\\your_bucket_name\your_file_name .
Then save this your_file_name to MongoDB and upload it from here. There are more tricks to use cloudformation and to make s3 public so you can only point to the file from your html page. If you need help with any of the steps please write me.

MongoDB aggregation returns empty array with NodeJS

I want to create a script which is taking the average of the Volume for last 7(for example) days.
I'm stuck with aggregation stages since first stage I need to take Date for last 7 days and in second stage calculate Average of Volume
Package list:
Node-schedule - */1 * * * * (Runs the script every minute)
Binance API - Taking data from them.
Screenshot for showcasing how the document looks like in MongoDB.
Aggregation part of the Code.
const average = await dbo.collection(symbol).aggregate([{
'$match': {
'Date': { '$gte': new Date((new Date().getTime() - (7 * 24 * 60 * 60 * 1000))) }
},
},
{
'$group': {
_id: null,
'Volume': { '$avg': '$Volume' }
},
}
]).toArray();
This code returns me an empty array in terminal like this > []
Full Code here.
const { MongoClient } = require('mongodb');
const schedule = require('node-schedule');
const fetch = require("node-fetch");
const symbols = ["ADABTC", "AEBTC", "AIONBTC", "ALGOBTC", "ARDRBTC"];
//a descriptive name helps your future self and others understand code easier
const getBTCData = async symbol => { //make this function accept the current symbol
//async/await lets us write this much nicer and with less nested indents
let data = await fetch(`https://api.binance.com/api/v3/klines?symbol=${symbol}&interval=30m&limit=1`).then(res => res.json());
const btcusdtdata = data.map(d => {
return {
Open: parseFloat(d[1]),
High: parseFloat(d[2]),
Low: parseFloat(d[3]),
Close: parseFloat(d[4]),
Volume: parseFloat(d[5]),
Timespan: 30,
}
});
console.log(btcusdtdata);
saveToDatebase(symbol, btcusdtdata);
//recursive functions are complicated, we can get rid of it here
//by moving the responsibility to the caller
};
//helper function for an awaitable timeout
const sleep = ms => new Promise(res => setTimeout(res, ms));
const j = schedule.scheduleJob('*/1 * * * *', async() => {
//expand this function to be responsible for looping the data
for (let symbol of symbols) {
//we can pass symbol to getBTCData instead of making it
//responsible for figuring out which symbol it should get
await getBTCData(symbol);
await sleep(8000);
}
});
//make this a helper function so `saveToDatabase()` isn't also responsible for it
const getDateTime = () => {
let today = new Date();
let date = today.getFullYear() + '-' + (today.getMonth() + 1) + '-' + today.getDate();
let time = today.getHours() + ":" + today.getMinutes() + ":" + today.getSeconds();
return date + ' ' + time;
};
const saveToDatebase = async(symbol, BTCdata) => {
try {
const url = 'mongodb://username:password#ipadress:port/dbname';
let dateTime = getDateTime();
let db = await MongoClient.connect(url, { useUnifiedTopology: true });
const dbo = db.db('Crypto');
const myobj = Object.assign({ Name: symbol, Date: dateTime }, BTCdata[0]);
await dbo.collection(symbol).insertOne(myobj);
const average = await dbo.collection(symbol).aggregate([{
'$match': {
'Date': { '$gte': new Date((new Date().getTime() - (7 * 24 * 60 * 60 * 1000))) }
},
},
{
'$group': {
_id: null,
'Volume': { '$avg': '$Volume' }
},
}
]).toArray();
console.log('1 document inserted');
console.log(average);
db.close();
} catch (e) {
console.error(e)
}
};
EDIT1
If I delete $match part my script is working and I receive average of Volume.
Screenshot of terminal after success try without $match
EDIT2
According to the last answer I understand that I need to change Date format from string to object, but I really can't get how I can do it in this part?
const getDateTime = () => {
let today = new Date();
let date = today.getFullYear() + '-' + (today.getMonth() + 1) + '-' + today.getDate();
let time = today.getHours() + ":" + today.getMinutes() + ":" + today.getSeconds();
return date + ' ' + time;
};
EDIT3
After editing the Date format I receive a Document in MongoDB in strange Date format like - Date:2020-07-20T13:24:02.390+00:00
Code here:
const getDateTime = () => {
let today = new Date();
let date = today.getFullYear() + '-' + (today.getMonth() + 1) + '-' + today.getDate();
let time = today.getHours() + ":" + today.getMinutes() + ":" + today.getSeconds();
return new Date();
};
The problem is on the Date field format.
The getDateTime function returns a string so Mongo is managing the field as a string not as a Date object so the $gte check will compare string not dates.
You should change the function to getDateTime = () => new Date(). Mongo will manage the date correctly storing in UTF Timezone.
Tring to query a date-string in the $match field would be really difficult.
Edit:
To update the typing just:
const getDateTime = () => {
return new Date();
};

How to get DocumentReferences in array inside a Document

Very new to await/async pattren in cloud functions. Have a cloud functions that fetches data from a doucment which has an array of DocumentReferences
/* Create new Chilla for a given date */
exports.createNewSalikChilla = functions.https.onRequest(async (req, res) => {
const email=req.body.email;
const date=req.body.date;
console.log('Creating chilla for Salik <' + email + '> on <' + date + '>');
const db = admin.firestore();
const habits = [];
const salikRef = db.collection('saliks').doc(email);
const salik = await salikRef.get();
if (!salik.exists) {
throw new Error("Salik not found for <" + email + ">");
}
const assignedHabits = salik.data()['assigned_habits'];
assignedHabits.forEach(element => {
//LOST on how to get these Document Reference and push to habits = []
});
res.send({status: "Success"});
});
The document in the saliks collection has the following structure on firestore
assigned_habits<array>:
0:
habit<reference>: /habits/first
required<number>: 200
1:
habit<reference>: /habits/second
required<number>: 4
name: "Hani Q"
But have tried everything and can't seem to figure out how to use async/await here to get all the DocumementReferecnce from the array and push to Habit Array and then after all is done i send back res.send({status: "Success"});
Answer
Below worked after implementing the accepted answer
const assignedHabits = salik.data()['assigned_habits'];
const habits_data = (await Promise.all(assignedHabits.map(element => element.habit.get())))
.map(snapshot => snapshot.data());
console.log(habits_data);
res.send({status: habits_data});
Whenever you need to wait for a bunch of asynchronous operations, you'll want to use Promise.all(). In your case that'd look something like:
const assignedHabits = salik.data()['assigned_habits'];
const habits = await Promise.all(assignedHabits.map(element => element.habit.get()));
res.send({status: "Success"});
The above code assumes that habit is a DocumentReference field type.

Discord.js Command Cooldown + Time Remaining

Okay so I am looking to make it so that the cooldown shows how much longer the user needs to wait until they can work again. The cool down works butI want it to show the time remainign rather than it saying you need to wait 15 mnutes before typing this command. Is it Possible?
const { RichEmbed } = require("discord.js");
const { stripIndents } = require("common-tags");
const { prefix } = require("../../botconfig.json");
const db = require('quick.db')
let bal = require("../../database/balance.json");
let works = require('../../database/works.json');
const fs = require('fs');
const talkedRecently = new Set();
//Set cooldown
module.exports = {
name: "work",
aliases: [],
category: "economy",
description: "Gets you money",
usage: "[command | alias]",
run: async (client, message, args) => {
if (talkedRecently.has(message.author.id)) {
message.channel.send("You have to wait TIME minutes before you can work again")
} else {
if(!bal[message.author.id]){
bal[message.author.id] = {
balance: 0
};
}
if(!works[message.author.id]) {
works[message.author.id] = {
work: 0
};
}
const Jwork = require('../../work.json');
const JworkR = Jwork[Math.floor(Math.random() * Jwork.length)];
var random = Math.floor(Math.random() * 20) + 3;
let curBal = bal[message.author.id].balance
bal[message.author.id].balance = curBal + random;
let curWork = works[message.author.id].work
works[message.author.id].work = curWork + 1;
fs.writeFile('././database/works.json', JSON.stringify(works, null, 2), (err) => {
if (err) console.log(err)
})
fs.writeFile('././database/balance.json', JSON.stringify(bal, null, 2), (err) => {
let embed = new RichEmbed()
.setColor("RANDOM")
.setDescription(`
**\💼 | ${message.author.username}**, ${JworkR} 💴 **${random}**
`)
message.channel.send(embed)
if (err) console.log(err)
});
// Adds the user to the set so that they can't talk for a minute
talkedRecently.add(message.author.id);
setTimeout(() => {
// Removes the user from the set after a minute
talkedRecently.delete(message.author.id);
}, 900000);
}
}
}
Unfortunately, your current system won't be any help. You'll have to store more than just the user if you want to use the timings of their cooldown.
Let's use a Map for our variable so we can have key-value pairs. This will make it easier to keep track of the information we need
// Replace talkedRecently's declaration with this...
const cooldowns = new Map();
To put a user on cooldown, use Map.set() to add the user and the time at which their cooldown should expire to cooldowns. Then, use Map.delete() when the cooldown should run out to allow the user access to the command again.
// Replace the talkedRecently.add(...) section with this...
cooldowns.set(message.author.id, Date.now() + 900000);
setTimeout(() => cooldowns.delete(message.author.id), 900000);
In order to determine the amount of time remaining on the cooldown, we have to subtract the current time from that at which it expires. However, this will give us milliseconds, rendering the value unreadable to us. A simple, easy way to convert duration into words is by using humanize-duration (moment is also an option). Finally, we can send the desired message, letting the user know how much time they have left on their cooldown.
// Put this where you require your other dependencies...
const humanizeDuration = require('humanize-duration');
// Replace the if (talkedRecently.has(...)) part with this...
const cooldown = cooldowns.get(message.author.id);
if (cooldown) {
const remaining = humanizeDuration(cooldown - Date.now());
return message.channel.send(`You have to wait ${remaining} before you can work again`)
.catch(console.error);
}

Resources