I'm developing a multiplayer word game for mobile in Unity. I've been using GameSparks as a backend, but felt that it didn't give me the control I wanted over the project. After looking around a bit, I decided to go with Firebase. It felt like a good decision, but now I'm experiencing some serious slowness with the database + Cloud Functions
The game has a turn based mode where you play a round and get a notification when your opponent has finished. A round is uploaded via a cloudscript call, with only the necessary data. The new game state is then pieced together in the cloudscript and the game database entry (activeCasualGames/{gameId}) updated. After this is done, I update a gameInfo (activeCasualGamesInfo/{gameId}) entry with some basic info about the game, then send a cloud message to notify the opponent that it's their turn.
The data sent is only 32 kb, and no complex changes are made to the code (see cloud function code below). The complete database entry for the game would be around 100kb max at this point, yet the time from sending the round to the game being updated on server and opponent notified ranges from 50 seconds to more than a minute. On GameSparks, the same operation takes maybe a few seconds.
Does anyone else experience this slowness with Firebase? Have I made some mistake in the cloud script function, or is this the performance I should expect? The game also has a live mode, which I haven't started implementing in Firebase yet, and with this kind of lag, I'm not feeling that it's gonna work out too well
Many thanks in advance!
exports.uploadCasualGameRound = functions.https.onCall(
(roundUploadData, response) => {
const roundData = JSON.parse(roundUploadData);
const updatedGameData = JSON.parse(roundData.gameData);
const updatedGameInfo = JSON.parse(roundData.gameInfo);
console.log("game data object:");
console.log(updatedGameData);
console.log("game info:");
console.log(updatedGameInfo);
const gameId = updatedGameData.gameId;
console.log("updating game with id: " + gameId);
admin
.database()
.ref("/activeCasualGames/" + gameId)
.once("value")
.then(function(snapshot: { val: any }) {
let game = snapshot.val();
console.log("old game:");
console.log(game);
const isNewGame = (game as boolean) === true;
if (isNewGame === false) {
console.log("THIS IS AN EXISTING GAME!");
} else {
console.log("THIS IS A NEW GAME!");
}
// make the end state of the currently stored TurnBasedGameData the beginning state of the uploaded one (limits the upload of data and prevents timeout errors)
if (isNewGame === true) {
updatedGameData.gameStateRoundStart.playerOneRounds =
updatedGameData.gameStateRoundEnd.playerOneRounds;
} else {
// Player one rounds are not uploaded by player two, and vice versa. Compensate for this here:
if (updatedGameData.whoseTurn === updatedGameData.playerTwoId) {
updatedGameData.gameStateRoundEnd.playerTwoRounds =
game.gameStateRoundEnd.playerTwoRounds;
} else {
updatedGameData.gameStateRoundEnd.playerOneRounds =
game.gameStateRoundEnd.playerOneRounds;
}
updatedGameData.gameStateRoundStart = game.gameStateRoundEnd;
}
game = updatedGameData;
console.log("Game after update:");
console.log(game);
// game.lastRoundFinishedDate = Date.now();
game.finalRoundWatchedByOtherPlayer = false;
admin
.database()
.ref("/activeCasualGames/" + gameId)
.set(game)
.then(() => {
updatedGameInfo.roundUploaded = true;
return admin
.database()
.ref("/activeCasualGamesInfo/" + gameId)
.set(updatedGameInfo)
.then(() => {
exports.sendOpponentFinishedCasualGameRoundMessage(gameId, updatedGameInfo.lastPlayer, updatedGameInfo.whoseTurn);
return true;
});
});
});
}
);
There is some things you must to consider when using a Firebase Cloud Function + Realtime Database. This question may help about the serverless performance issues, such cold start. Also, I think the code itself may be refactored to make less calls to the realtime database, as every external request, it may take time. Some tips are to use more local-stored resources (in memory, browser cache, etc) and use some global variables.
Related
I am trying to listen to the changes in a specific field in a specific document in Firebase Realtime database. I am using Node JS for the cloud functions. this is the function.
const functions = require("firebase-functions");
const admin = require('firebase-admin');
admin.initializeApp(functions.config.firebase);
const delay = ms => new Promise(res => setTimeout(res, ms));
exports.condOnUpdate = functions.database.ref('data/').onWrite(async snapshot=> {
const snapBefore = snapshot.before;
const snapAfter = snapshot.after;
const dataB = snapBefore.val();
const data = snapAfter.val();
const prev= dataB['cond'];
const curr= data['cond'];
// terminate if there is no change
if(prev==curr)
{
return;
}
if(curr==0){
// send notifications every 10 seconds until value becomes 1
while(true){
admin.messaging().sendToTopic("all", payload);
await delay(10000);
}
}else if(curr==1){
// send one notification
admin.messaging().sendToTopic("all", payload);
return;
}
});
The function works as expected but the loop never stops as it never exists the loop, instead, the function runs again (with new instance I suppose).
so is there any way to listen to the data changes in one function just like streams in other languages, or perhaps stop all cloud functions from running.
Thanks in advance!
From the comments it seems that the cond property is updated from outside of the Cloud Function. When that happens, it will trigger a new instance of the Cloud Function to run with its own curr variable, but it won't cause the curr value in the current instance to be updated. So the curr variable in the original instance of your code will never become true, and it will continue to run until it times out.
If you want the current instance to detect the change to the property, you will need to monitor that property inside the code too by calling onValue on a reference to it.
An easier approach though might be to use a interval trigger rather than a database trigger to:
have code execute every minute, and then in there
query the database with the relevant cond value, and then
send a notification to each of those.
This requires no endless loop or timeouts in your code, which is typically a better approach when it comes to Cloud Functions.
This Meteor code failed to return the number of documents in Vehicles collection when console.log in the client code. Please help find the problem where I went wrong. Thx
//imports/api/vehicles.js
import {Mongo} from 'meteor/mongo';
export const Vehicles = new Mongo.Collection('vehicles');
///////////////////////////////////////////
//server/publish.js
import {Vehicles} from '../imports/api/vehicles.js'
Meteor.publish('vehicles', function(){
return Vehicles.find({})
})
///////////////////////////////////////////
//server/main.js
import { Vehicles } from '../imports/api/vehicles.js';
//so I added this tying to fix the problem for no avail
Meteor.startup(() => {
Meteor.publish('vehicles', function () {
return Vehicles.find();
});
});
///////////////////////////////////////////
//client/main.js
import {Vehicles} from '../imports/api/vehicles.js'
Meteor.startup(function(){
Meteor.subscribe('vehicles');
console.log('subscribed') //<<<<<< prints "subscribed"
let rec = Vehicles.find({}).count()
console.log(rec) //<<<<<< prints "0"
})
///////////////////////////////////////////
Blaze collection findOne return no document
After reading the first response, in regarding Blaze will do it automatically, and I don't have to do Meteor.startup Meteor.subscribe.onReady... please note below that blaze template helper gave undefined.
//client/main.js
Tracker.autorun(() => {
Meteor.subscribe('vehicles', {plate: Session.get('plate')});
});
Template.vehicle.helpers({
'vehicle' : function(){
let vehicle = Vehicles.findOne({'plate':Session.get('plate')})
console.log(vehicle) //prints undefined
}
})
Template.vehicle.events({
'keyup #plate'(e, inst){
let str = e.currentTarget.value
Session.set('plate', str)
console.log(str) // prints the value OK
}
})
The subscription will not be ready immediately. You need to first wait for the data to arrive at the client. Remember, everything on the client is asynchronous, as there are no Fibers in the browser (unlike on the server). This is why the subscribe function has a onReady callback you can specify:
Meteor.startup(function(){
Meteor.subscribe('vehicles', {onReady: () => {
console.log('subscription ready')
let rec = Vehicles.find({}).count()
console.log(rec)
}});
console.log('subscribed') // << this will print first
})
In practice you won't need this though, because you will usually use the data from a collection in a UI component, and there the UI framework (e.g., Blaze or React) will take take of the reactive re-rendering once data has arrived. In Blaze that is completely automatic, in React you will need to use useTracker or withTracker. Either way, the test you are running is still good to develop understanding of Meteor.
If you want to learn even more, open the developer console in your browser, go to Network, and watch the messages on the WS (web socket). You'll learn a lot about how DDP works, the protocol used by Meteor for synchronizing data between server and client. Knowing even just the basics of DDP can really help you build better Meteor applications and feel more confident with it.
I have been developing a game where on the user submitting data, the client writes some data to a Firebase Realtime Database. I then have a Google Cloud Function which is triggered onUpdate. That function checks the submissions from various players in a particular game and if certain criteria are met, the function writes an update to the DB which causes the clients to move to the next round of the game.
This is all working, however, I have found performance to be quite poor.
I've added logging to the function and can see that the function takes anywhere from 2-10ms to complete, which is acceptable. The issue is that the update is often written anywhere from 10-30 SECONDS after the function has returned the update.
To determine this, my function obtains the UTC epoch timestamp from just before writing the update, and stores this as a key with the value being the Firebase server timestamp.
I then have manually checked the two timestamps to arrive at the time between return and database write:
The strange thing is that I have another cloud function which is triggered by an HTTP request, and found that the update from that function is typically from 0.5-2 seconds after the function calls the DB update() API.
The difference between these two functions, aside from how they are triggered, is how the data is written back to the DB.
The onUpdate() function writes data by returning:
return after.ref.update(updateToWrite);
Whereas the HTTP request function writes data by calling the update API:
dbRef.update({
// object to write
});
I've provided a slightly stripped out version of my onUpdate function here (same structure but sanitised function names) :
exports.onGameUpdate = functions.database.ref("/games/{gameId}")
.onUpdate(async(snapshot, context) => {
console.time("onGameUpdate");
let id = context.params.gameId;
let after = snapshot.after;
let updatedSnapshot = snapshot.after.val();
if (updatedSnapshot.serverShouldProcess && !isFinished) {
processUpdate().then((res)=>{
// some logic to check res, and if criteria met, move on to promises to determine update data
determineDataToWrite().then((updateToWrite)=>{
// get current time
var d = new Date();
let triggerTime = d.getTime();
// I use triggerTime as the key, and firebase.database.ServerValue.TIMESTAMP as the value
if(updateToWrite["timestamps"] !== null && updateToWrite["timestamps"] !== undefined){
let timestampsCopy = updateToWrite["timestamps"];
timestampsCopy[triggerTime] = firebase.database.ServerValue.TIMESTAMP;
updateToWrite["timestamps"][triggerTime] = firebase.database.ServerValue.TIMESTAMP;
}else{
let timestampsObj = {};
timestampsObj[triggerTime] = firebase.database.ServerValue.TIMESTAMP;
updateToWrite["timestamps"] = timestampsObj;
}
// write the update to the database
return after.ref.update(updateToWrite);
}).catch((error)=>{
// error handling
})
}
// this is just here because ES Linter complains if there's no return
return null;
})
.catch((error) => {
// error handling
})
}
});
I'd appreciate any help! Thanks :)
Edit: Removing irrelevant code to improve readability
Edit 2: Reducing example to only uploadGameRound function and adding log output with times.
I'm working on a mobile multiplayer word game and was previously using the Firebase Realtime Database with fairly snappy performance apart from the cold starts. Saving an updated game and setting stats would take at most a few seconds. Recently I made the decision to switch to using Firestore for my game data and player stats / top lists, primarily because of the more advanced queries and the automatic scaling with no need for manual sharding. Now I've got things working on Firestore, but the time it takes to save an updated game and update a number of stats is just ridiculous. I'm clocking average between 3-4 minutes before the game is updated, stats added and everything is available in the database for other clients and viewable in the web interface. I'm guessing and hoping that this is because of something I've messed up in my implementation, but the transactions all go through and there are no warnings or anything else to go on really. Looking at the cloud functions log, the total time from function call to completion log statement appears to be a bit more than a minute, but that log doesn't appear until after same the 3-4 minute wait for the data.
Here's the code as it is. If someone has time to have a look and maybe spot what's wrong I'd be hugely grateful!
This function is called from Unity client:
exports.uploadGameRound = functions.https.onCall((roundUploadData, response) => {
console.log("UPLOADING GAME ROUND. TIME: ");
var d = new Date();
var n = d.toLocaleTimeString();
console.log(n);
// CODE REMOVED FOR READABILITY. JUST PREPARING SOME VARIABLES TO USE BELOW. NOTHING HEAVY, NO DATABASE TRANSACTIONS. //
// Get a new write batch
const batch = firestoreDatabase.batch();
// Save game info to activeGamesInfo
var gameInfoRef = firestoreDatabase.collection('activeGamesInfo').doc(gameId);
batch.set(gameInfoRef, gameInfo);
// Save game data to activeGamesData
const gameDataRef = firestoreDatabase.collection('activeGamesData').doc(gameId);
batch.set(gameDataRef, { gameDataCompressed: updatedGameDataGzippedString });
if (foundWord !== undefined && foundWord !== null) {
const wordId = foundWord.timeStamp + "_" + foundWord.word;
// Save word to allFoundWords
const wordRef = firestoreDatabase.collection('allFoundWords').doc(wordId);
batch.set(wordRef, foundWord);
exports.incrementNumberOfTimesWordFound(gameInfo.language, foundWord.word);
}
console.log("COMMITTING BATCH. TIME: ");
var d = new Date();
var n = d.toLocaleTimeString();
console.log(n);
// Commit the batch
batch.commit().then(result => {
return gameInfoRef.update({ roundUploaded: true }).then(function (result2) {
console.log("DONE COMMITTING BATCH. TIME: ");
var d = new Date();
var n = d.toLocaleTimeString();
console.log(n);
return;
});
});
});
Again, any help with understanding this weird behaviour massively appreciated!
Ok, so I found the problem now and thought I should share it:
Simply adding a return statement before the batch commit fixed the function and reduced the time from 4 minutes to less than a second:
RETURN batch.commit().then(result => {
return gameInfoRef.update({ roundUploaded: true }).then(function (result2) {
console.log("DONE COMMITTING BATCH. TIME: ");
var d = new Date();
var n = d.toLocaleTimeString();
console.log(n);
return;
});
});
Your function isn't returning a promise that resolves with the data to send to the client app. In the absence of a returned promise, it will return immediately, with no guarantee that any pending asynchronous work will terminate correctly.
Calling then on a single promise isn't enough to handle promises. You likely have lots of async work going on here, between commit() and other functions like incrementNumberOfTimesWordFound. You will need to handle all of the promises correctly, and make sure your overall function returns only a single promise that resolves when all that work is complete.
I strongly suggest taking some time to learn how promises work in JavaScript - this is crucial to writing effective functions. Without a full understanding, things will appear to go wrong, or not at all, in strange ways.
I have the following function that gets a message from aws SQS, the problem is I get one at a time and I wish to get all of them, because I need to check the ID for each message:
function getSQSMessages() {
const params = {
QueueUrl: 'some url',
};
sqs.receiveMessage(params, (err, data) => {
if(err) {
console.log(err, err.stack)
return(err);
}
return data.Messages;
});
};
function sendMessagesBack() {
return new Promise((resolve, reject) => {
if(Array.isArray(getSQSMessages())) {
resolve(getSQSMessages());
} else {
reject(getSQSMessages());
};
});
};
The function sendMessagesBack() is used in another async/await function.
I am not sure how to get all of the messages, as I was looking on how to get them, people mention loops but I could not figure how to implement it in my case.
I assume I have to put sqs.receiveMessage() in a loop, but then I get confused on what do I need to check and when to stop the loop so I can get the ID of each message?
If anyone has any tips, please share.
Thank you.
I suggest you to use the Promise api, and it will give you the possibility to use async/await syntax right away.
const { Messages } = await sqs.receiveMessage(params).promise();
// Messages will contain all your needed info
await sqs.sendMessage(params).promise();
In this way, you will not need to wrap the callback API with Promises.
SQS doesn't return more than 10 messages in the response. To get all the available messages, you need to call the getSQSMessages function recursively.
If you return a promise from getSQSMessages, you can do something like this.
getSQSMessages()
.then(data => {
if(!data.Messages || data.Messages.length === 0){
// no messages are available. return
}
// continue processing for each message or push the messages into array and call
//getSQSMessages function again.
});
You can never be guaranteed to get all the messages in a queue, unless after you get some of them, you delete them from the queue - thus ensuring that the next requests returns a different selection of records.
Each request will return 'upto' 10 messages, if you don't delete them, then there is a good chance that the next request for 'upto' 10 messages will return a mix of messages you have already seen, and some new ones - so you will never really know when you have seen them all.
It maybe that a queue is not the right tool to use in your use case - but since I don't know your use case, its hard to say.
I know this is a bit of a necro but I landed here last night while trying to pull some all messages from a dead letter queue in SQS. While the accepted answer, "you cannot guarantee to get all messages" from the queue is absolutely correct I did want to drop an answer for anyone that may land here as well and needs to get around the 10 message limit per request from AWS.
Dependencies
In my case I have a few dependencies already in my project that I used to make life simpler.
lodash - This is something we use in our code for help making things functional. I don't think I used it below but I'm including it since it's in the file.
cli-progress - This gives you a nice little progress bar on your CLI.
Disclaimer
The below was thrown together during troubleshooting some production errors integrating with another system. Our DLQ messages contain some identifiers that I need in order to formulate cloud watch queries for troubleshooting. Given that these are two different GUIs in AWS switching back and forth is cumbersome given that our AWS session are via a form of federation and the session only lasts for one hour max.
The script
#!/usr/bin/env node
const _ = require('lodash');
const aswSdk = require('aws-sdk');
const cliProgress = require('cli-progress');
const queueUrl = 'https://[put-your-url-here]';
const queueRegion = 'us-west-1';
const getMessages = async (sqs) => {
const resp = await sqs.receiveMessage({
QueueUrl: queueUrl,
MaxNumberOfMessages: 10,
}).promise();
return resp.Messages;
};
const main = async () => {
const sqs = new aswSdk.SQS({ region: queueRegion });
// First thing we need to do is get the current number of messages in the DLQ.
const attributes = await sqs.getQueueAttributes({
QueueUrl: queueUrl,
AttributeNames: ['All'], // Probably could thin this down but its late
}).promise();
const numberOfMessage = Number(attributes.Attributes.ApproximateNumberOfMessages);
// Next we create a in-memory cache for the messages
const allMessages = {};
let running = true;
// Honesty here: The examples we have in existing code use the multi-bar. It was about 10PM and I had 28 DLQ messages I was looking into. I didn't feel it was worth converting the multi-bar to a single-bar. Look into the docs on the github page if this is really a sticking point for you.
const progress = new cliProgress.MultiBar({
format: ' {bar} | {name} | {value}/{total}',
hideCursor: true,
clearOnComplete: true,
stopOnComplete: true
}, cliProgress.Presets.shades_grey);
const progressBar = progress.create(numberOfMessage, 0, { name: 'Messages' });
// TODO: put in a time limit to avoid an infinite loop.
// NOTE: For 28 messages I managed to get them all with this approach in about 15 seconds. When/if I cleanup this script I plan to add the time based short-circuit at that point.
while (running) {
// Fetch all the messages we can from the queue. The number of messages is not guaranteed per the AWS documentation.
let messages = await getMessages(sqs);
for (let i = 0; i < messages.length; i++) {
// Loop though the existing messages and only copy messages we have not already cached.
let message = messages[i];
let data = allMessages[message.MessageId];
if (data === undefined) {
allMessages[message.MessageId] = message;
}
}
// Update our progress bar with the current progress
const discoveredMessageCount = Object.keys(allMessages).length;
progressBar.update(discoveredMessageCount);
// Give a quick pause just to make sure we don't get rate limited or something
await new Promise((resolve) => setTimeout(resolve, 1000));
running = discoveredMessageCount !== numberOfMessage;
}
// Now that we have all the messages I printed them to console so I could copy/paste the output into LibreCalc (excel-like tool). I split on the semicolon for rows out of habit since sometimes similar scripts deal with data that has commas in it.
const keys = Object.keys(allMessages);
console.log('Message ID;ID');
for (let i = 0; i < keys.length; i++) {
const message = allMessages[keys[i]];
const decodedBody = JSON.parse(message.Body);
console.log(`${message.MessageId};${decodedBody.id}`);
}
};
main();