Before I asked, I`ve searched a lot and there are many articles about it. But my question is a little deeper.
I have an application using Nodejs Expressjs + MongoDB + Reddit + PM2 clustering mode + Bitcoin and card getaway + API system.
My problem is when I'm developing this application in real mode and it`s really awful. sometimes I release little updates in codes, and I press "pm2 log" it shows me some error in syntax or something else and I try to fix that and release again. During this time, the application with many users is down.
Also, I have to say something such as Bitcoin payment, needs real tests. Needs request and response from Blockchain. How can I have a test environment that I can test everything exactly same as real mode and then if everything was fine, then deploy that to real mode?
An environment that easy to code and test then easy to deploy? Can Mocha help me exactly what I need? I`m using PM2 clustering mode.
Your question is not a proper question, but rather a layer of questions, some opinion-based, some too broad to answer. But let's try breaking it down.
The stated problem is when I'm developing this application in real mode.... I release updates ... it shows me syntax error... application is down. I will read it as the main problem is that you're developing on a production environment. Let's forget for a while how lousy a practice that is, and let's focus on something constructive.
Let's define rough steps to take.
Live environment
The most pressing problem seems to be that you work on live app, where crashing it during dev means crashing it for your users as well. Let's deal with that.
Immediatelly change all your access codes, keys, usernames and passwords so that you store them in an environment file (which is safe to encrypt and backup, but not commited in source code), say, environment-prod.env.
Then create a second set of credentials for all the services that you use. For MongoDB, e.g. it's easy, just create a local database instance, called, say, test_database. For Reddit, create a second app, cal it my-app-test, for example. Some services might have an option to create a set of test credentials right there in the app, with others you'll simply have one app for test, one for production.
Create a new environment file, e.g. test-environment.env, with all the same keys (e.g. REDDIT_APPID, REDDIT_SECRET, MONGODB_URL, BLOCKCHAIN_GATEWAY_KEY etc), but new values.
Now, for one, you have a test environment. Make an alias, e.g. alias dev="cd $HOME/projects/my-reddit-bitcoin-app && source test-environment.env". Every day you come to work on the app, type dev, then you can start pm2 etc and work safely in dev environment. Your users will never see your crashes.
Only when you're sure you have a new feature or bugfix completed, switch environment (source environment-production.env) and then deploy the new app to the server where it runs, and pm2 restart or whatever you use for these deployments. Switch back to test env immediately before working on the code anew.
Read up more on how to separate test/prod environments. Read a bit on git workflows (e.g. branch off of latest master to a feature-branch or bugfix branch, when tested, merge it back in. Then tag it "release-" and deploy to production. Then go automate all that if possible.)
Testing
Mocha is perfectly suitable for running tests for a Node/Express app. It's the tests that matter.
You say bitcoin payment....Needs request and response. Let's see how to do that.
Add [nock])(https://www.npmjs.com/package/nock) to your app (npm i -D nock).
import it and put it on top of your test file. E.g. at the top of the some-test.spec.js file:
const nock = require('nock')
start recording requests e.g. add this in your before() block in the app:
describe('My tests', function () {
before(function () {
nock.recorder.rec();
});
// ... tests
Now, run one test at a time (e.g. write one test that does one specific task from your app) and check what's in the console. E.g. if you make a request (request.post('http://reddit.com/api/submit', jsonData)), you'll see nock printing the exact response (in JSON format) in console, as the test runs. Copy that in the test file, e.g. put it at the bottom as:
var testResponse = <whatever was in the console in json format. Or string, whatever>. // homework is to find out why var and not const, if this is at the end of the file.
Now stop the recorder (comment it out), and in your actual test, run this instead:
const pipe = nock('http://www.example.com')
.get('/resource')
.reply(200, testResponse);
do that for all your requests.
Now what you have is a test setup so that when you change the code, it should not run against the real Reddit api, or real payment gateway api, but get your mocked responses instead. Pair it up with some good assertions and you should be fine. Make sure you mock everything. If you add new types of requests, make sure to record them, and add them to your procedure.
Now, all this is very vague. Broad. Just one way to do it. Lengthy process. Probably not the best one. Not tailored to your specific conditions. But it should get you started. Take those things, step by step, and if you get stuck, come back to Stackoverflow. But do start working on it, because your current method seems to be unsustainable in the long run.
Related
Bots are amazing, unless you're Google Analytics
After many months of learning to host my own Discord bot, I finally figured it out! I now have a node server running on my localhost that sends and receives data from my Discord server; it works great. I can do all kinds of the things I want to with my Discord bot.
Given that I work with analytics everyday, one project I want to figure out is how to send data to Google Analytics (specifically GA4) from this node server.
NOTE: I have had success in sending data to my Universal Analytics property. However, as awesome as that was to finally see pageviews coming into, it was equally heartbreaking to recall that Google will be getting rid of Universal Analytics in July of this year.
I have tried the following options:
GET/POST requests to the collect endpoint
This option presented itself as impossible from the get-go. In order to send a request to the collection endpoint, a client_id must be sent along with the request itself. And this client_id is something that must be generated using Google's client id algorithm. So, I can't just make one up.
If you consider this option possible, please let me know why.
Install googleapis npm package
At first, I thought I could just install the googleapis package and be ready to go, but that idea fell on its face immediately too. With this package, I can't send data to GA, I can only read with it.
Find and install a GTM npm package
There are GTM npm packages out there, but I quickly found out that they all require there to be a window object, which is something my node server would not have because it isn't a browser.
How I did this for Universal Analytics
My biggest goal is to do this without using Python, Java, C++ or any other low level languages. Because, that route would require me to learn new languages. Surely it's possible with NodeJS alone... no?
I eventually stumbled upon the idea of actually hosting a webpage as some sort of pseudo-proxy that would send data from the page to GA when accessed by something like a page scraper. It was simple. I created an HTML file that has Google Tag Manager installed on it, and all I had to do was use the puppeteer npm package.
It isn't perfect, but it works and I can use Google Tag Manager to handle and manipulate input, which is wonderful.
Unfortunately, this same method will not work for GA4 because GA4 automatically excludes all identified bot traffic automatically, and there is no way to turn that setting off. It is a very useful feature for GA4, giving it quite a bit more integrity than UA, and I'm not trying to get around that fact, but it is now the Bane of my entire goal.
https://support.google.com/analytics/answer/9888366?hl=en
Where to go from here?
I'm nearly at the end of my wits on figuring this one out. So, either an npm package exists out there that I haven't found yet, or this is a futile project.
Does anyone have any experience in sending data from NodeJS to GA4? (or even GTM?) How did you do it?
...and this client_id is something that must be generated using Google's client id algorithm. So, I can't just make one up...
Why, of course you can. GA4 generates it pretty much the same as UA does. You don't need anything from google to do it.
Besides, instead of mimicking just requests to the collect endpoint, you may just wanna go the MP route right away: https://developers.google.com/analytics/devguides/collection/protocol/ga4 The links #dockeryZ gave, work perfectly fine. Maybe try opening them in incognito, or in a different browser? Maybe you have a plugin blocking analytics urls.
Moreover, you don't really need to reinvent the bicycle. Node already has a few packages to send events to GA4, here's one looking good: https://www.npmjs.com/package/ga4-mp?activeTab=readme
Or you can just use gtag directly to send events. I see a lot of people doing it even on the front-end: https://www.npmjs.com/package/ga-gtag Gtag has a whole api not described in there. Here's more on gtag: https://developers.google.com/tag-platform/gtagjs/reference Note how the library allows you to set the client id there.
The only caveat there is that you'll have to track client ids and session ids manually. Shouldn't be too bad though. Oh, and you will have to redefine the concept of a pageview, I guess. Well, the obvious one is whenever people post in the chan that is different from the previous post in a session. Still, this will have to be defined in the code.
Don't worry about google's bot traffic detection. It's really primitive. Just make sure your useragent doesn't scream "bot" in it. Make something better up.
I need to find the best possible way to do the following task.
I have different users let's say (over 500) and all users have a scheduled function that need to be run twice every day. But if any of the user's phone is off. Then of course that function wont work since its code is written on client side.
now what i want to do is Run a scheduled function in the backend using Node js, but idk how to run that for every user. (note : every user has different schedules). Thats why i wrote that in client side, but with with a possibility of phone might be switched off so its bit off to do that.
What should i do in this scenario? any leads?
I'm developing APIs for social networking web application for learning perspective. When I started to write test cases I stuck around how to organize/write test cases.I'm initially proceeding like this:
First setup global data base initialization:
I need some users' auth tokens to test my routes so I decide to set up these information at global context. Also there are other information that also need to setup at global context so I'm setting that too.
Then for each route:
I start to write test cases and thought that I would write test cases in
such a way that each route test cases would be independent to each other.
and after completing all test suites:
I thought that I would clean up my data base.
The problems with this approach I'm facing are:
Say I want to test four routes named /users , /users/:id/my_invites , /send_invites, /response_invites. And further suppose I'm only interested in writing test cases for GET request and response for /users , /users/:id/invites and POSTing data in the case of others.
/send_invites, /response_invites definitely trigger some actions on the server side that modifies the data base state.
As we see that these routes effect the state of other routes' data say one user sends invites to another user and gets response true/false, so for that user his request was successful but how to ensure that another user actually received the invitation if we don't checking his received invitation documents(i.e through another route) in the first route test cases. Means /send_invites effects the /users/:id/my_invites .
because these routes are dependent on each others
So The questions I want to ask are:
how to write test cases for these routes so that each route would be independent ?
I tried with three dummy testing users in the global context and trying all sort of combinations for them in all test suites.My test suit presently deals with more than one routes to check 1 route true functionality.
Can anyone suggest me better solution for writing test cases for the above mentioned scenarios?
May be my question is too long or not clear. Please let me know and help me if you can.
My opinion:
First the bases EACH test case should have a fresh new state. That means that before you check one scenario, you want to flush your database and insert your new data prepared to test that case scenario. You can use a real database o mock one with data in-memory or however you prefer.
Second, each endpoint potentially affects a lot of tables in your database. So it's perfectly normal to check the state of your data, independently of the tables to check that the information is correct.
And well mocha and other test frameworks have functions that help you to do that. Like beforeEach and afterEach to set up and tear down your data before each test case.
I am working on a small hobby project, where I would really like some input and advice.
This is my first "real" node project, and I hope it will teach me a lot about node.js development. I am a .net developer by day, and have been for about 15 years professionally. I have had periods of doing Java as well. I have created small node.js projects to be used as micro services.
But this project can no longer be classified as a micro service ;-)
The purpose of the project is to sample some sensor data, and do some reporting. An idea I got from playing around with a PLC at university. I do that by sampling from a PLC, and emitting the data using ZeroMQ. My node.js server then listens for this sensor data, and stores it in a MongoDB.
I expose that data in a REST api. The REST api also exposes resources like batches and other stuff like authentication etc. On top of that I have an AngularJS app, that creates defines the UI.
The one thing that really annoys me, is that I want to globally assign what batch is running. I have a collection of batches, and one of them is the running one. There are a two ways I see to do this, and both illustrate my novice status in the node.js world. All users should be able to see what batch is running, and I want to be able to easily tell from anywhere in the code as well.
1) Set a flag on the object in Mongo. This has a number of problems. The obvious one being performance. I receive sensor data 10 times a second, and I don't want to ask the database every time what batch to save it under.
2) Save the info on the global object. I really don't like this either. I don't like global state in my code.
What is a good pattern for doing something like this? Does my question make any sense?
Thanks in advance
You can make a simple REST call to set the active batch and call it inside the batch when is started up and ready to accept requests. For example:
app.put('/active-batch', function(req, res, next){
// Make sure req.body is defined
app.set('active-batch', req.body);
res.end();
});
Then everywhere in the code you can use:
app.get('active-batch');
The app.set let you save data globally accessible in your app and app.get let you read previously stored data.
How would I mock out the database in my node.js application, which in this case uses mongodb as the backend for a blog REST API ?
Sure, I could set the database to a specific testing -database, but I would still save data and not test my code only, but also the database, so I am actually not doing unit testing but integration testing.
So what should one do? Create database wrappers as a middle layer between application and db and replace the DAL when in testing?
// app.js
var express = require('express');
app = express(),
mongo = require('mongoskin'),
db = mongo.db('localhost:27017/test?auto_reconnect');
app.get('/posts/:slug', function(req, res){
db.collection('posts').findOne({slug: req.params.slug}, function (err, post) {
res.send(JSON.stringify(post), 200);
});
});
app.listen(3000);
// test.js
r = require('requestah')(3000);
describe("Does some testing", function() {
it("Fetches a blogpost by slug", function(done) {
r.get("/posts/aslug", function(res) {
expect(res.statusCode).to.equal(200);
expect(JSON.parse(res.body)["title"]).to.not.equal(null);
return done();
});
});
));
I don't think database related code can be properly tested without testing it with the database software. That's because the code you're testing is not just javascript but also the database query string. Even though in your case the queries look simple you can't rely on it being that way forever.
So any database emulation layer will necessarily implement the entire database (minus disk storage perhaps). By then you end up doing integration testing with the database emulator even though you call it unit testing. Another downside is that the database emulator may end up having a different set of bugs compared to the database and you may end up having to code for both the database emulator and the database (kind of like the situation with IE vs Firefox vs Chrome etc.).
Therefore, in my opinion, the only way to correctly test your code is to interface it with the real database.
There is a general rule of thumb when it comes to mocking which is
Don't mock anything you don't own.
If you want to mock out the database hide it behind an abstracted service layer and mock that layer. Then make sure you integration test the actual service layer.
Personally I've gone away from using mocks for testing and use them for top-to-bottom design helping me drive development from the top towards the bottom mocking out service layers as I go and then eventually implementing those layers and writing integration tests. Used as a test tool they tend to make your test very brittle and in the worst case leads to a divergence between actual behavior and mocked behavior.
I don't agree with the selected answer or other replies so far.
Wouldn't it be awesome if you could catch errors spawned by the chaotic and many times messy changes made to DB schemas and your code BEFORE it gets to QA? I bet the majority would shout heck yes!
You most certainly can and should isolate and test you DB schemas. And you don't do it based on an emulator or heavy image or recreation of you DB and machine. This is what stuff like SQLite is for just as one example. You mock it based on an in memory lightweight instance running and with static data that does not change in that in memory instance which means you are truly testing your DB in isolation and you can trust your tests as well. And obviously it's fast because it's in memory, a skeleton, and is scrapped at the end of a test run.
So yes you should and you should test the SCHEMA that is exported into a very lightweight in memory instance of whatever DB engine/runtime you are using, and that along with adding a very small amount of static data becomes your isolated mocked DB.
You export your real schemas from your real DB periodically (in an automated fashion) and import/update those into your light in memory DB instance before every push to QA and you will know instantly if any latest DB changes done by your DB admins or other developers who have changed the schema lately have broken any tests .
As for the person who replied with the "don't mock anything you don't own". I think he meant to say "don't test anything you don't own". But you DO mock things you do not own! Because those are the things not under test that need to be isolated!
This is what many test driven teams do all the time. You just have to understand the how.
My preferred approach to unit test DB code in any language is to access Mongo through a Repository abstraction (there's an example here http://iainjmitchell.com/blog/?p=884). Implementations will vary in terms of DB specific functionality exposed but by removing all the Mongo code from your own logic you're in a position to Unit Test. Simply replace the Mongo Repository implementation with a stubbed out version which is trivially easy. For instance, just store objects in a simple in-memory dictionary collection.
You'll get the benefits of unit testing your own code this way without DB dependencies but you'll still need to do integration tests against the main DB because you'll probably never be able to emulate the idiosyncrasies of the real database as others have said here. The kind of things I've found are as simple as indexing in safe mode vs without safe mode. Specifically, if you have a unique index your dummy memory implementation might honour that in all cases, but Mongo won't without safe-mode.
So whilst you'll still need to test against the DB for some operations, you'll certainly be able to unit test your own logic properly with a stubbed out Repository implementation.
The purpose of mocking is to skip the complexity and unit test own code. If you want to write e2e tests then use the db.
Writing code to setup/teardown a testing DB for unit testing is technical debt and incredibly unsatisfying.
There are mock libraries in npm:
mongo - https://www.npmjs.com/package/mongomock
mongoose - https://www.npmjs.com/package/mockgoose
If those don't support the features you need, then yes you may need to use the real thing.
I had this dilemma and chosen to work with a test DB and clean it every time the test begins. (how to drop everything: https://stackoverflow.com/a/25639377/378594)
With NPM you can even make a test script that creates the db file and cleans it up after.