What is best folder structure for a project that includes:
2 database (mongodb, influxdb)
socket io
...
It all depends on the personal preferences. Make your folder structure the way you are most comfortable working with.
For example, I love my code to be split in submodules or simply said to be in different directories, accordingly to the job the code is used for.
I'd personally go with a couple of folders:
.{src}
├── controllers # All controller operations according the routes are stored here
│ ├── authController.ts # Handles authentication requests
│ ├── usersController.ts # Handles users route requests
│ └── ...
│
├── database # All database connections are stored here. For example you have two databases
│ ├── db.ts # Initialize DB connection
│ └── ...
│
├── middleware
│ ├── authenticated.ts # Decode and verify JWT token
│ ├── error.ts # Common Error Handler
│ ├── logger.ts # Control logging levels
│ └── ...
│
├── models # Simple descriptor of the database tables
│ ├── usersModel.ts # DB model for users
│ └── ...
│
├── schema # Schemas that are used for CRUD operations with the models
│ ├── users.ts # DB Schema for users
│ └── ...
│
├── listeners
│ ├── socketsManager.ts # Socket listeners/emitters handle
│ └── ...
│
├── app.ts # Entry file for the project
├── .env # Store environment variables
├── routes.ts # All routes initializer
└── ...
Although, you might not like the naming or the order I've put on, you can create your own one you feel comfortable with. Another thing you might do is take a look at some conventions that are up on the internet or some of the most popular projects like frameworks.
It's all depending on your needs and what you feel comfortable working with, after all.
Related
I currently have data stored as csv files in an s3 bucket. The structure of the data is as follows
s3_bucket/
├── dir_1/
│ ├── part_0000.csv
│ ├── part_0001.csv
│ ├── part_0002.csv
│ └── ...
├── dir_2/
│ ├── part_0000.csv
│ ├── part_0001.csv
│ ├── part_0002.csv
│ └── ...
├── dir_3/
│ └── ...
├── dir_4/
└── ...
I want to write some kind of Spark job to go into each subdirectory dir_n/ and merge all the data into a single file, resulting in the following structure
s3_bucket/
├── dir_1/
│ └── merged.csv
├── dir_2/
│ └── merged.csv
├── dir_3/
│ └── merged.csv
├── dir_4/
└── ...
I was thinking of somehow spawning multiple workers to crawl each subdirectory, read the data into memory, and merge them using repartition(1), however I am not to sure of how to do this.
Any help is greatly appreciated!
You can just loop the directories using hadoop, and then read each directory and do a coalesce(1), here's the code in scala:
import org.apache.spark.sql._
import org.apache.hadoop.fs.Path
val path = "/path/s3_bucket/"
val hdfs = new Path(path).getFileSystem(spark.sparkContext.hadoopConfiguration)
hdfs.listStatus(new Path(path)).foreach(file => if (file.isDirectory)
spark.read.csv(path + file.getPath.getName).coalesce(1)
.write.mode(SaveMode.Overwrite).csv(path + file.getPath.getName)
)
I'm working on a terragrunt code base for the first time, having used terraform a lot in the past without terragrunt. I'm a bit confused as to the structure terragrunt seems to enforce. I would usually organise my terraform thus:
main.tf
--> module
main.tf
--> module2
main.tf
This is listed as best practice on the terraform docs:
The Root Module
Terraform always runs in the context of a single root module. A
complete Terraform configuration consists of a root module and the
tree of child modules (which includes the modules called by the root
module, any modules called by those modules, etc.).
Source
But none of the terragrunt structures seem to represent this. It seems to be designed so that each module is independent and run using the run-all command.
This seems problematic to me, from the existing code base I can see that this initialises terraform for every module and I'd say causes issues with sharing secrets between modules. So I'd prefer to work with one root module and multiple child modules.
I can't find a terragrunt pattern that will allow me to do this?
I'm also confused as to how this responsibility is decomposed, do I actually structure my terraform (as above) or do I need an extra root .hcl file?
I'm after something a little like this I guess
└── live
├── prod
│ ├── terragrunt.hcl
│ ├── app
│ │ └── terragrunt.hcl
│ ├── mysql
│ │ └── terragrunt.hcl
│ └── vpc
│ └── terragrunt.hcl
├── qa
│ ├── terragrunt.hcl
│ ├── app
│ │ └── terragrunt.hcl
│ ├── mysql
│ │ └── terragrunt.hcl
│ └── vpc
│ └── terragrunt.hcl
└── stage
├── terragrunt.hcl
├── app
│ └── terragrunt.hcl
├── mysql
│ └── terragrunt.hcl
└── vpc
└── terragrunt.hcl
But this example just talks about specifying the provider block and nothing about a root main.tf. So I'm lost?
Each TF module you used to use consumed inputs, created resources and provided outputs. The wiring of the modules was done via main.tf you are referring to in plain Terraform.
In your case with Terragrunt now, the wiring will be done by terragrunt.hcl files. On the module level (e.g.live/prod/app/terragrunt.hcl) you could define the module dependencies, i.e. where are the input values for this modules input variables, e.g.:
inputs {
username = dependency.iam.output.user.name
}
With this in mind, you might or might not use root-level terragrunt.hcl files. If you want to invoke the parent-folder terragrunt.hcl code, you need to add the following block into your module:
include "root" {
path = find_in_parent_folders()
expose = true
}
See the docs for this function here.
You can create controller for each route file to better manage your code. The routes have it's own folder in the root directory of a express project but where does the controllers belong?
I have seen that they have an own folder named "controllers" in the root directory too but I have also seen that each router has his own folder where router.js and index.js (the controller) are.
But what is the right folder structure for controllers for express js projects?
Ohk, just don't get too much confused. Organizing the files and folders totally depends on you, Express or node doesn't force you to maintain specific structure. In general developers like to keep things like below:
.
├── config # App configuration files
│ ├── sequalize.json # Sequalize config
│ ├── serviceOne.json # ServiceOne config
│ └── ... # Other configurations
├── routes
│ ├── controllers # Request managers
│ ├── middlewares # Request middlewares
│ └── routes.js # Define routes and middlewares here
├── services # External services implementation
│ ├── serviceOne
│ └── serviceTwo
│ └── ... # Other services
├── db # Data access stuff (Sequalize mostly)
│ ├── models # Models
│ ├── migrations # Migrations
│ ├── seeds # Seeds
│ └── index.js # Sequalize instantiation
├── core # Business logic implementation
│ ├── accounts.js
│ ├── sales.js
│ ├── comments.js
│ └── ... # Other business logic implementations
├── utils # Util libs (formats, validation, etc)
├── tests # Testing
├── scripts # Standalone scripts for dev uses
├── pm2.js # pm2 init
├── shipitfile.js # deployment automation file
├── package.json
├── README.md
└── app.js # App starting point
For more details you can refer https://medium.com/codebase/structure-of-a-nodejs-api-project-cdecb46ef3f8
If you have an exigence to implement your server-side project with Express js follow the answer below , or I suggest for you Nest.js
Nest (NestJS) is a framework for building efficient, scalable Node.js™ server-side applications
It has his own structure (services, controllers, routers ) well organized, and implementing good practices for server-side projects, take a look
It all depends on what you want, express don't have a "you need to use this format or else your api will surely fail" rule.
If you have an experience with using a PHP framework before, you can follow their MVC project structure if you want to or you can follow this nodejs best practices.
I have been working on a project for a url shortener, where I am trying to increment views every time someone hit that api or someone make a get request. I don't know what is getting wrong but the views are just incrementing for once only.
Here's the code I wrote to increment the views every time a get request is made.
Get - link
Incrementing in mongodb - link
I do not know what is going wrong, I have already searched through lot of forums, stack overflow, articles already asked problems.
Note -
The default value of views is 0 , here is the value in post request.
Distinct users, generated by are dummy values as of now.
Mongoose Schema Link - link
I am creating this project with nodejs, expressjs, mongodb(database), mongoose(orm), pugjs(view engine).
Project Link : link
Here's the file tree
├── LICENSE
├── README.md
├── client
│ ├── assets
│ │ ├── bg.svg
│ │ ├── favicon
│ │ │ └── favicon.ico
│ │ └── fonts
│ │ ├── Apercu\ Medium.woff
│ │ ├── Apercu\ Mono.woff
│ │ └── Apercu_Regular.woff
│ ├── css
│ │ └── style.css
│ ├── index.html
│ └── js
│ └── script.js
├── index.js
├── models
│ ├── admin_model.js
│ └── urlshorten.js
├── package-lock.json
├── package.json
├── routes
│ ├── admin.js
│ ├── auth.js
│ ├── custom.js
│ ├── stats.js
│ └── urlShorten.js
├── static
│ ├── css
│ │ └── style.css
│ ├── favicon.ico
│ ├── fonts
│ │ └── Inter-Regular.woff
│ └── urlshort.gif
└── views
├── index.pug
└── script.js
In simple terms:
1. It is because of 301 Status Code for Redirect you are using.
2. 301 Status Code represents a Permanent Redirect.
3. The browser stores the main URL in the memory.
4. The next time you request the short link from your browser, it gets the Original
URL from its memory and sends you there, without going through the Server.
5. Since the request doesn't go through the server, the server fails to increment it.
Solution:
Change 301 (Permanent) Status code to 307 (Temporary) Status Code.
FILE: ShortLink/routes/urlShortner.js
Change the below lines
Line 37: res.writeHead(301, {
Line 38: Location: url.inputUrl
Line 39: });
to
Line 37: res.writeHead(307, {
Line 38: Location: url.inputUrl
Line 39: });
I have also created a pull request to your Github Repo which you can verify.
I am using Terraform to deploy an 3-tiers application on Azure: Frontend, Backend, and DB. It works well.
The application is a product, and I need several instances for several customers.
My problem is, I have a hard time understanding applying several sets of values (variables.tf) to the same script, in order to get several environments.
I would like to have the following structure:
main.tf
customer1
variables.tf
customer2
variables.tf
And select whether I deploy customer1 or customer2. I read about terraform workspaces, and started creating one workspace per customer. But I don't understand how to apply a different set of values for the same scripts depending on the working workspace. It's difficult to find a comprehensive example online, what I am trying to do should be the bread and butter of terraform.
Thanks!
As both of you and #ydaetskcoR knew, workspace is one choice. If it is not suitable for your business. There is another way for you, and I would recommend it for your use case.
Terragrunt is a thin wrapper for Terraform that provides extra tools for working with multiple Terraform modules.
So in your case, it can be easily used to manage different customers as below structure.
└── customer1
├── prod
│ ├── app
│ │ └── terraform.tfvars
│ ├── mysql
│ │ └── terraform.tfvars
│ └── vpc
│ └── terraform.tfvars
└── stage
├── app
│ └── terraform.tfvars
├── mysql
│ └── terraform.tfvars
└── vpc
└── terraform.tfvars
└── customer2
├── prod
│ ├── app
│ │ └── terraform.tfvars
│ ├── mysql
│ │ └── terraform.tfvars
│ └── vpc
│ └── terraform.tfvars
└── stage
├── app
│ └── terraform.tfvars
├── mysql
│ └── terraform.tfvars
└── vpc
└── terraform.tfvars