Transform, generate and serve dynamic content with Vite [closed] - vite

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 months ago.
Improve this question
I was wondering if any of the following is possible to implement using vite build tool.
Consider that I have files in directory matching the pattern: /content/file-[id].md
/content/file-1.md
/content/file-2.md
Every time I serve the SPA app with vite command or building an app with vite build I would like to
grab all the files /content/file-[id].md and transform them into /content_parsed/file-[id].html
/content_parsed/file-1.html
/content_parsed/file-2.html
grab all files /content_parsed/file-[id].html and generated a manifest file /files.manifest containing all paths of files.
/files.manifest
This has to be done automatically in watch mode, when the app is served (vite command) and on-demand when app is built (vite build).
I am pretty sure this is possible to be done with a manual script that I could run with node ./prepareFiles.js && vite, but in this case I am loosing the reactivity when serving the app (i.e. the watch-mode).. so a direct integration into vite would be a step-up in terms of usability and testability (I think).
Given the above use-case - can vite do this? Do I need to write a custom plugin for that? or do you recommend creating a separate watch-files/watch-directory script for that?

I have been able to partially accomplish what I wanted. The only issue right now is the hot reload functionality.
if you import the manifest as
import doc from 'docs.json'
then the page will be auto-reloaded if the module is updated.
On the other had, if you want to dynamically load the data with fetch API:
fetch('docs.json')
.then(r => r.json())
.then(json => {
//...
})
Then the only way to refresh page contents is by manual refresh.. If anyone has a suggestion how to trigger reload from within vite plugin context please let me know.. I will update the post once I figure it out.
Also I should mention that I have decided not to pre-generate the html pages so this functionality is missing from the plugin but could easily be extended with marked, markdown-it remarked etc..
Plugin: generateFilesManifest.ts
import {PluginOption} from "vite";
import fs from "fs";
import path from 'path'
const matter = require('front-matter');
const chokidar = require('chokidar');
import {FSWatcher} from "chokidar";
export type GenerateFilesManifestConfigType = {
watchDirectory: string,
output: string
}
export type MatterOutputType<T> = {
attributes: T,
body: string,
bodyBegin: number,
frontmatter: string,
path: string,
filename: string,
filenameNoExt: string,
}
export default function generateFilesManifest(userConfig: GenerateFilesManifestConfigType): PluginOption {
let config: GenerateFilesManifestConfigType = userConfig
let rootDir: string
let publicDir: string
let command: string
function generateManifest() {
const watchDirFullPath = path.join(rootDir, config.watchDirectory)
const files = fs.readdirSync(watchDirFullPath);
// regenerate manifest
const manifest: any[] = []
files.forEach(fileName => {
const fileFullPath = path.join(watchDirFullPath, fileName)
// get front matter data
const fileContents = fs.readFileSync(fileFullPath).toString()
//const frontMatter = matter.read(fileFullPath)
const frontMatter = matter(fileContents)
//console.log(frontMatter);
// get file path relative to public directory
//const basename = path.basename(__dirname)
const fileRelativePath = path.relative(publicDir, fileFullPath);
const fileInfo = JSON.parse(JSON.stringify(frontMatter)) as MatterOutputType<any>;
fileInfo.path = fileRelativePath
fileInfo.filename = fileName
fileInfo.filenameNoExt = fileName.substring(0, fileName.lastIndexOf('.'));
fileInfo.frontmatter = ''
manifest.push(fileInfo);
});
const outputString = JSON.stringify(manifest, null, 2);
fs.writeFileSync(config.output, outputString, {encoding: 'utf8', flag: 'w'})
console.log('Auto-generated file updated')
}
let watcher: FSWatcher | undefined = undefined;
return {
name: 'generate-files-manifest',
configResolved(resolvedConfig) {
publicDir = resolvedConfig.publicDir
rootDir = resolvedConfig.root
command = resolvedConfig.command
},
buildStart(options: NormalizedInputOptions) {
generateManifest();
if (command === 'serve') {
const watchDirFullPath = path.join(rootDir, config.watchDirectory)
watcher = chokidar.watch(watchDirFullPath,
{
ignoreInitial: true
}
);
watcher
.on('add', function (path) {
//console.log('File', path, 'has been added');
generateManifest();
})
.on('change', function (path) {
//console.log('File', path, 'has been changed');
generateManifest();
})
.on('unlink', function (path) {
//console.log('File', path, 'has been removed');
generateManifest();
})
.on('error', function (error) {
console.error('Error happened', error);
})
}
},
buildEnd(err?: Error) {
console.log('build end')
watcher?.close();
}
}
}
in vite.config.ts, use as
export default defineConfig({
plugins: [
vue(),
generateFilesManifest({
watchDirectory: '/public/docs',
output: './public/docs.json'
})
]
})
you might want to cover such as edge-cases as watch directory not present etc...
front-matter is the library that parses markdown files. Alternative is gray-matter
EDIT: thanks to #flydev response I was able to dig some more examples on page reload functionality. Here's the experimental functionality that you could add:
function generateManifest() {
// ...
ws?.send({type: 'full-reload', path: '*'})
}
let ws: WebSocketServer | undefined = undefined;
return {
name: 'generate-files-manifest',
//...
configureServer(server: ViteDevServer) {
ws = server.ws
}
// ...
}
Currently the whole page is reloaded regardless of the path.. Not sure if there is a way to make it smart enough to just reload pages that loaded the manifest file.. I guess it's currently limited by my own ability to write a better code :)

Related

Evaluate a Cypress configuration file

In my build scripts I need to evaluate a Cypress configuration file. I'm using the following script:
let appdata = process.env.LOCALAPPDATA;
let version = `11.0.1`;
let src = `${appdata}/Cypress/Cache/${version}/Cypress/resources/app/node_modules/#packages/data-context/src`;
const DataContext = require(`${src}/DataContext.js`).DataContext;
const ProjectConfigManager = require(`${src}/data/ProjectConfigManager.js`).ProjectConfigManager;
(async() => {
const ctx = new DataContext({
schema: null,
schemaCloud: null,
modeOptions: "run",
appApi: {},
localSettingsApi: {},
authApi: {
} ,
configApi: {
},
projectApi: {
} ,
electronApi: {
} ,
browserApi: {
},
})
let configManager = new ProjectConfigManager({
ctx,
configFile: 'C:\\work\\sample\\sample.config.ts',
projectRoot: 'C:\\work\\sample',
handlers: [],
hasCypressEnvFile: false,
eventRegistrar: null/*new EventRegistrar()*/,
onError: (error) => {},
onInitialConfigLoaded: () => {},
onFinalConfigLoaded: () => Promise.resolve(),
refreshLifecycle: () => Promise.resolve(),
})
configManager.configFilePath = "sample.config.ts"
configManager.setTestingType('e2e')
let cfg = await configManager.getConfigFileContents()
console.log(JSON.stringify(cfg));
})();
It works well for Cypress 10 version.
However, Cypress 11 has introduced some changes that break this script. Though I adjusted the paths, I'm still unable to make it work again.
It currently fails with this error:
Error: Cannot find module 'C:\Users\mbolotov\AppData\Local\Cypress\Cache\11.0.1\Cypress\resources\app\node_modules\graphql\index'. Please verify that the package.json has a valid "main" entry
How can I fix this problem (without making changes to the Cypress installation)?
OR
Is there any other way to evaluate a Cypress configuration file (say from the command line) and obtain its values?
The exact usage is unclear to me, but making some assumptions - a nodejs script in the /scripts folder of the project can compile and resolve the config using the Cypress module API.
It would need a test to run, a "null-test" can be generated from inside the script.
Note, the null-test must conform to the spec pattern of the project (below it's the std .cy.js)
const cypress = require('cypress')
const fs = require('fs')
fs.writeFileSync('../cypress/e2e/null-test.cy.js', 'it("", ()=>{})')
cypress.run({
project: '..',
spec: '../cypress/e2e/null-test.cy.js',
quiet: true
}).then(results => {
if (results.status === 'failed') {
console.log(results)
} else {
console.log(results.config.resolved) // output resolved config
}
})
I haven't attempted to duplicate everything you have in your code, as it's using Cypress internals and not publicly documented.
This may be because of Changelog 11.0.0
We have also massively improved our startup performance by shipping a snapshot of our binary instead of the source files.
Looking inside the cache folder for v10.11.0 (${process.env.CYPRESS_CACHE_FOLDER}/10.11.0/Cypress/resources/app), the /node_modules is fully populated and /node_modules/graphql/index.js exists.
But in v11.0.1 /node_modules/graphql is not fully populated.

Node js TypeScript: Good way to copy one file content from File object to another File object

I am in Node version 16.5.0 with TypeScript.
Problem Statement
I have a two File Objects - src and destination(Note: not file paths of type string, but file object of type File).
I want to optimally (best way) to copy the src content to destination content. I can use, if needed streaming also.
Tried so far
import fs from 'fs';
import { isUndefined } from 'lodash';
import path from 'path';
export class FileUtil {
public saveLogFile(parentDir: File, instanceId: string, content: File) {
const srcFilePath: string = `${parentDir}/instance_${instanceId}_mediatorLog.tgz`;
const destFilePath: string = `${parentDir}/${content.name}`;
const srcFileWithAbsolutePath = path.resolve(srcFilePath);
const destFileAbsolutePath = path.resolve(destFilePath);
fs.stat(srcFilePath, function(err) {
if (isUndefined(err)) {
fs.copyFile(srcFileWithAbsolutePath, destFileAbsolutePath, (err) => {
if (err) {
throw err;
}
});
}
});
}
}
The solution is glitchy so far. I also don't think it's pretty optimized (though did not yet bench marked yet).
Reaching out to experts if there is a better way in NodeJS+TS world to do it.

Module not found: Can't resolve 'fs' in Next.js application

Unable to identify what's happening in my next.js app. As fs is a default file system module of nodejs. It is giving the error of module not found.
If you use fs, be sure it's only within getInitialProps or getServerSideProps. (anything includes server-side rendering).
You may also need to create a next.config.js file with the following content to get the client bundle to build:
For webpack4
module.exports = {
webpack: (config, { isServer }) => {
// Fixes npm packages that depend on `fs` module
if (!isServer) {
config.node = {
fs: 'empty'
}
}
return config
}
}
For webpack5
module.exports = {
webpack5: true,
webpack: (config) => {
config.resolve.fallback = { fs: false };
return config;
},
};
Note: for other modules such as path, you can add multiple arguments such as
{
fs: false,
path: false
}
I spent hours on this and the solution is also here on Stackoverflow but on different issue -> https://stackoverflow.com/a/67478653/17562602
Hereby I asked for MOD permission to reshare this, since this issue is the first one to show up on Google and probably more and more people stumble would upon the same problem as I am, so I'll try to saved them some sweats
Soo, You need to add this in your next.config.js
module.exports = {
future: {
webpack5: true, // by default, if you customize webpack config, they switch back to version 4.
// Looks like backward compatibility approach.
},
webpack(config) {
config.resolve.fallback = {
...config.resolve.fallback, // if you miss it, all the other options in fallback, specified
// by next.js will be dropped. Doesn't make much sense, but how it is
fs: false, // the solution
};
return config;
},
};
It works for like a charm for me
Minimal reproducible example
A clean minimal example will be beneficial to Webpack beginners since auto splitting based on usage is so mind-blowingly magic.
Working hello world baseline:
pages/index.js
// Client + server code.
export default function IndexPage(props) {
return <div>{props.msg}</div>
}
// Server-only code.
export function getStaticProps() {
return { props: { msg: 'hello world' } }
}
package.json
{
"name": "test",
"version": "1.0.0",
"scripts": {
"dev": "next",
"build": "next build",
"start": "next start"
},
"dependencies": {
"next": "12.0.7",
"react": "17.0.2",
"react-dom": "17.0.2"
}
}
Run with:
npm install
npm run dev
Now let's add a dummy require('fs') to blow things up:
// Client + server code.
export default function IndexPage(props) {
return <div>{props.msg}</div>
}
// Server-only code.
const fs = require('fs')
export function getStaticProps() {
return { props: { msg: 'hello world' } }
}
fails with:
Module not found: Can't resolve 'fs'
which is not too surprising, since there was no way for Next.js to know that that fs was server only, and we wouldn't want it to just ignore random require errors, right? Next.js only knows that for getStaticProps because that's a hardcoded Next.js function name.
OK, so let's inform Next.js by using fs inside getStaticProps, the following works again:
// Client + server code.
export default function IndexPage(props) {
return <div>{props.msg}</div>
}
// Server-only code.
const fs = require('fs')
export function getStaticProps() {
fs
return { props: { msg: 'hello world' } }
}
Mind equals blown. So we understand that any mention of fs inside of the body of getStaticProps, even an useless one like the above, makes Next.js/Webpack understand that it is going to be server-only.
Things would work the same for getServerSideProps and getStaticPaths.
Higher order components (HOCs) have to be in their own files
Now, the way that we factor out IndexPage and getStaticProps across different but similar pages is to use HOCs, which are just functions that return other functions.
HOCs will normally be put outside of pages/ and then required from multiple locations, but when you are about to factor things out to generalize, you might be tempted to put them directly in the pages/ file temporarily, something like:
// Client + server code.
import Link from 'next/link'
export function makeIndexPage(isIndex) {
return (props) => {
return <>
<Link href={isIndex ? '/index' : '/notindex'}>
<a>{isIndex ? 'index' : 'notindex'}</a>
</Link>
<div>{props.fs}</div>
<div>{props.isBlue}</div>
</>
}
}
export default makeIndexPage(true)
// Server-only code.
const fs = require('fs')
export function makeGetStaticProps(isBlue) {
return () => {
return { props: {
fs: Object.keys(fs).join(' '),
isBlue,
} }
}
}
export const getStaticProps = makeGetStaticProps(true)
but if you do this you will be saddened to see:
Module not found: Can't resolve 'fs'
So we understand another thing: the fs usage has to be directly inside the getStaticProps function body, Webpack can't catch it in subfunctions.
The only way to solve this is to have a separate file for the backend-only stuff as in:
pages/index.js
// Client + server code.
import { makeIndexPage } from "../front"
export default makeIndexPage(true)
// Server-only code.
import { makeGetStaticProps } from "../back"
export const getStaticProps = makeGetStaticProps(true)
pages/notindex.js
// Client + server code.
import { makeIndexPage } from "../front"
export default makeIndexPage(false)
// Server-only code.
import { makeGetStaticProps } from "../back"
export const getStaticProps = makeGetStaticProps(false)
front.js
// Client + server code.
import Link from 'next/link'
export function makeIndexPage(isIndex) {
return (props) => {
console.error('page');
return <>
<Link href={isIndex ? '/notindex' : '/'}>
<a>{isIndex ? 'notindex' : 'index'}</a>
</Link>
<div>{props.fs}</div>
<div>{props.isBlue}</div>
</>
}
}
back.js
// Server-only code.
const fs = require('fs')
export function makeGetStaticProps(isBlue) {
return () => {
return { props: {
fs: Object.keys(fs).join(' '),
isBlue,
} }
}
}
Webpack must see that name makeGetStaticProps getting assigned to getStaticProps, so it decides that the entire back file is server-only.
Note that it does not work if you try to merge back.js and front.js into a single file, probably because when you do export default makeIndexPage(true) webpack necessarily tries to pull the entire front.js file into the frontend, which includes the fs, so it fails.
This leads to a natural (and basically almost mandatory) split of library files between:
front.js and front/*: front-end + backend files. These are safe for the frontend. And the backend can do whatever the frontend can do (we are doing SSR right?) so those are also usable from the backend.
Perhaps this is the idea behind the conventional "components" folder in many official examples. But that is a bad name, because that folder should not only contain components, but also any library non-component helpers/constants that will be used from the frontend.
back.js and back/* (or alternatively anything outside of front/*): backend only files. These can only be used by the backend, importing them on frontend will lead to the error
fs,path or other node native modules can be used only inside server-side code, like "getServerSide" functions. If you try to use it in client you get error even you just console.log it.. That console.log should run inside server-side functions as well.
When you import "fs" and use it in server-side, next.js is clever enough to see that you use it in server-side so it wont add that import into the client bundle
One of the packages that I used was giving me this error, I fixed this with
module.exports = {
webpack: (config, { isServer }) => {
if (!isServer) {
config.resolve.fallback.fs = false
}
return config
},
}
but this was throwing warning on terminal:
"Critical dependency: require function is used in a way in which
dependencies cannot be statically extracted"
Then I tried to load the node module on the browser. I copied the "min.js" of the node module from the node_modules and placed in "public/js/myPackage.js" and load it with Script
export default function BaseLayout({children}) {
return (
<>
<Script
// this in public folder
src="/js/myPackage.js"
// this means this script will be loaded first
strategy="beforeInteractive"
/>
</>
)
}
This package was attached to window object and in node_modules source code's index.js:
if (typeof window !== "undefined") {
window.TruffleContract = contract;
}
So I could access to this script as window.TruffleContract. BUt this was not an efficient way.
While this error requires a bit more reasoning than most errors you'll encounter, it happens for a straightforward reason.
Why this happens
Next.js, unlike many frameworks allows you to import server-only (Node.js APIs that don't work in a browser) code into your page files. When Next.js builds your project, it removes server only code from your client-side bundle by checking which code exists inside one any of the following built-in methods (code splitting):
getServerSideProps
getStaticProps
getStaticPaths
Side note: there is a demo app that visualizes how this works.
The Module not found: can't resolve 'xyz' error happens when you try to use server only code outside of these methods.
Error example 1 - basic
To reproduce this error, let's start with a working simple Next.js page file.
WORKING file
/** THIS FILE WORKS FINE! */
import type { GetServerSideProps } from "next";
import fs from "fs"; // our server-only import
type Props = {
doesFileExist: boolean;
};
export const getServerSideProps: GetServerSideProps = async () => {
const fileExists = fs.existsSync("/some-file");
return {
props: {
doesFileExist: fileExists,
},
};
};
const ExamplePage = ({ doesFileExist }: Props) => {
return <div>File exists?: {doesFileExist ? "Yes" : "No"}</div>;
};
export default ExamplePage;
Now, let's reproduce the error by moving our fs.existsSync method outside of getServerSideProps. The difference is subtle, but the code below will throw our dreaded Module not found error.
ERROR file
import type { GetServerSideProps } from "next";
import fs from "fs";
type Props = {
doesFileExist: boolean;
};
/** ERROR!! - Module not found: can't resolve 'fs' */
const fileExists = fs.existsSync("/some-file");
export const getServerSideProps: GetServerSideProps = async () => {
return {
props: {
doesFileExist: fileExists,
},
};
};
const ExamplePage = ({ doesFileExist }: Props) => {
return <div>File exists?: {doesFileExist ? "Yes" : "No"}</div>;
};
export default ExamplePage;
Error example 2 - realistic
The most common (and confusing) occurrence of this error happens when you are using modules that contain multiple types of code (client-side + server-side).
Let's say I have the following module called file-utils.ts:
import fs from 'fs'
// This code only works server-side
export function getFileExistence(filepath: string) {
return fs.existsSync(filepath)
}
// This code works fine on both the server AND the client
export function formatResult(fileExistsResult: boolean) {
return fileExistsResult ? 'Yes, file exists' : 'No, file does not exist'
}
In this module, we have one server-only method and one "shared" method that in theory should work client-side (but as we'll see, theory isn't perfect).
Now, let's try incorporating this into our Next.js page file.
/** ERROR!! */
import type { GetServerSideProps } from "next";
import { getFileExistence, formatResult } from './file-utils.ts'
type Props = {
doesFileExist: boolean;
};
export const getServerSideProps: GetServerSideProps = async () => {
return {
props: {
doesFileExist: getFileExistence('/some-file')
},
};
};
const ExamplePage = ({ doesFileExist }: Props) => {
// ERROR!!!
return <div>File exists?: {formatResult(doesFileExist)}</div>;
};
export default ExamplePage;
As you can see, we get an error here because when we attempt to use formatResult client-side, our module still has to import the server-side code.
To fix this, we need to split our modules up into two categories:
Server only
Shared code (client or server)
// file-utils.ts
import fs from 'fs'
// This code (and entire file) only works server-side
export function getFileExistence(filepath: string) {
return fs.existsSync(filepath)
}
// file-format-utils.ts
// This code works fine on both the server AND the client
export function formatResult(fileExistsResult: boolean) {
return fileExistsResult ? 'Yes, file exists' : 'No, file does not exist'
}
Now, we can create a WORKING page file:
/** WORKING! */
import type { GetServerSideProps } from "next";
import { getFileExistence } from './file-utils.ts' // server only
import { formatResult } from './file-format-utils.ts' // shared
type Props = {
doesFileExist: boolean;
};
export const getServerSideProps: GetServerSideProps = async () => {
return {
props: {
doesFileExist: getFileExistence('/some-file')
},
};
};
const ExamplePage = ({ doesFileExist }: Props) => {
return <div>File exists?: {formatResult(doesFileExist)}</div>;
};
export default ExamplePage;
Solutions
There are 2 ways to solve this:
The "correct" way
The "just get it working" way
The "Correct" way
The best way to solve this error is to make sure that you understand why it is happening (above) and make sure you are only using server-side code inside getStaticPaths, getStaticProps, or getServerSideProps and NOWHERE else.
And remember, if you import a module that contains both server-side and client-side code, you cannot use any of the imports from that module client-side (revisit example #2 above).
The "Just get it working" way
As others have suggested, you can alter your next.config.js to ignore certain modules at build-time. This means that when Next.js attempts to split your page file between server only and shared code, it will not try to polyfill Node.js APIs that fail to build client-side.
In this case, you just need:
/** next.config.js - with Webpack v5.x */
module.exports = {
... other settings ...
webpack: (config, { isServer }) => {
// If client-side, don't polyfill `fs`
if (!isServer) {
config.resolve.fallback = {
fs: false,
};
}
return config;
},
};
Drawbacks of this approach
As shown in the resolve.fallback section of the Webpack documentation, the primary reason for this config option is because as-of Webpack v5.x, core Node.js modules are no longer polyfilled by default. Therefore, the main purpose for this option is to provide a way for you to define which polyfill you want to use.
When you pass false as an option, this means, "do not include a polyfill".
While this works, it can be fragile and require ongoing maintenance to include any new modules that you introduce to your project. Unless you are converting an existing project / supporting legacy code, it is best to go for option #1 above as it promotes better module organization according to how Next.js actually splits the code under the hood.
If trying to use fs-extra in Next.js, this worked for me
module.exports = {
webpack: (config) => {
config.resolve.fallback = { fs: false, path: false, stream: false, constants: false };
return config;
}
}
I got this error in my NextJS app because I was missing export in
export function getStaticProps()
/** #type {import('next').NextConfig} */
module.exports = {
reactStrictMode: false,
webpack5: true,
webpack: (config) => {
config.resolve.fallback = {
fs: false,
net: false,
dns: false,
child_process: false,
tls: false,
};
return config;
},
};
This code fixed my problem and I want to share.Add this code to your next.config file.i'm using
webpack5
For me clearing the cache
npm cache clean -f
and then updating the node version to the latest stable release(14.17.0) worked
It might be that the module you are trying to implement is not supposed to run in a browser. I.e. it's server-side only.
For me, the problem was the old version of the node.js installed. It requires node.js version 14 and higher. The solution was to go to the node.js web page, download the latest version and just install it. And then re-run the project. All worked!
I had the same issue when I was trying to use babel.
For me this worked:
#add a .babelrc file to the root of the project and define presets and plugins
(in my case, I had some issues with the macros of babel, so I defined them)
{
"presets": ["next/babel"],
"plugins": ["macros"]
}
after that shut down your server and run it again
I had this exact issue. My problem was that I was importing types that I had declared in a types.d.ts file.
I was importing it like this, thanks to the autofill provided by VSCode.
import {CUSTOM_TYPE} from './types'
It should have been like this:
import {CUSTOM_TYPE} from './types.d'
In my case, I think the .d was unnecessary so I ended up removing it entirely and renamed my file to types.ts.
Weird enough, it was being imported directly into index.tsx without issues, but any helper files/functions inside the src directory would give me errors.
I ran into this in a NextJS application because I had defined a new helper function directly below getServerSideProps(), but had not yet called that function inside getServerSideProps().
I'm not sure why this created a problem, but it did. I could only get it to work by either calling that function, removing it, or commenting it out.
Don't use fs in the pages directory, since next.js suppose that files in pages directory are running in browser environment.
You could put the util file which uses fs to other directory such as /core
Then require the util in getStaticProps which runs in node.js environment.
// /pages/myPage/index.tsx
import View from './view';
export default View;
export async function getStaticProps() {
const util = require('core/some-util-uses-fs').default; // getStaticProps runs in nodes
const data = await util.getDataFromDisk();
return {
props: {
data,
},
};
}
In my case, this error appeared while refactoring the auth flow of a Next.js page. The cause was some an unused imports that I had not yet removed.
Previously I made the page a protected route like so:
export async function getServerSideProps ({ query, req, res }) {
const session = await unstable_getServerSession(req, res, authOptions)
if (!session) {
return {
redirect: {
destination: '/signin',
permanent: false,
},
}
}
//... rest of server-side logic
}
Whilst refactoring, I read up on NextAuth useSession. Based on what I read there, I was able to change the implementation such that I simply needed to add
MyComponent.auth = true to make a page protected. I then deleted the aforementioned code block inside of getServerSideProps. However, I had not yet deleted the two imports used by said code block:
import { unstable_getServerSession } from 'next-auth/next'
import { authOptions } from 'pages/api/auth/[...nextauth]'
I believe the second of those two imports was causing the problem. So the summary is that in addition to all of the great answers above, it could also be an unused import.
Sometimes this error can be because you have imported something but not mastered it anywhere. This worked for me. I reviewed my code and removed the unused dependencies.

Exclude SVG from gulp-svg-sprite build process

I'm currently building an SVG Icon system using gulp-svg-sprite and have run into a situation where I need to exclude some icons from the build process.
Is there a way to EXCLUDE an SVG from running through these 2 pipes? Somehow I need to get the src filename and compare it to the SVG I want to exclude and so something like:
if src != svgToExclude then run the pipes.
I don't want specific icons being optimized via SVGO and other plugins for those 1-off SVG's that require 2 styleable paths.
Here is the code I'm working with:
const gulp = require('gulp');
const svgo = require('gulp-svgo');
const rsp = require('remove-svg-properties').stream;
const dom = require('gulp-dom');
const xmlEdit = require('gulp-edit-xml');
const gulpIf = require('gulp-if');
const gulpIgnore = require('gulp-ignore');
const { toPath } = require('svg-points');
var excludeIcon = './utilities/checkbox-checked/checkbox-checked--s.svg';
const svgBuild = src => {
return gulp
.src(src)
.pipe(
rsp.remove({
properties: ['fill', rsp.PROPS_STROKE],
log: false,
})
)
.pipe(
svgo({
js2svg: {
indent: 2,
pretty: true,
},
plugins: [{ removeTitle: true }],
})
)
};
module.exports = svgBuild;
I'm new to gulp & node so any help would be appreciated!
Thanks,
- Ryan
Using gulp-filter:
const filter = require('gulp-filter');
something like
var excludeIconArray = ["iconToExclude1.svg", "iconToExclude2.svg", etc.]
const svgBuild = src => {
const svgFilter = filter(file => {
// will probably need file.path string manipulations here
// file.path is the full path but you can select portions of it
// so the below is just pseudocode
return !excludeIconArray.includes(file.path)
});
return gulp
.src(src)
// put the next pipe wherever you want to exclude certain files
// either right after source or just before the svgo pipe
.pipe(svgFilter())

node.js require all files in a folder?

How do I require all files in a folder in node.js?
need something like:
files.forEach(function (v,k){
// require routes
require('./routes/'+v);
}};
When require is given the path of a folder, it'll look for an index.js file in that folder; if there is one, it uses that, and if there isn't, it fails.
It would probably make most sense (if you have control over the folder) to create an index.js file and then assign all the "modules" and then simply require that.
yourfile.js
var routes = require("./routes");
index.js
exports.something = require("./routes/something.js");
exports.others = require("./routes/others.js");
If you don't know the filenames you should write some kind of loader.
Working example of a loader:
var normalizedPath = require("path").join(__dirname, "routes");
require("fs").readdirSync(normalizedPath).forEach(function(file) {
require("./routes/" + file);
});
// Continue application logic here
I recommend using glob to accomplish that task.
var glob = require( 'glob' )
, path = require( 'path' );
glob.sync( './routes/**/*.js' ).forEach( function( file ) {
require( path.resolve( file ) );
});
Base on #tbranyen's solution, I create an index.js file that load arbitrary javascripts under current folder as part of the exports.
// Load `*.js` under current directory as properties
// i.e., `User.js` will become `exports['User']` or `exports.User`
require('fs').readdirSync(__dirname + '/').forEach(function(file) {
if (file.match(/\.js$/) !== null && file !== 'index.js') {
var name = file.replace('.js', '');
exports[name] = require('./' + file);
}
});
Then you can require this directory from any where else.
Another option is to use the package require-dir which let's you do the following. It supports recursion as well.
var requireDir = require('require-dir');
var dir = requireDir('./path/to/dir');
I have a folder /fields full of files with a single class each, ex:
fields/Text.js -> Test class
fields/Checkbox.js -> Checkbox class
Drop this in fields/index.js to export each class:
var collectExports, fs, path,
__hasProp = {}.hasOwnProperty;
fs = require('fs');
path = require('path');
collectExports = function(file) {
var func, include, _results;
if (path.extname(file) === '.js' && file !== 'index.js') {
include = require('./' + file);
_results = [];
for (func in include) {
if (!__hasProp.call(include, func)) continue;
_results.push(exports[func] = include[func]);
}
return _results;
}
};
fs.readdirSync('./fields/').forEach(collectExports);
This makes the modules act more like they would in Python:
var text = new Fields.Text()
var checkbox = new Fields.Checkbox()
One more option is require-dir-all combining features from most popular packages.
Most popular require-dir does not have options to filter the files/dirs and does not have map function (see below), but uses small trick to find module's current path.
Second by popularity require-all has regexp filtering and preprocessing, but lacks relative path, so you need to use __dirname (this has pros and contras) like:
var libs = require('require-all')(__dirname + '/lib');
Mentioned here require-index is quite minimalistic.
With map you may do some preprocessing, like create objects and pass config values (assuming modules below exports constructors):
// Store config for each module in config object properties
// with property names corresponding to module names
var config = {
module1: { value: 'config1' },
module2: { value: 'config2' }
};
// Require all files in modules subdirectory
var modules = require('require-dir-all')(
'modules', // Directory to require
{ // Options
// function to be post-processed over exported object for each require'd module
map: function(reqModule) {
// create new object with corresponding config passed to constructor
reqModule.exports = new reqModule.exports( config[reqModule.name] );
}
}
);
// Now `modules` object holds not exported constructors,
// but objects constructed using values provided in `config`.
I know this question is 5+ years old, and the given answers are good, but I wanted something a bit more powerful for express, so i created the express-map2 package for npm. I was going to name it simply express-map, however the people at yahoo already have a package with that name, so i had to rename my package.
1. basic usage:
app.js (or whatever you call it)
var app = require('express'); // 1. include express
app.set('controllers',__dirname+'/controllers/');// 2. set path to your controllers.
require('express-map2')(app); // 3. patch map() into express
app.map({
'GET /':'test',
'GET /foo':'middleware.foo,test',
'GET /bar':'middleware.bar,test'// seperate your handlers with a comma.
});
controller usage:
//single function
module.exports = function(req,res){
};
//export an object with multiple functions.
module.exports = {
foo: function(req,res){
},
bar: function(req,res){
}
};
2. advanced usage, with prefixes:
app.map('/api/v1/books',{
'GET /': 'books.list', // GET /api/v1/books
'GET /:id': 'books.loadOne', // GET /api/v1/books/5
'DELETE /:id': 'books.delete', // DELETE /api/v1/books/5
'PUT /:id': 'books.update', // PUT /api/v1/books/5
'POST /': 'books.create' // POST /api/v1/books
});
As you can see, this saves a ton of time and makes the routing of your application dead simple to write, maintain, and understand. it supports all of the http verbs that express supports, as well as the special .all() method.
npm package: https://www.npmjs.com/package/express-map2
github repo: https://github.com/r3wt/express-map
Expanding on this glob solution. Do this if you want to import all modules from a directory into index.js and then import that index.js in another part of the application. Note that template literals aren't supported by the highlighting engine used by stackoverflow so the code might look strange here.
const glob = require("glob");
let allOfThem = {};
glob.sync(`${__dirname}/*.js`).forEach((file) => {
/* see note about this in example below */
allOfThem = { ...allOfThem, ...require(file) };
});
module.exports = allOfThem;
Full Example
Directory structure
globExample/example.js
globExample/foobars/index.js
globExample/foobars/unexpected.js
globExample/foobars/barit.js
globExample/foobars/fooit.js
globExample/example.js
const { foo, bar, keepit } = require('./foobars/index');
const longStyle = require('./foobars/index');
console.log(foo()); // foo ran
console.log(bar()); // bar ran
console.log(keepit()); // keepit ran unexpected
console.log(longStyle.foo()); // foo ran
console.log(longStyle.bar()); // bar ran
console.log(longStyle.keepit()); // keepit ran unexpected
globExample/foobars/index.js
const glob = require("glob");
/*
Note the following style also works with multiple exports per file (barit.js example)
but will overwrite if you have 2 exports with the same
name (unexpected.js and barit.js have a keepit function) in the files being imported. As a result, this method is best used when
your exporting one module per file and use the filename to easily identify what is in it.
Also Note: This ignores itself (index.js) by default to prevent infinite loop.
*/
let allOfThem = {};
glob.sync(`${__dirname}/*.js`).forEach((file) => {
allOfThem = { ...allOfThem, ...require(file) };
});
module.exports = allOfThem;
globExample/foobars/unexpected.js
exports.keepit = () => 'keepit ran unexpected';
globExample/foobars/barit.js
exports.bar = () => 'bar run';
exports.keepit = () => 'keepit ran';
globExample/foobars/fooit.js
exports.foo = () => 'foo ran';
From inside project with glob installed, run node example.js
$ node example.js
foo ran
bar run
keepit ran unexpected
foo ran
bar run
keepit ran unexpected
One module that I have been using for this exact use case is require-all.
It recursively requires all files in a given directory and its sub directories as long they don't match the excludeDirs property.
It also allows specifying a file filter and how to derive the keys of the returned hash from the filenames.
Require all files from routes folder and apply as middleware. No external modules needed.
// require
const { readdirSync } = require("fs");
// apply as middleware
readdirSync("./routes").map((r) => app.use("/api", require("./routes/" + r)));
I'm using node modules copy-to module to create a single file to require all the files in our NodeJS-based system.
The code for our utility file looks like this:
/**
* Module dependencies.
*/
var copy = require('copy-to');
copy(require('./module1'))
.and(require('./module2'))
.and(require('./module3'))
.to(module.exports);
In all of the files, most functions are written as exports, like so:
exports.function1 = function () { // function contents };
exports.function2 = function () { // function contents };
exports.function3 = function () { // function contents };
So, then to use any function from a file, you just call:
var utility = require('./utility');
var response = utility.function2(); // or whatever the name of the function is
Can use : https://www.npmjs.com/package/require-file-directory
Require selected files with name only or all files.
No need of absoulute path.
Easy to understand and use.
Using this function you can require a whole dir.
const GetAllModules = ( dirname ) => {
if ( dirname ) {
let dirItems = require( "fs" ).readdirSync( dirname );
return dirItems.reduce( ( acc, value, index ) => {
if ( PATH.extname( value ) == ".js" && value.toLowerCase() != "index.js" ) {
let moduleName = value.replace( /.js/g, '' );
acc[ moduleName ] = require( `${dirname}/${moduleName}` );
}
return acc;
}, {} );
}
}
// calling this function.
let dirModules = GetAllModules(__dirname);
Create an index.js file in your folder with this code :
const fs = require('fs')
const files = fs.readdirSync('./routes')
for (const file of files) {
require('./'+file)
}
And after that you can simply load all the folder with require("./routes")
If you include all files of *.js in directory example ("app/lib/*.js"):
In directory app/lib
example.js:
module.exports = function (example) { }
example-2.js:
module.exports = function (example2) { }
In directory app create index.js
index.js:
module.exports = require('./app/lib');

Resources