I am trying to build a website where a only 2 pages will be updated and 1 will be created ~daily. In my case, that would imply getting the new page to be created via an API route in a NodeJs backend, and the updated content also would come from an API, but to update the redux state.
All other pages would stay completely the same. The problem is, if I build from next.js, this build time would increment daily and this is not a good option.
Is there a way to build only the differences / force some pages to stay the same?
try the code below. for more check this link: Incremental Static Regeneration
export async function getStaticProps() {
const res = await fetch('https://.../newPost')
const newPost = await res.json()
return {
props: {
newPost,
},
// Next.js will attempt to re-generate the page:
// - When a request comes in
// - At most once every day
revalidate: 86400, // seconds in a day
}
}
Related
I'm very new to Heroku and node so have a basic question just about how to implement background processes in a graphql server app I have hosted on Heroku.
I have a working graphql server written in Keystone CMS and hosted on Heroku.
In the database I have a schema called `Item` which basically just takes a URL from the user and then tries to scrape a Hero Image from that URL.
As the URL can be anything, I'm trying to use a headless browser via Playwright in order to get images
This is a memory intensive process though and Heroku is OOM'ing with R14 errors. For this they recommend migrating intensive work like this to a Background Job via Redis, implemented in Bull and Throng
I've never used redis before nor these other libraries so I'm out of my element. I've looked at the Heroku implementation examples "server" and "worker" but haven't been able to translate those into a working implementation. To be honest I just don't understand the flow and design pattern I'm supposed to use with those even after reading the docs and examples.
Here is my code:
Relevant CMS schema where I call the getImageFromURL() function which is memory intensive
# Item.ts
import getImageFromURL from '../lib/imageFromURL'
export const Item = list({
...
fields: {
url: text({
validation: { isRequired: false },
}),
imageURL: text({
validation: { isRequired: false },
}),
....
},
hooks: {
resolveInput: async ({ resolvedData }) => {
if (resolvedData.url) {
const imageURL: string | undefined = await getImageFromURL(
// pass the user-provided url to the image scraper
resolvedData.url
)
if (imageURL) {
return {
...resolvedData,
// if we scraped successfully, return URL to image asset
imageURL,
}
}
return resolvedData
}
return resolvedData
},
}
Image scraping function getImageFromURL() (where I believe the background job needs to go?)
filtered to relevant parts
# imageFromURL.ts
// set up redis for processing
const Queue = require('bull')
const throng = require('throng')
const REDIS_URL = process.env.REDIS_URL || 'redis://127.0.0.1:6379'
let workers = 2
async function scrapeURL(urlString){
...
scrape images with playwright here
...
return url to image asset here
}
// HERE IS WHERE I'M STUCK
// How do I do `scrapeURL` in a background process?
export default async function getImageFromURL(
urlString: string
): Promise<string | undefined> {
let workQueue = new Queue('scrape_and_uppload', REDIS_URL)
// Something like this?
// const imageURL = await scrapeURL(urlString) ??
// Or this?
// This fails with:
// "TypeError: handler.bind is not a function"
// but I'm just lost as to how this should even work
// workQueue.process(2, scrapeURL(urlString))
return Promise.resolve(imageURL)
}
Then when testing I call this with throng((url) => getImageFromURL(url), { workers }).
I have my local redis db running but I'm not even seeing any log spew when I run this so I don't think I'm even successfully connecting to redis?
Thanks in advance let me know where I'm unclear or can add code examples
Recently, I have been working on a personal project involving the creation of some API endpoints using NextJs and TypeScript that call back on the Discord API using discord.js. Please don't get scared off at the mention of the discord API if you have never touched it, I don't think that library is the issue, hence why it is not included in the thread title.
Problem:
I have implemented a singleton for the discord.js API client as the client can take about a second or two to login and initialize, time I don't want to add to each response. This works great on one file/endpoint, where once that file has the instance, it keeps its. However, as soon as I load another file/endpoint, it creates another instance of the singleton, however, after its creation works fine again within that file.
My problem is that I dont want an instance per file, but instead want one instance for the entire application.
DiscordClient.ts:
import { Client } from 'discord.js';
class DiscordClient {
private static discordClient: DiscordClient;
public APIClient: Client;
private constructor() {
this.APIClient = new Client();
this.APIClient.login($TOKEN);
}
public static get Instance() {
if (!this.discordClient) {
this.discordClient = new DiscordClient();
}
return this.discordClient;
}
}
export const DiscordClientInstance = DiscordClient.Instance;
NOTE: token is merely a placeholder for the unique token of my bot application registered with discord.
/pages/api/test1.ts
import { DiscordClientInstance } from "../../DiscordClient";
export default (req, res) => {
let guild = DiscordClientInstance.APIClient.guilds.fetch($GUILD_1_ID)
.then(guild => {
console.log(guild.name);
res.statusCode = 200;
res.json({ name: guild.name });
})
.catch(console.error);
}
/pages/api/test2.ts
import { DiscordClientInstance } from "../../DiscordClient";
export default (req, res) => {
let guild = DiscordClientInstance.APIClient.guilds.fetch($GUILD_2_ID)
.then(guild => {
console.log(guild.name);
res.statusCode = 200;
res.json({ name: guild.name });
})
.catch(console.error);
}
NOTE: $GUILD_#_ID is merely a placeholder for where the the id of the discord server I am fetching would go.
As can be seen above, test1.ts and test2.ts are nearly identical and are inheriting the same const.
If anyone had any clues as to why this is happening, I would be very appreciative. Some people on other sites from my late-night googling have suggested this could be an issue with node, however, I honestly have no clue.
Thanks,
Matt :)
I use this very pattern without issues - Have you tried this in production mode? When in development mode Next.js will compile each page on-demand which I've observed breaking this pattern. Essentially, if you see "compiling..." then you've lost your persistence. In production mode this doesn't happen and you should see your single instance persisted.
I'm making an app that displays a list of orders. The problem is when I submit a new order by hitting an endpoint with a post request with the data for a new order, the page or the components need to refresh automatically to display this new order. I don't know how this is achieved with nuxt. The client-side HTML rendering needs to be actively reacting to events happening on the server-side.
So, you don't actually need to refresh the page to do this, you can use axios to POST an order and update your page with that particular order
export default {
data(){
return {
newOrder: null
}
},
methods:{
newOrder(){
const newOrder = await this.$axios.post(...)
this.newOrder = newOrder
},
}
}
I'm confused how to run a one-off script as part of a Loopback 4 project that fills in a value for old records. E.g. I want to include a createdAt field on the user record and fill in old records based on the timestamp of the first post each user made in the system.
I've tried creating a component that will run one time only to update all these records.
export class OneTimeComponent implements Component {
constructor(
#repository(UserRepository)
public userRepository : UserRepository,
){
this.start()
}
async start(){
const users = await this.userRepository.find();
users.forEach( user => {
// find first post for user
// get date
// update user.createdAt with date
})
}
}
export async function run(args: string[]) {
const app = new BlogApiApplication();
app.component(OneTimeComponent);
await app.boot();
// Connectors usually keep a pool of opened connections,
// this keeps the process running even after all work is done.
// We need to exit explicitly.
process.exit(0);
}
run(process.argv).catch(err => {
console.error('Cannot run this', err);
process.exit(1);
});
If I run app.start() then the script will try to run by booting up the app and then I get an error saying this address is in use – obviously because there are 2 apps running.
And I'd call this file with package script.
To access the repositories does this need to be a boot script? Or something else?
Any help much appreciated.
How to run a one-off script as part of a Loopback 4 project that fills in a value for old records. E.g. I want to include a createdAt field on the user record and fill in old records based on the timestamp of the first post each user made in the system.
Personally, I would make these change as part of database-schema migration, see Database Migrations in LoopBack documentation.
Most LoopBack connectors are smart enough to add createdAt field to your User table if it's not present yet. Filling the values is left for you to implement.
Quoting from Implement additional migration steps:
In some scenarios, the application may need to define additional schema constraints or seed the database with predefined model instances. This can be achieved by overriding the migrateSchema method provided by the mixin.
The example below shows how to do so in our Todo example application.
/** FILE: src/application.ts **/
import {TodoRepository} from './repositories';
// skipped: other imports
export class TodoListApplication extends BootMixin(
ServiceMixin(RepositoryMixin(RestApplication)),
) {
// skipped: the constructor, etc.
async migrateSchema(options?: SchemaMigrationOptions) {
// 1. Run migration scripts provided by connectors
await super.migrateSchema(options);
// 2. Make further changes. When creating predefined model instances,
// handle the case when these instances already exist.
const todoRepo = await this.getRepository(TodoRepository);
const found = await todoRepo.findOne({where: {title: 'welcome'}});
if (found) {
todoRepo.updateById(found.id, {isComplete: false});
} else {
await todoRepo.create({title: 'welcome', isComplete: false});
}
}
}
I've tried creating a component that will run one time only to update all these records.
(...)
To access the repositories does this need to be a boot script? Or something else?
If you don't want to migrate the database schema & data explicitly by running npm run migrate, you can modify your Application to call autoMigrate automatically as part of the exported main function.
Quoting from Auto-update database at start:
To automatically update the database schema whenever the application is started, modify your main script to execute app.migrateSchema() after the application was bootstrapped (all repositories were registered) but before it is actually started.
/** FILE: src/index.ts **/
export async function main(options: ApplicationConfig = {}) {
const app = new TodoListApplication(options);
await app.boot();
await app.migrateSchema();
await app.start();
const url = app.restServer.url;
console.log(`Server is running at ${url}`);
return app;
}
The last option is to leverage life-cycle observers (a.k.a "boot scripts" in LoopBack v3) - see Life cycle events:
Scaffold a new observer by running lb4 observer (see Life cycle observer generator).
Modify the constructor of the observer class to inject model repositories you want to work with.
Modify the start method to perform the required database changes.
In my routes array in main.js, I have set
meta : { requiresAuth: true }
for every component except on the UserLogin page.
Regarding navigation resolution, I have set a check before each route as follows:
router.beforeEach((to, from, next) => {
const currentUser = firebase.auth().currentUser;
const requiresAuth = to.matched.some(record => record.meta.requiresAuth);
const currentUserState = !!currentUser;
console.table({ currentUserState, requiresAuth });
if (requiresAuth && !currentUser) {
next('/login');
} else {
next();
}
}
With this setup, everything almost works as expected/intended. Logged in users can go anywhere, and anonymous users are redirected to localhost/login.
However, this checking and redirecting only functions when users are navigating by clicking the links in the web app (which are all <router-link>s that go :to="{ name: 'SomeComponent' }".
When any user (registered or anonymous) attempts to navigate their way around the app by typing or copy pasting urls, the app will automatically send them to the login page.
From the console.table output, I can see that manually entering urls always results in currentUserState being false.
It does not remain false: a user that has logged in (currentUserState === true), who then manually navigates somewhere (currentUserState === false), has their currentUserState revert back to true
I can only vaguely guess that the problem is related to rendering and firebase.auth().
firebase.initializeApp(config)
// you need to let firebase try and get the auth state before instantiating the
// the Vue component, otherwise the router will be created and you haven't
// resolved whether or not you have a user
firebase.auth().onAuthStateChanged(function(user) {
app = new Vue({
el: '#app',
template: '<App/>',
components: { App },
router
})
});
I have found a solution to this bug.
In my Vuex store.js, a snapshot of the state is stored into localStorage. This snapshot is updated any time there is a change to the store itself, so the localStorage backup is always up-to-date with the actual state of the Vue.js application itself.
store.subscribe((mutation, state) => {
localStorage.setItem('store', JSON.stringify(state));
});
This store.subscribe is set in my store.js just before the export line.
A function called initStore is also defined in the store as follows
initStore(state) {
if (localStorage.getItem('store')) {
this.replaceState(Object.assign(state, JSON.parse(localStorage.getItem('store'))));
}
},
and after my store object, it is immediately called with
store.commit('initStore').