I've a GCP pub/sub messaging event and it's configured to my nodejs application. In other words. there's topic and there's a subscription assigned to it and the subscription delivery type set to Push. This is how the subscription setup in GCP.
and exmaple node app api as follows:
group.post(
'/test-data',
testFunction
);
export const testFunction = (
req: Request,
res: Response,
next: NextFunction
) => {
const service = (req as any).service as ServiceContainer;
const available = service.UserDao.find(req.body.id);
if (result.length > 0) {
resutun res.json({found:1});
}
return service.UserDao
.saveIds(req.body.ids)
.then((response) => {
const deviceMap: { [id: string]: Device } = {};
(response || []).forEach((v) => {
deviceMap[v._id] = v;
});
res.json(deviceMap);
})
.catch((err) => {
next(err);
});
};
Now, before one message is processed, another message is sent to the URL from the subscription. So what I want is, to process the request sequentially, one topic should completely processed before the next gets processed.
Accoding to Google:
And this option only available for pull type:
Related
While not a front-end developer, I'm trying to set up a web app to show up a demo for a product. That app is based on the Sigma.js demo app demo repository.
You'll notice that this app relies on a graph which is hosted locally, which is loaded as:
/src/views/Root.tsx :
useEffect(() => {
fetch(`${process.env.PUBLIC_URL}/dataset.json`)
.then((res) => res.json())
.then((dataset: Dataset) => {...
// do things ....
and I wish to replace this by a call to another service which I also host on Cloud Run.
My first guess was to use the gcloud-auth-library, but I could not make it work - especially since it does not seem to support Webpack > 5 (I might be wrong here), the point here this lib introduces many problems in the app, and I thought I'd be better off trying the other way GCP suggests to handle auth tokens: by calling the Metadata server.
So I replaced the code above with:
Root.tsx :
import { getData } from "../getGraphData";
useEffect(() => {
getData()
.then((res) => res.json())
.then((dataset: Dataset) => {
// do even more things!
getGraphData.js :
import { getToken } from "./tokens";
const graphProviderUrl = '<my graph provider service URL>';
export const getData = async () => {
try {
const token = await getToken();
console.log(
"getGraphData.js :: getData : received token",
token
);
const request = await fetch(
`${graphProviderUrl}`,
{
headers: {
Authorization: `Bearer ${token}`,
},
}
);
const data = await request.json();
console.log("getGraphData.js :: getData : received graph", data);
return data;
} catch (error) {
console.log("getGraphData.js :: getData : error getting graph data", error);
return error.message;
}
};
tokens.js :
const targetAudience = '<my graph provider service base URL>'; // base URL as audience
const metadataServerAddress = "169.254.169.254"; // use this to shortcut DNS call to metadata.google.internal
export const getToken = async () => {
if (tokenExpired()) {
const token = await getValidTokenFromServer();
sessionStorage.setItem("accessToken", token.accessToken);
sessionStorage.setItem("expirationDate", newExpirationDate());
return token.accessToken;
} else {
console.log("tokens.js 11 | token not expired");
return sessionStorage.getItem("accessToken");
}
};
const newExpirationDate = () => {
var expiration = new Date();
expiration.setHours(expiration.getHours() + 1);
return expiration;
};
const tokenExpired = () => {
const now = Date.now();
const expirationDate = sessionStorage.getItem("expirationDate");
const expDate = new Date(expirationDate);
if (now > expDate.getTime()) {
return true; // token expired
}
return false; // valid token
};
const getValidTokenFromServer = async () => {
// get new token from server
try {
const request = await fetch(`http://${metadataServerAddress}/computeMetadata/v1/instance/service-accounts/default/token?audience=${targetAudience}`, {
headers: {
'Metadata-Flavor': 'Google'
}
});
const token = await request.json();
return token;
} catch (error) {
throw new Error("Issue getting new token", error.message);
}
};
I know that this kind of call will need to be done server-side. What I don't know is how to have it happen on a React + Node app. I've tried my best to integrate good practices but most questions related to this topic (request credentials through a HTTP (not HTTPS!) API call) end with answers that just say "you need to do this server-side", without providing more insight into the implementation.
There is a question with similar formulation and setting here but the single answer, no upvote and comments is a bit underwhelming. If the actual answer to the question is "you cannot ever call the metadata server from a react app and need to set up a third-party service to do so (e.g. firebase)", I'd be keen on having it said explicitly!
Please assume I have only a very superficial understanding of node.js and React!
I'm using the Node JS EventHubConsumerClient to read events from an EventHub hosted in a third party account, using this guide:
https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-node-get-started-send
I am also using a blob storage container that is hosted in my own Azure account to maintain the checkpoint of what the last read event was, the blob storage client is passed to the EventHubConsumerClient as an argument.
When I run the application locally it is able to successfully read from the EventHub and use the blob storage container for checkpointing. However, when I deploy this application to AWS and use it within a Lambda I receive this error upon invoking the subscribe method on the consumer client:
Method get TypedArray.prototype.length called on incompatible receiver [object Object]
I can't find any documentation online related to EventHub to help identify the root cause of this issue and i'm not sure how to debug this. I have confirmed that the blob storage is being successfully connected to from AWS via the logs.
Has anyone encountered this issue?
Code for creating blob checkpoint store:
const createCheckpointStore = async (
blobStorageUrl: string,
accountName: string,
accountKey: string
): Promise<BlobCheckpointStore> => {
const credentials = new StorageSharedKeyCredential(accountName, accountKey)
const blobServiceClient = new BlobServiceClient(blobStorageUrl, credentials)
const blobContainerClient = blobServiceClient.getContainerClient(
'event-hub-checkpoint'
)
const iter = blobContainerClient.listBlobsFlat()
for await (const item of iter) {
console.log(`\tBlobItem: name - ${item.name}`)
const blockBlobClient = blobContainerClient.getBlockBlobClient(item.name)
const meta = (await blockBlobClient.getProperties()).metadata
console.log(meta)
}
return new BlobCheckpointStore(blobContainerClient)
}
Code for creating the event hub consumer client:
const createEventHubConsumerClient = async () => {
try {
const accountName = await getSecretFromEnvVariable(
'BLOB_STORAGE_ACCOUNT_NAME'
)
const accountKey = await getSecretFromEnvVariable(
'BLOB_STORAGE_ACCOUNT_KEY'
)
const checkpointStore = await createCheckpointStore(
getEnvironmentVariable('BLOB_STORAGE_URL'),
accountName,
accountKey
)
const consumerGroup = '$Default'
const eventHubConnectionString = getEnvironmentVariable(
'EXPORT_EVENT_HUB_CONNECTION_STRING'
)
const eventHubName = 'export'
return new EventHubConsumerClient(
consumerGroup,
eventHubConnectionString,
eventHubName,
checkpointStore
)
} catch (error) {
console.error('error creating eventhub consumer client', error)
throw error
}
}
Read events method code:
const getEvents = async () => {
const consumerClient = await createEventHubConsumerClient()
try {
await new Promise<void>((resolve, reject) => {
const subscription: Subscription = consumerClient.subscribe(
{
processEvents: async (events, context) => {
console.log('received events', events)
await asyncForEach(events, async (event: ReceivedEventData) => {
const processedEntities = extractEntitiesFromEventHubEvent(event)
await asyncForEach(
processedEntities,
async (entity: any) => {
try {
console.log(entity)
} catch (err) {
throw new Error(err)
}
}
)
await context.updateCheckpoint(events[events.length - 1])
})
},
processError: async err => {
console.error('error encountered', err)
await subscription.close()
await consumerClient.close()
console.error('error processing events', err)
reject(new Error('error processing events'))
},
},
{
startPosition: earliestEventPosition,
maxBatchSize: 100,
maxWaitTimeInSeconds: 5,
}
)
})
} catch (err) {
// log error here that event hub consume failed
console.error('error encountered', err)
throw new Error(err)
}
}
Description:
I have created a Firebase app where a user can insert a Firestore document. When this document is created a timestamp is added so that it can be automatically deleted after x amount of time, by a cloud function.
After the document is created, a http/onCreate cloud function is triggered successfully, and it creates a cloud task. Which then deletes the document on the scheduled time.
export const onCreatePost = functions
.region(region)
.firestore.document('/boxes/{id}')
.onCreate(async (snapshot) => {
const data = snapshot.data() as ExpirationDocData;
// Box creation timestamp.
const { timestamp } = data;
// The path of the firebase document('/myCollection/{docId}').
const docPath = snapshot.ref.path;
await scheduleCloudTask(timestamp, docPath)
.then(() => {
console.log('onCreate: cloud task created successfully.');
})
.catch((error) => {
console.error(error);
});
});
export const scheduleCloudTask = async (timestamp: number, docPath: string) => {
// Convert timestamp to seconds.
const timestampToSeconds = timestamp / 1000;
// Doc time to live in seconds
const documentLifeTime = 20;
const expirationAtSeconds = timestampToSeconds + documentLifeTime;
// The Firebase project ID.
const project = 'my-project';
// Cloud Tasks -> firestore time to life queue.
const queue = 'my-queue';
const queuePath: string = tasksClient.queuePath(project, region, queue);
// The url to the callback function.
// That gets envoked by Google Cloud tasks when the deadline is reached.
const url = `https://${region}-${project}.cloudfunctions.net/callbackFn`;
const payload: ExpirationTaskPayload = { docPath };
// Google cloud IAM & ADMIN principle account.
const serviceAccountEmail = 'myServiceAccount#appspot.gserviceaccount.com';
// Configuration for the Cloud Task
const task = {
httpRequest: {
httpMethod: 'POST',
url,
oidcToken: {
serviceAccountEmail,
},
body: Buffer.from(JSON.stringify(payload)).toString('base64'),
headers: {
'Content-Type': 'application/json',
},
},
scheduleTime: {
seconds: expirationAtSeconds,
},
};
await tasksClient.createTask({
parent: queuePath,
task,
});
};
export const callbackFn = functions
.region(region)
.https.onRequest(async (req, res) => {
const payload = req.body as ExpirationTaskPayload;
try {
await admin.firestore().doc(payload.docPath).delete();
res.sendStatus(200);
} catch (error) {
console.error(error);
res.status(500).send(error);
}
});
Problem:
The user can also extend the time to live for the document. When that happens the timestamp is successfully updated in the Firestore document, and a http/onUpdate cloud function runs like expected.
Like shown below I tried to update the cloud tasks "time to live", by calling again
the scheduleCloudTask function. Which obviously does not work and I guess just creates another task for the document.
export const onDocTimestampUpdate = functions
.region(region)
.firestore.document('/myCollection/{docId}')
.onUpdate(async (change, context) => {
const before = change.before.data() as ExpirationDocData;
const after = change.after.data() as ExpirationDocData;
if (before.timestamp < after.timestamp) {
const docPath = change.before.ref.path;
await scheduleCloudTask(after.timestamp, docPath)
.then((res) => {
console.log('onUpdate: cloud task created successfully.');
return;
})
.catch((error) => {
console.error(error);
});
} else return;
});
I have not been able to find documentation or examples where an updateTask() or a similar method is used to update an existing task.
Should I use the deleteTask() method and then use the createTask() method and create a new task after the documents timestamp is updated?
Thanks in advance,
Cheers!
Yes, that's how you have to do it. There is no API to update a task.
I am trying to publish messages via google pubsub batch publishing feature. The batch publishing code looks like below.
const gRPC = require("grpc");
const { PubSub } = require("#google-cloud/pubsub");
const createPublishEventsInBatch = (topic) => {
const pubSub = new PubSub({ gRPC });
const batchPublisher = pubSub.topic(topic, {
batching: {
maxMessages: 100,
maxMilliseconds: 1000,
},
});
return async (logTrace, eventData) => {
console.log("Publishing batch events for", eventData);
try {
await batchPublisher.publish(Buffer.from(JSON.stringify(eventData)));
} catch (err) {
console.error("Error in publishing", err);
}
};
};
And this batch publisher is getting called from a service like this.
const publishEventsInBatch1 = publishEventFactory.createPublishEventsInBatch(
"topicName1"
);
const publishEventsInBatch2 = publishEventFactory.createPublishEventsInBatch(
"topicName2"
);
events.forEach((event) => {
publishEventsInBatch1(logTrace, event);
publishEventsInBatch2(logTrace, event);
});
I am using push subscription to receive the messages with the below settings.
Acknowledgement deadline: 600 Seconds
Retry policy: Retry immediately
The issue I am facing is, if the total number of events/messages is 250k, the push subscription is supposed to get less than or equal to 250k messages based on the message execution. But in my case, I am getting 3-4 M records on subscription and it is getting varied.
My fastify and pubsub configuration is
fastify: 3.10.1
#google-cloud/pubsub: 2.12.0
Adding the subscription code
fastify.post("/subscription", async (req, reply) => {
const message = req.body.message;
let event;
let data;
let entityType;
try {
let payload = Buffer.from(message.data, "base64").toString();
event = JSON.parse(payload);
data = event.data;
entityType = event.entityType;
if (entityType === "EVENT") {
if (event.version === "1.0") {
console.log("Processing subscription");
await processMessage(fastify, data);
} else {
console.error("Unknown version of stock event, being ignored");
}
} else {
console.error("Ignore event");
}
reply.code(200).send();
} catch (err) {
if (err.status === 409) {
console.error("Ignoring stock update due to 409: Conflict");
reply.code(200).send();
} else {
console.error("Error while processing event from subscription");
reply.code(500).send();
}
}
});
Can any one guide me where I am doing the mistakes. It's a simple fastify application. Do I am making any mistake in coding or any configuration.
I have a service bus topic that ingest telemetry messages outgoing from one of my servers ('telemetry-topic') at a rate of approximately 1 message every 10 seconds.
I have a web (site) front that subscribes to the 'telemetry-subscription' that consumes the 'telemetry-topic'.
What I witness is that the more front I run, the least message frequency it receives. I sounds like my topic acts as a queue…
What I should see is that any front -no matter the number of them— should sustain de approx 1 message very 10 seconds.
Note: My stack is node.
The receiver:
async function serviceBusTelemetrySubscribe(props) {
const module = '🚌 Service bus telemetry'
const {enter, leave, log, warn, error} = SDK.logFunctions(module, {
verboseLevel: 1,
verboseEnterLeave: true,
})
try {
enter()
log('Subscribing with given processMessage', props.processMessage, 'processError', props.processError)
const {processMessage, processError} = props
// connection string to your Service Bus namespace
const connectionString = 'XXXX'
// == Topic & subscription names
const topicName = 'telemetry-topic'
const subscriptionName = 'telemetry'
// == Create a Service Bus client using the connection string to the Service Bus namespace
const sbClient = new ServiceBusClient(connectionString)
log('Service bus client', sbClient)
// == Create a receiver for "all telemetry" subscription.
const receiver = sbClient.createReceiver(topicName, subscriptionName)
log('Service bus receiver for subscription name "', subscriptionName, '" is:', sbClient)
// == Function to handle messages, which provide a default one
const myMessageHandler = processMessage || (async (messageReceived) => {
log(`received message: ${messageReceived.body}`)
})
// == Function to handle any errors, which provide a default one
const myErrorHandler = processError || (async (error) => {
error(error)
})
log('Subscribing with actual processMessage', myMessageHandler, 'processError',myErrorHandler)
// == Subscribe and specify the message and error handlers
const sbSubscription = receiver.subscribe({
processMessage: myMessageHandler,
processError: myErrorHandler
})
log('Service bus subscription', sbSubscription)
// == Return cleanup method
return async () => {
log('Closing Service bus…')
// Waiting long enough before closing the sender to send messages
await delay(5000)
log('Closing Service bus subscription…')
await sbSubscription.close()
log('Closing Service bus receiver…')
await receiver.close()
log('Closing Service bus client…')
await sbClient.close()
}
}
catch(err) {
error(err)
}
finally {
leave()
}
}