Serializing Azure long running operation for later reuse - node.js

I'm trying to use Azure SDK for javascript (#azure/arm-sql version 8.0.0) to copy SQL database but I don't want to wait until the operation is done. Instead, I'd like to exit once the request is created and later (let's say each minute) check whether the operation has finished. The SDK seems to support my use case through functions:
getPollState()
Get an LROPollState object that can be used to poll this LRO in a different context (such as on a different process or a different machine). If the LRO couldn't produce an LRO polling strategy, then this will return undefined.
and restoreLROPoller()
Restore an LROPoller from the provided LROPollState. This method can be used to recreate an LROPoller on a different process or machine.
However, the documentation doesn't specify how the state should be serialized/transferred over the wire. I naively tried to serialize it into JSON but when I run the snippet below, I get the following error:
TypeError: operationSpec.serializer.deserialize is not a function occurred in deserializing the responseBody - {"name":"b9952e45-85ff-41f8-b01c-83050c9d9a2c","status":"InProgress","startTime":"2021-10-14T15:38:01.59Z"}
Here is a simplified code snippet:
import { SqlManagementClient } from "#azure/arm-sql";
import { DefaultAzureCredential } from "#azure/identity";
import { LROPoller } from "#azure/ms-rest-azure-js";
const subscription = "<subscription ID>";
const rg = "myResourceGroup";
const server = "mySqlServer";
const dbName = "myDb";
const credentials = new DefaultAzureCredential();
const sqlClient = new SqlManagementClient(credentials, subscription);
const originalDb = await sqlClient.databases.get(rg, server, dbName);
const operation: LROPoller = await sqlClient.databases.beginCreateOrUpdate(rg, server, dbName + "_copy", {
location: "westeurope",
createMode: "Copy",
sourceDatabaseId: originalDb.id
});
const operationState = operation.getPollState()!;
const serializedState = JSON.stringify(operationState);
// The program would save the state somewhere and exit now, but let's make it simple.
const deserializedState = JSON.parse(serializedState);
const restoredOperation: LROPoller = sqlClient.restoreLROPoller(deserializedState);
// Following line throws the exception
// TypeError: operationSpec.serializer.deserialize is not a function occurred in deserializing the responseBody…
await restoredOperation.poll();
So my question is how can I save the operation state in a way that I can later reuse it.

For those who might want to achieve something similar, here is the workaround. However, I still want to get rid of extra code and use SDK functionality itself, so if anyone can answer the original question, I'd be more than happy.
Here is a file AzureOperations.ts with helper functions
import { TokenCredential } from "#azure/core-auth";
import { LROPoller } from "#azure/ms-rest-azure-js";
import fetch from "node-fetch";
export interface AzureOperationReference {
statusUrl: string
}
export interface AzureOperation {
status: "InProgress" | "Succeeded" | "Failed" | "Canceled"
error?: {
code: string,
message: string
}
}
export const createAzureOperationReference = (operation: LROPoller): AzureOperationReference => {
const asyncOperationHeader = "Azure-AsyncOperation";
const headers = operation.getInitialResponse().headers;
if (!headers.contains(asyncOperationHeader)) {
throw new Error(`Given operation is currently not supported because it does not contain header '${asyncOperationHeader}'. If you want to track this operation, implement logic that uses header 'Location' first.`);
}
return {
statusUrl: headers.get(asyncOperationHeader)!
};
};
export const createAzureOperationChecker = (operationReference: AzureOperationReference, credentials: TokenCredential) => {
let token: string = null!;
let tokenExpiration = 0;
let previousOperation: AzureOperation = null!;
let retryAfter = 0;
return async () => {
const now = new Date().getTime();
if (now < retryAfter) {
return previousOperation;
}
if (tokenExpiration < now) {
const newToken = await credentials.getToken("https://management.azure.com/.default");
if (newToken === null) {
throw new Error("Cannot obtain new Azure access token.");
}
tokenExpiration = newToken.expiresOnTimestamp;
token = newToken.token;
}
const response = await fetch(operationReference.statusUrl, {
method: "GET",
headers: {
Authorization: `Bearer ${token}`
}
});
const retryLimitInMiliseconds = Number(response.headers.get("Retry-After")) * 1000;
retryAfter = new Date().getTime() + retryLimitInMiliseconds;
return previousOperation = await response.json() as AzureOperation;
}
}
Then you can import and use them to track pending operations:
// unimportant code removed for brevity
import { createAzureOperationReference, createAzureOperationChecker } from "./AzureOperations.js";
const credentials = new DefaultAzureCredential();
const operation: LROPoller = await sqlClient.databases.beginCreateOrUpdate(…);
// You can serialize this reference as json and store it wherever you want
const reference = createAzureOperationReference(operation);
// You can deserialize it later and use it to fetch the operation status
const checkOperation = createAzureOperationChecker(reference, credentials);
const operationStatus = await checkOperation();
console.log(operationStatus.status);

Related

NodeJS GC function cannot be initialized

Trying out my first NodeJS cloud function so far unsuccessfully despite working fine VS code. Getting following error
Function cannot be initialized. Error: function terminated.
Looking through the logs I see some potential issues
Detailed stack trace: ReferenceError: supabase_public_url is not defined
Provided module can't be loaded (doesn't specify)
Thoughts: Am I doing it wrong with the secret manager and using the pub/sub incorrect?
My Code index.js
import { createClient } from '#supabase/supabase-js'
import sgMail from "#sendgrid/mail"
import { SecretManagerServiceClient } from '#google-cloud/secret-manager'
//activate cloud secret manager
const client = new SecretManagerServiceClient()
const supabaseUrl = client.accessSecretVersion(supabase_public_url)
const supabaseKey = client.accessSecretVersion(supabase_service_key)
const sendgridKey = client.accessSecretVersion(sendgrid_service_key)
sgMail.setApiKey(sendgridKey)
const supabase = createClient(supabaseUrl, supabaseKey)
// get data for supabase where notifications coins are true
const supabaseNotifications = async() => {
let { data, error } = await supabase
.from('xxx')
.select('*, xxx!inner(coin, xx, combo_change, combo_signal, combo_prev_signal), xxx!inner(email)')
.eq('crypto_signals.combo_change', true)
if(error) {
console.error(error)
return
}
return data
}
//create an array of user emails from supabase data
const userEmail = (data) => {
try {
const emailList = []
for (let i of data) {
if (emailList.includes(i.profiles.email) != true) {
emailList.push(i.profiles.email)
} else {}
}
return emailList
}
catch(e) {
console.log(e)
}
}
// function to take email list and supabase data to generate emails to users
const sendEmail = (e, data ) => {
try {
for (let i of e) {
const signalList = []
for (let x of data) {
if(i == x.profiles.email) {
signalList.push(x)
} else {}
}
// create msg and send from my email to the user
const msg = {
to: i,
from:"xxxx",
subject: "Coin notification alert from CryptoOwl",
text: "One or more of you coins have a new signal",
html: signalList.toString()
}
sgMail.send(msg)
console.log(i)
}
}
catch(e) {
console.log(e)
}
}
// main function combines all 3 functions (supabase is await)
async function main(){
let supabaseData = await supabaseNotifications();
let supabaseEmails = userEmail(supabaseData);
let sendgridEmails = sendEmail(supabaseEmails, supabaseData);
}
exports.sendgridNotifications = (event, context) => {
main()
};
my package.json with type module to use import above
{
"type":"module",
"dependencies":{
"#sendgrid/mail":"^7.6.1",
"#supabase/supabase-js":"1.30.0",
"#google-cloud/secret-manager": "^3.11.0"
}
}
I'm not at all versed in Google Secret Manager but a rapid look at the Node.js library documentation shows (if I'm not mistaking) that accessSecretVersion() is an asynchronous method.
As a matter of facts, we find in the doc examples like the following one:
async function accessSecretVersion() {
const [version] = await client.accessSecretVersion({
name: name,
});
// Extract the payload as a string.
const payload = version.payload.data.toString();
// WARNING: Do not print the secret in a production environment - this
// snippet is showing how to access the secret material.
console.info(`Payload: ${payload}`);
}
See https://cloud.google.com/secret-manager/docs/samples/secretmanager-access-secret-version#secretmanager_access_secret_version-nodejs

Unable to update an item in CosmosDB using the replace method with JavaScript

I am trying to create a basic REST API using Azure functions and the cosmosDB client for JavaScript. I have been successful with all the actions except the UPDATE. The cosmosDB client uses conainter.item(id,category).replace(newObject) I am unable to get the container.item().replace method to work. When I test the function in the portal or using Postman, I get a 500 error and in the portal, I get the error: Result: Failure Exception: Error: invalid input: input is not string Stack: Error: invalid input: input is not string at trimSlashFromLeftAndRight.
Example of my basic document/item properties
{
id:002,
project:"Skip rope",
category:"task",
completed: false
}
const config = require("../sharedCode/config");
const { CosmosClient } = require("#azure/cosmos");
module.exports = async function (context, req) {
const endpoint = config.endpoint;
const key = config.key;
const client = new CosmosClient({ endpoint, key });
const database = client.database(config.databaseId);
const container = database.container(config.containerId);
const theId = req.params.id;
// I am retrieving the document/item that I want to update
const { resource: docToUpdate } = await container.item(theId).read();
// I am pulling the id and category properties from the retrieved document/item
// they are used as part of the replace method
const { id, category } = docToUpdate;
// I am updating the project property of the docToUpdate document/item
docToUpdate.project = "Go fly a kite";
// I am replacing the item referred to with the ID with the updated docToUpdate object
const { resource: updatedItem } = await container
.item(id, category)
.replace(docToUpdate);
const responseMessage = {
status: 200,
message: res.message,
data: updatedItem,
};
context.res = {
// status: 200, /* Defaults to 200 */
body: responseMessage,
};
};
I Googled the heck out of this and been through the Microsoft Azure CosmosDB documents from top-to-bottom, but I can't figure out how to get this to work. I can get the other CRUD operations to work based on the examples Microsoft docs provide, but not this. Any help would be greatly appreciated.
I believe the reason you’re getting this error is because the data type of your “id” field is numeric. The data type of “id” field should be string.
UPDATE
So I tried your code and was able to run it successfully. There was one issue I noticed in your code though:
const { resource: docToUpdate } = await container.item(theId).read();
In the above line of code, you are not specifying the partition key value. If you don't specify the value, then your docToUpdate would come as undefined. In my code I used task as partition key value (I created a container with /category as the partition key).
This is the code I wrote:
const { CosmosClient } = require("#azure/cosmos");
const endpoint = 'https://account.documents.azure.com:443/';
const key = 'accountkey==';
const databaseId = 'database-name';
const containerId = 'container-name';
// const docToUpdate = {
// 'id':'e067cbae-1700-4016-bc56-eb609fa8189f',
// 'project':"Skip rope",
// 'category':"task",
// 'completed': false
// };
async function readAndUpdateDocument() {
const client = new CosmosClient({ endpoint, key });
const database = client.database(databaseId);
const container = database.container(containerId);
const theId = 'e067cbae-1700-4016-bc56-eb609fa8189f';
const { resource: docToUpdate } = await container.item(theId, 'task').read();
console.log(docToUpdate);
console.log('==============================');
const { id, category } = docToUpdate;
docToUpdate.project = "Go fly a kite";
console.log(docToUpdate);
console.log('==============================');
const { resource: updatedItem } = await container
.item(id, category)
.replace(docToUpdate);
console.log(updatedItem);
console.log('==============================');
}
readAndUpdateDocument();
Can you try by using this code?

get GCP cloud function logs label.executionid

i want to get the execution id, that GCP gave to the Cloud functions, in order to store it in a database.
this is what i want to get =>
let currId = log.label.execution_id
In order to do it i'm fetching the logs thanks to this function (inside my cloudFunctions):
const logging = new Logging();
console.log(`executed ${eId}`)
printEntryMetadata(eId, sId);
async function printEntryMetadata(eId, sId) {
const options = {
filter: `textPayload = "executed ${eId}"`
};
const [entries] = await logging.getEntries(options);
console.log('Logs:');
console.log(`textPayload = "executed ${eId}"`)
console.log(JSON.stringify(entries))
// const metadata = entries[0].metadata
console.log(`${metadata.labels.execution_id}`)
}
But the JSON.stringify(entries) return an empty array. And when i use the filter mannualy it's working...
is the cloud function unable to fetch it own logs?
This is what i've done:
exports.LogMetadata = async(executionId, scopeId, ProjectID, Logging) => {
const logging = new Logging({ProjectID});
const options = {
filter: `textPayload = "executed ${executionId}.${scopeId}"`
};
const [entries] = await logging.getEntries(options);
console.log(JSON.stringify(entries))
try {
const metadata = entries[0].metadata
console.log(`${metadata.labels.execution_id}`)
} catch (error) {
console.log("can't find the log, cause' it's the first function executed...")
}
}
The only thing that doesn't work is that i can't fetch the first log of the first esxecuted function.

GCP Document AI Example Not Working - Receiving INVALID_ARGUMENT: Request contains an invalid argument

The error is with the batchProcessDocuments line and has the error:
{
code: 3,
details: 'Request contains an invalid argument.',
metadata: Metadata {
internalRepr: Map { 'grpc-server-stats-bin' => [Array] },
options: {}
},
note: 'Exception occurred in retry method that was not classified as transient'
}
I've tried to copy the example as much as possible but without success. Is there a way of finding out more information regarding the input parameters that are required? There are very few examples of using Document AI on the web with this being a new product.
Here is my code sample:
const projectId = "95715XXXXX";
const location = "eu"; // Format is 'us' or 'eu'
const processorId = "a1e1f6a3XXXXXXXX";
const gcsInputUri = "gs://nmm-storage/test.pdf";
const gcsOutputUri = "gs://nmm-storage";
const gcsOutputUriPrefix = "out_";
// Imports the Google Cloud client library
const {
DocumentProcessorServiceClient,
} = require("#google-cloud/documentai").v1beta3;
const { Storage } = require("#google-cloud/storage");
// Instantiates Document AI, Storage clients
const client = new DocumentProcessorServiceClient();
const storage = new Storage();
const { default: PQueue } = require("p-queue");
async function batchProcessDocument() {
const name = `projects/${projectId}/locations/${location}/processors/${processorId}`;
// Configure the batch process request.
const request = {
name,
inputConfigs: [
{
gcsSource: gcsInputUri,
mimeType: "application/pdf",
},
],
outputConfig: {
gcsDestination: `${gcsOutputUri}/${gcsOutputUriPrefix}/`,
},
};
// Batch process document using a long-running operation.
// You can wait for now, or get results later.
// Note: first request to the service takes longer than subsequent
// requests.
const [operation] = await client.batchProcessDocuments(request); //.catch(err => console.log('err', err));
// Wait for operation to complete.
await operation.promise();
console.log("Document processing complete.");
}
batchProcessDocument();
I think this is the solution: https://stackoverflow.com/a/66765483/15461811
(you have to set the apiEndpoint parameter)

Google api SQL database dump via google cloud functions

With a node.js script via googleapis I done dump of all databases created on my Google SQL instance, the dump generate a single file for all databases which i store in a bucket. My target is to have one file for each database and not one file for all, the main problem is that if I run an export request for database A i can't run another for database B until the first is done.
You may use Async with callback in order to run the exports sequentially, you may use the operations list method that will get you the status of the exports in order to be able to know when the export has finished and when to move to the next step using callback. For more information check this other post
My solution is to use a recursive function like this:
"use strict"
const { google } = require("googleapis");
const { auth } = require("google-auth-library");
const dateFormat = require('date-format');
var sqladmin = google.sql("v1beta4");
const project = "my-project-name";
const instanceName = "my-sql-instance-name";
const dbToDump = [];
exports.dumpDatabase = (_req, res) => {
async function dump() {
let count = 0;
let currentRequestName = '';
const authRes = await auth.getApplicationDefault();
var authClient = authRes.credential;
let databases = await sqladmin.databases.list({
project: project,
instance: instanceName,
auth: authClient
});
for (let i = 0; i < databases.data.items.length; i++) {
const element = databases.data.items[i];
// the system databases will be omitted
if (
element.name != "information_schema" &&
element.name != "sys" &&
element.name != "mysql" &&
element.name != "performance_schema"
) {
dbToDump.push(element.name);
}
}
async function recursiveCall() {
//exit condition
if (count >= dbToDump.length) {
res.status(200).send("Command complete");
return true;
}
// no request running
if (currentRequestName == '') {
// set data for next export call
var request = {
project: project,
instance: instanceName,
resource: {
exportContext: {
kind: "sql#exportContext",
fileType: "SQL",
uri: 'gs://my-gsc-bucket/${dbToDump[count]}-${dateFormat.asString('yyyyMMddhhmm', new Date())}.gz',
databases: [dbToDump[count]]
}
},
auth: authClient
};
let r = await sqladmin.instances.export(request); //dump start
currentRequestName = r.data.name;
}
// call to monitor request status
let requestStatus = await sqladmin.operations.get({ project: project, operation: currentRequestName, auth: authClient });
if (requestStatus.data.status == 'DONE') {
// the current request is completed, prepare for next call
count++;
currentRequestName = '';
recursiveCall();
} else {
// wait 10 seconds before check status
setTimeout(recursiveCall, 10000)
}
}
recoursiveCall();
}
dump();
};
This work for me, the only one more setting is to increase the timeout over the 60s.
Thank's Andres S for the support

Resources