VM firewall rules update - node.js

Is there an API to update the firewall rules using NodeJS, an example would be really appreciated.
Requirement: I have a list of CDN trusted IPs around 1700, to be allowed to access specific VM in GCP on port 80.
As I understand, we can have a maximum of 256 source ips per firewall rule. I can create and update 8 of them, and tag with the same name,
Question: can we do it using NodeJS API?
This API doesn't return firewall rules.
Equivalent of cli commands is as below
gcloud compute firewall-rules describe alltraffic
gcloud compute firewall-rules update alltraffic --source-ranges="14.201.176.140/32","14.201.176.144/32"
gcloud compute firewall-rules create ramtest1 --allow="tcp:80" --description="ramtest1" --source-ranges="205.251.192.0/19","52.95.174.0/24" --target-tags="tcp-111"
https://cloud.google.com/sdk/gcloud/reference/compute/firewall-rules/
don't see the update command in the nodejs api
https://cloud.google.com/nodejs/docs/reference/compute/0.10.x/Firewall#create
https://cloud.google.com/nodejs/docs/reference/compute/0.10.x/Compute#createFirewall
exports.run_process = async (req, res) => {
const Compute = require('#google-cloud/compute');
const compute = new Compute();
const network = compute.network('default');
const firewalls = (await network.getFirewalls())[0];
for(const firewall of firewalls) {
// console.log('firewall == '+JSON.stringify(firewall));
console.log('firewall = '+firewall.metadata.name);
if(firewall.metadata.name === 'alltraffic') {
console.log(' xxxxxxxxxxxxxxxxxxxx changing all traffic xxxxxxxxxxxxxx ');
}
}
return res.status(200).send('ok');
};
This code above lists the firewall rule, NFI why its called as firewall, when in the console its called as firewall rules, it's so confusing

You should use the setMetadata function to update a firewall rule. For example, take this nodejs snippet which reads and updates the description of a firewall rule:
async function doit() {
const Compute = require('#google-cloud/compute');
const compute = new Compute();
const f = compute.firewall('default-allow-10000');
f.get().then(data => {
const firewall = data[0];
console.log('initial description: ' + firewall.metadata.description);
const metadata = {
description: 'new description for this rule'
};
return firewall.setMetadata(metadata);
}).then(data => {
const firewall = data[0];
console.log('description set');
return compute.firewall('default-allow-10000').get();
}).then(data => {
const firewall = data[0];
console.log('current description: ' + firewall.metadata.description);
});
}
doit();
In my example, this gives the output of:
initial description: old description
description set
current description: new description for this rule
To see what exists on the metadata object, you should look at the definition of the Firewall resource in the REST API.

Related

How to call cloud function nearest to the user

I have a cloud function like this which has been set to run in multiple regions.
export const cloudFunction = functions
.region(["asia-south1", "us-central1", "europe-west1", "southamerica-east1"])
.https.onCall(async (data, context) => {});
How can I call the cloud function region nearest to the user? From any client side framework?
The best solution is to use an HTTPS Load Balancer and to create a serverless NEG with your Cloud Functions. The HTTPS Load Balancer will deploy a unicast IP, I mean an IP known in different PoP (Point of Presence) of Google, and will route the request to the closest location (from the PoP). It's native and out of the box, nothing to code.
You'll have to find the closet region based on user's timezone/location yourself and specify the region on client side for routing based on region w/o a balancer as each Cloud Function has it's own URL containing the region. For example, one way would be like:
const getClosestGcpRegion = () => {
const regions = ['asia-south1', 'us-central1', 'europe-west1']
const regionOffsets = {
'asia-south1': '+05:30',
'us-central1': '-06:00',
'europe-west1': '+01:00',
}
let closestRegion = ''
let closestOffset = Number.MAX_SAFE_INTEGER
for (const region of regions) {
const offset = regionOffsets[region].split(':')
const offsetMinutes = Number(offset[0]) * 60 + Number(offset[1])
const offsetDiff = Math.abs(DateTime.local().offset - offsetMinutes)
if (offsetDiff < closestOffset) {
closestOffset = offsetDiff
closestRegion = region
}
}
console.log({ closestRegion })
return closestRegion;
}
export const functions = getFunctions(app, getClosestGcpRegion())
Alternatively, also checkout Global external HTTP(S) load balancer with Cloud Functions that can help you achieve the same goal.

Azure Function connect Azure PostgreSQL ETIMEDOUT, errno: -4039

I have an Azure (AZ) Function does two things:
validate submitted info involving 3rd party packages.
when ok call a postgreSQL function at AZ to fetch a small set of data
Testing with Postman, this AF localhost response time < 40 ms. Deployed to Cloud, change URL to AZ, same set of data, took 30 seconds got Status: 500 Internal Server Error.
Did a search, thought this SO might be the case, that I need to bump my subscription to the expensive one to avoid cold start.
But more investigation running part 1 and 2 individually and combined, found:
validation part alone runs perfect at AZ, response time < 40ms, just like local, suggests cold start/npm-installation is not an issue.
pg function call always long and status: 500 regardless it runs alone or succeeding part 1, no data returned.
Application Insight is enabled and added a Diagnostic settings with:
FunctionAppLogs and AllMetrics selected
Send to LogAnalytiscs workspace and Stream to an event hub selected
Following queries found no error/exceptions:
requests | order by timestamp desc |limit 100 // success is "true", time taken 30 seconds, status = 500
traces | order by timestamp desc | limit 30 // success is "true", time taken 30 seconds, status = 500
exceptions | limit 30 // no data returned
How complicated my pg call is? Standard connection, simple and short:
require('dotenv').config({ path: './environment/PostgreSql.env'});
const fs = require("fs");
const pgp = require('pg-promise')(); // () = taking default initOptions
require('dotenv').config({ path: './environment/PostgreSql.env'});
const fs = require("fs");
const pgp = require('pg-promise')(); // () = taking default initOptions
db = pgp(
{
user: process.env.PGuser,
host: process.env.PGhost,
database: process.env.PGdatabase,
password: process.env.PGpassword,
port: process.env.PGport,
ssl:
{
rejectUnauthorized: true,
ca: fs.readFileSync("./environment/DigiCertGlobalRootCA.crt.pem").toString(),
},
}
);
const pgTest = (nothing) =>
{
return new Promise((resolve, reject) =>
{
var sql = 'select * from schema.test()'; // test() does a select from a 2-row narrrow table.
db.any(sql)
.then
(
good => resolve(good),
bad => reject({status: 555, body: bad})
)
}
);
}
module.exports = { pgTest }
AF test1 is a standard httpTrigger anonymous access:
const x1 = require("package1");
...
const xx = require("packagex");
const pgdb = require("db");
module.exports = function(context)
{
try
{
pgdb.pgTest(1)
.then
(
good => {context.res={body: good}; context.done();},
bad => {context.res={body: bad}; context.done();}
)
.catch(err => {console.log(err)})
}
catch(e)
{ context.res={body: bad}; context.done(); }
}
Note:
AZ = Azure.
AZ pg doesn't require SSL.
pg connectivity method: public access (allowed IP addresses)
Postman tests on Local F5 run against the same AZ pg database, all same region.
pgAdmin and psql all running fast against the same.
AF-deploy is zip-file deployment, my understanding it is using the same configuration.
I'm new to Azure but based on my experience, if it's about credential then should come back right away.
Update 1, FunctionAppLogs | where TimeGenerated between ( datetime(2022-01-21 16:33:20) .. datetime(2022-01-21 16:35:46) )
Is it because my pg network access set to Public access?
My AZ pgDB is a flexible server, current Networking is Public access (allowed IP address), and I have added some Firewall rule w/ client IP address. My assumption is access is allowed within AZ, but it's not.
Solution 1, simply check this box: Allow public access from any Azure servcie within Azure to this server at the bottom of the Settings -> Networking.
Solution 2, find out all AF's outbound IP and add them into Firewall rule, under Settings -> Networking. Reason to add them all is Azure select an outbound IP randomly.

GKE REST/Node API call to get number of nodes in a pool?

How can I get the current size of a GKE node pool using the REST (or Node) API?
I'm managing my own worker pool using my Express app running on my cluster, and can set the size of the pool and track the success of the setSize operation, but I see no API for getting the current node count. The NodePool resource only includes the original node count, not the current count. I don't want to use gcloud or kubectl on one of my production VMs.
I could go around GKE and try to infer the size using the Compute Engine (GCE) API, but I haven't looked into that approach yet. Note that it seems difficult to get the node count even from Stack Driver. Has anyone found any workarounds to get the current node size?
The worker pool size can be retrieved from the Compute Engine API by getting the instance group associated with the node pool.
const { google } = require('googleapis')
const Compute = require('#google-cloud/compute')
const container = google.container('v1')
const compute = new Compute()
const projectId = 'project-12345'
const zone = 'us-central1-a'
const nodePoolId = 'worker-pool'
const clusterId = 'cluster-name'
async function authorize() {
const auth = new google.auth.GoogleAuth({
scopes: [ 'https://www.googleapis.com/auth/cloud-platform' ],
})
return auth.getClient()
}
const getNodePoolSize = async () => {
const auth = await authorize()
const clusterName = `projects/${projectId}/zones/${zone}/clusters/${clusterId}`
const request = { name: clusterName, auth }
const response = await container.projects.locations.clusters.get(request)
const nodePool = response.data.nodePools.find(({ name }) => name === nodePoolId)
const igName = nodePool.instanceGroupUrls[0].match(/.*\/instanceGroupManagers\/([a-z0-9-]*)$/)[1]
const instanceGroup = await compute.zone(zone).instanceGroup(igName).get()
return instanceGroup[1 /* 0 is config, 1 is instance */].size
}
Note that this is using two different Node API mechanisms. We could use google.compute instead of #google-cloud/compute. Also, the two APIs are authenticated differently. The former uses the authorize() method to get a client, while the latter is authenticated via the default account set in environment variables.

Calling CosmosDB server from Azure Cloud Function

I am working on an Azure Cloud Function (runs on node js) that should return a collection of documents from my Azure Cosmos DB for MongoDB API account. It all works fine when I build and start the function locally, but fails when I deploy it to Azure. This is the error: MongoNetworkError: failed to connect to server [++++.mongo.cosmos.azure.com:++++] on first connect ...
I am new to CosmosDB and Azure Cloud Functions, so I am struggling to find the problem. I looked at the Firewall and virtual networks settings in the portal and tried out different variations of the connection string.
As it seems to work locally, I assume it could be a configuration setting in the portal. Can someone help me out?
1.Set up the connection
I used the primary connection string provided by the portal.
import * as mongoClient from 'mongodb';
import { cosmosConnectionStrings } from './credentials';
import { Context } from '#azure/functions';
// The MongoDB Node.js 3.0 driver requires encoding special characters in the Cosmos DB password.
const config = {
url: cosmosConnectionStrings.primary_connection_string_v1,
dbName: "****"
};
export async function createConnection(context: Context): Promise<any> {
let db: mongoClient.Db;
let connection: any;
try {
connection = await mongoClient.connect(config.url, {
useNewUrlParser: true,
ssl: true
});
context.log('Do we have a connection? ', connection.isConnected());
if (connection.isConnected()) {
db = connection.db(config.dbName);
context.log('Connected to: ', db.databaseName);
}
} catch (error) {
context.log(error);
context.log('Something went wrong');
}
return {
connection,
db
};
}
2. The main function
The main function that execute the query and returns the collection.
const httpTrigger: AzureFunction = async function (context: Context, req: HttpRequest): Promise<void> {
context.log('Get all projects function processed a request.');
try {
const { db, connection } = await createConnection(context);
if (db) {
const projects = db.collection('projects')
const res = await projects.find({})
const body = await res.toArray()
context.log('Response projects: ', body);
connection.close()
context.res = {
status: 200,
body
}
} else {
context.res = {
status: 400,
body: 'Could not connect to database'
};
}
} catch (error) {
context.log(error);
context.res = {
status: 400,
body: 'Internal server error'
};
}
};
I had another look at the firewall and private network settings and read the offical documentation on configuring an IP firewall. On default the current IP adddress of your local machine is added to the IP whitelist. That's why the function worked locally.
Based on the documentation I tried all the options described below. They all worked for me. However, it still remains unclear why I had to manually perform an action to make it work. I am also not sure which option is best.
Set Allow access from to All networks
All networks (including the internet) can access the database (obviously not advised)
Add the inbound and outbound IP addresses of the cloud function project to the whitelistThis could be challenging if the IP addresses changes over time. If you are on the consumption plan this will probably happen.
Check the Accept connections from within public Azure datacenters option in the Exceptions section
If you access your Azure Cosmos DB account from services that don’t
provide a static IP (for example, Azure Stream Analytics and Azure
Functions), you can still use the IP firewall to limit access. You can
enable access from other sources within the Azure by selecting the
Accept connections from within Azure datacenters option.
This option configures the firewall to allow all requests from Azure, including requests from the subscriptions of other customers deployed in Azure. The list of IPs allowed by this option is wide, so it limits the effectiveness of a firewall policy. Use this option only if your requests don’t originate from static IPs or subnets in virtual networks. Choosing this option automatically allows access from the Azure portal because the Azure portal is deployed in Azure.

Azure node SDK to get more than 50 virtual machines

I' using the Azure node SDK to get all virtual machines for the subscription :
var computeClient = new computeManagementClient.ComputeManagementClient(credentials, subscriptionId);
var clientNetworkManagement = new NetworkManagementClient(credentials, subscriptionId);
computeClient.virtualMachines.listAll(function (err, result) {
returnResult(result);
});
But I have subscription with more than 50 vm's and that call only return 50 vm's max.
It's possible to get more than 50 vms with this function computeClient.virtualMachines.listAll ?
https://github.com/Azure-Samples/compute-node-manage-vm
Thx
I tried to reproduce your issue, but failed that I can list all VMs via my code as below. Before to run my code, I assigned a role Virtual Machine Contributor(or you can use higher level role like Contributer or Owner) to my app registed in AzureAD for my current subscription, you can refer to the offical document Manage access to Azure resources using RBAC and the Azure portal to know it.
var msRestAzure = require('ms-rest-azure');
var ComputeManagementClient = require('azure-arm-compute');
var clientId = process.env['CLIENT_ID'] || '<your client id>';
var domain = process.env['DOMAIN'] || '<your tenant id>';
var secret = process.env['APPLICATION_SECRET'] || '<your client secret>';
var subscriptionId = process.env['AZURE_SUBSCRIPTION_ID'] || '<your subscription id for listing all VMs in it>';
var computeClient;
msRestAzure.loginWithServicePrincipalSecret(clientId, secret, domain, function (err, credentials, subscriptions) {
computeClient = new ComputeManagementClient(credentials, subscriptionId);
computeClient.virtualMachines.listAll(function (err, result) {
console.log(result.length);
});
});
On Azure portal, there are 155 VMs list in my current subscription as the figure below. However, the result of my code only is 153 VMs. I don't know why the results are different, but my code result is same with Azure CLI command az vm list | grep vmId | wc -l.
Fig 1. The number of VMs in my current subscription
Fig 2. The result of my code
Fig 3. The result of Azure CLI command az vm list|grep vmId|wc -l
Per my experience, I guess your issue was caused by assigning the lower permission role for your app to only list VMs that you have default accessing permission.
Any concern or update is very helpful for understanding what your real issue is, please feel free to let me know.
I don't know if this is the best way to solve the problem, but I find a solution:
msRestAzure.loginWithServicePrincipalSecret(clientId, secret, domain, function (err, credentials, subscriptions) {
computeClient = new ComputeManagementClient(credentials, subscriptionId);
computeClient.virtualMachines.listAll(function (err, result, httpRequest, response) {
let myResult = JSON.parse(response.body);
console.log(result.length);
nextLink = myResult.nextLink;
console.log(nextLink);
computeClient.virtualMachines.listAllNext(nextLink, function (err, result, request, response) {
console.log(result.length);
});
});
});
The first call (listAll) return 50 Vm's and "nextLink" value.
Than I call listAllNext(nextLink,... that return the others 39 Vm's

Resources