How do you call an external API from Selenium script to populate a field using Node JS? - node.js

I have a use case where I need to call an external API, parse the JSON that is returned and populate a form field in a web page all within a Selenium script written using Node JS.
Something like this:
// in Selenium script get the form field
let inputElement = await getElementById(driver, "my-id");
// then call an API including callback function
// in the callback function with the JSON response from the API
const myText = response.data.text;
await inputElement.sendKeys(myText,Key.ENTER);
I actually not even sure where to start with this - because I would be adding asynchronous code (the API call and waiting for the response in the callback) to the existing asynchronous code that is running as part of the Selenium script. And I need to not lose references to the web driver and the input element.
Some advice and recommendations to get me going would be very helpful.

If you are using node's inbuild https module the you can do something like this..
const { Builder, By, Key, until } = require("selenium-webdriver");
const https = require("https");
(async function example() {
let driver = await new Builder().forBrowser("chrome").build();
try {
await driver.get("http://www.google.com/ncr");
await https.get("https://jsonplaceholder.typicode.com/users/1", (resp) => {
let data = "";
resp.on("data", (chunk) => {
data += chunk;
});
resp.on("end", async () => {
// console.log(JSON.parse(data)["name"]);
await driver
.findElement(By.name("q"))
.sendKeys(JSON.parse(data)["name"], Key.RETURN);
});
});
await driver.wait(until.titleContains("- Google Search"), 1000);
} finally {
await driver.quit();
}
})();
Or if you are already using library like axios, then you can do something like this
const { Builder, By, Key, until } = require("selenium-webdriver");
const axios = require("axios");
(async function example() {
let driver = await new Builder().forBrowser("chrome").build();
try {
await driver.get("http://www.google.com/ncr");
const { data } = await axios.get(
"https://jsonplaceholder.typicode.com/users/1"
);
await driver.findElement(By.name("q")).sendKeys(data["name"], Key.RETURN);
await driver.wait(until.titleContains("- Google Search"), 1000);
} finally {
await driver.quit();
}
})();
Hope this is what you are looking for..

Related

problem with making api request in node.js and getting the response in/to function caller

Ive spent a bit of time trying to understand this. I hope the answer is obvious and just show my lack of experience
My goal is to send API request to steam for various IDs of game mods and find the time_updated for each one, to put these all into an array and then find out which one most the most recently updated
I have got the code below, but its not quite doing what I want, I think I am just getting muddled over timings
My plan was to have a few different values in arrMODID = [], and to loop through each one, get the time_updated, push that to an array and for const result = await myfunction(); to be able to access the data in the modinfoArray
however that is returning an array with just [{test},{}] in it and is being fired before the function has put any data into the array
can anyone give me a shove in the right direction please
thank you
import request from 'request';
const myfunction = async function(x, y) {
var arrMODID = ["2016338122"];
var modinfoArray = []
var timeUpdated
for (const element of arrMODID) {
request.post({
headers: {'content-type' : 'application/x-www-form-urlencoded'},
url: 'http://api.steampowered.com/ISteamRemoteStorage/GetPublishedFileDetails/v1',
body: 'itemcount=1&publishedfileids[0]=2016338122',
},
function(error, response, body){
var response = JSON.parse(body);
var myResponse = response.response.publishedfiledetails
myResponse.forEach(function(arrayItem) {
//console.log(arrayItem.time_updated)
timeUpdated = arrayItem.time_updated
//console.log(timeUpdated)
modinfoArray.push({"2016338122":arrayItem.time_updated})
console.log(modinfoArray) // only this log returns the added items
})
});
}
return ["test", modinfoArray];
};
// Start function
const start = async function(a, b) {
const result = await myfunction();
console.log(result); // this returns the empty array
}
// Call start
start();
You need to use an http request library that supports promises so you can await that inside your function. You cannot successfully mix promises and asynchronous operations like request.post() that uses plain callbacks because you can manage the control flow in a promise-like way with plain callbacks.
I'd suggest using the got() library. Also, the request() library has been deprecated and is not recommended for new code. If you absolutely wanted to stay with the request() library, you could use the request-promise module instead, but keep in mind that the request() library is in maintenance mode only (no new feature development) whereas this list of alternatives are all being actively developed.
Here's a runnable implementation using the got() library:
import got from 'got';
const myfunction = async function() {
const arrMODID = ["2016338122"];
const modinfoArray = [];
for (const element of arrMODID) {
const response = await got.post({
headers: { 'content-type': 'application/x-www-form-urlencoded' },
url: 'http://api.steampowered.com/ISteamRemoteStorage/GetPublishedFileDetails/v1',
body: 'itemcount=1&publishedfileids[0]=2016338122',
}).json();
const myResponse = response.response.publishedfiledetails;
for (const arrayItem of myResponse) {
modinfoArray.push({ "2016338122": arrayItem.time_updated });
}
}
return ["test", modinfoArray];
};
// Start function
const start = async function() {
const result = await myfunction();
console.log(result);
return result;
}
// Call start
start().then(result => {
console.log("done");
}).catch(err => {
console.log(err);
});

Node js repeating a get request until there is a change in response

I will start off by saying I am a complete newbie when it comes to node js. I have the code below which currently sends a get request to the URL. It parses a specific value of the response and stores it as the search variable. It then uses the instagram api to change the bio on my instagram account to that search variable. However I would like the get request to continue until it detects a change. Ex. When the program is first run it fires off a get request. The first response value we get we will call 1. However after the first response I want it to continue to do get requests say every 5 seconds. The moment the response value changes from 1 to anything else I want that new value to be sent to the instagram bio. Can anyone help?
const { IgApiClient } = require("instagram-private-api")
const ig = new IgApiClient()
const https = require('https')
const USERNAME = "MYUSERNAME"
const PASSWORD = "MYPASS"
ig.state.generateDevice(USERNAME)
const main = async () => {
let url = "https://11z.co/_w/14011/selection";
https.get(url,(res) => {
let body = "";
res.on("data", (chunk) => {
body += chunk;
});
res.on("end", async () => {
try {
search = await JSON.parse(body).value;
} catch (error) {
console.error(error.message);
};
});
}).on("error", (error) => {
console.error(error.message);
});
await ig.simulate.preLoginFlow()
await ig.account.login(USERNAME, PASSWORD)
// log out of Instagram when done
process.nextTick(async () => await ig.simulate.postLoginFlow())
// fill in whatever you want your new Instagram bio to be
await ig.account.setBiography(search)
}
main()
// code is written in main() so that I can use async/await
to be good citizen to the target endpoint:
have a look on exponential-backoff package - https://www.npmjs.com/package/exponential-backoff
A utility that allows retrying a function with an exponential delay between attempts.

Retrieve JSON from URL and convert it to Cloud Firestore Collection with Cloud Functions

Here is what I want to achieve : I want to get a JSON on a daily basis from a URL and convert it to a cloud firestore collection in order to be able to use it in my Flutter app. Ideally, the script would only add new data to the collection.
I saw that I can use scheduler from Firebase cloud functions to run tasks daily. That's not the problem for now.
However, I don't know how to use Firebase cloud functions properly to get data from URL and convert it to collection. Maybe that's not the point of cloud functions and I misunderstood something. So first question : Can I run classic nodeJS stuff inside cloud functions? I suppose I can
Next, I initialized a cloud function project locally, connected it to my Google account and started to write code into index.js.
const functions = require("firebase-functions");
const admin = require('firebase-admin');
const fetch = require('node-fetch');
const db = admin.firestore();
const collectionToiletRef = db.collection('mycollection');
let settings = { method: "Get" };
let url = "my-url.com"
fetch(url, settings)
.then(res => res.json())
.then((json) => {
print(json);
// TODO for each json object, add new document
});
Second question : How can I run this code to see if it works ? I saw that emulator can be used but how can I check visually my cloud firestore collection ? On this simple example, I only want to print my json to see if I can get the data correctly. Where would the printing be done ?
Maybe cloud functions is not what I need for this task. Maybe my code is bad. I don't know. Thanks for your help.
EDIT
I tried this but the call never ends. I think it's waiting for a promise that never returns or something like that.
const functions = require("firebase-functions");
const admin = require('firebase-admin');
const fetch = require('node-fetch');
admin.initializeApp();
const db = admin.firestore();
exports.tempoCF = functions
.firestore.document('/tempo/{docId}')
.onCreate(async (snap, context) => {
console.log("onCreate");
let settings = { method: "Get" };
let url = "https://opendata.paris.fr/api/records/1.0/search/?dataset=sanisettesparis&q=&rows=-1"
try {
let response = await fetch(url, settings);
let json = await response.json();
// TODO for each json object, add new document
await Promise.all(json["records"].map(toiletJsonObject => {
return db.collection('toilets').doc(toiletJsonObject["recordid"]).set({}); // Only to create documents, I will deal with the content later
}));
}
catch(error) {
console.log(error);
return null;
}
}
);
This code works and create all the documents I want but never return. However, the async (snap, context) => {} passed to onCreate is a Promise. And this promise ends when Promise.all ends. I'm missing something but I don't know why. I'm struggling a lot with async programming with Dart or JS. Not very clear in my mind.
Can I run classic nodeJS stuff inside cloud functions?
Sure! Since the fetch method returns a Promise you can very well use it in a background triggered or a scheduled Cloud Function.
How can I run this code to see if it works?
Your code will work perfectly in the emulator suite, but you will need to trigger the Cloud Function with one of the Firebase services that can run in the emulator. For example you can trigger the Cloud Function by creating a document in the Firestore emulator console.
The following Cloud Function will do the trick: just create a doc in a dummy tempo collection and the CF will add a new doc in a newDocscollection. It's up to you to adapt the fields values for this doc, I've just used the entire JSON object.
exports.tempoCF = functions
.firestore.document('/tempo/{docId}')
.onCreate((snap, context) => {
let settings = { method: "Get" };
let url = "https://..."
return fetch(url, settings)
.then(res => res.json())
.then((json) => {
console.log(json);
// TODO for each json object, add new document
return admin.firestore().collection('newDocs').add(json);
})
.catch(error => {
console.log(error);
return null;
});
});
You could also deploy your Cloud Function to the Firebase backend, and if you want to schedule it, just change the code as follows (change the trigger):
exports.scheduledFunction = functions.pubsub.schedule('every 5 minutes').onRun((context) => {
let settings = { method: "Get" };
let url = "https://..."
return fetch(url, settings)
.then(res => res.json())
.then((json) => {
console.log(json);
// TODO for each json object, add new document
return admin.firestore().collection('newDocs').add(json);
})
.catch(error => {
console.log(error);
return null;
});
});
Edit following your edit:
The following code does work correctly in the emulator, creating docs in the toilets collection.
exports.tempoCF = functions.firestore
.document('/tempo/{docId}')
.onCreate(async (snap, context) => {
console.log('onCreate');
let settings = { method: 'Get' };
let url =
'https://opendata.paris.fr/api/records/1.0/search/?dataset=sanisettesparis&q=&rows=-1';
try {
let response = await fetch(url, settings);
let json = await response.json();
return Promise.all( // Here we return the promise returned by Promise.all(), so the life cycle of the CF is correctly managed
json['records'].map((toiletJsonObject) => {
admin
.firestore()
.collection('toilets')
.doc(toiletJsonObject['recordid'])
.set({ adresse: toiletJsonObject.fields.adresse });
})
);
} catch (error) {
console.log(error);
return null;
}
});

Node js extract data from a nested function

I am trying to process signup data for a uni project . I am using basic koa modules and I am not allowed to use express, ideally I want to get the data inside the variable post. I want to process the data for example to see if the password has less than 5 characters , if so i would like that the program would not redirect the user to different address but if no errors occur i would like the program to redirect to regOk.html, I tried many other ways like initializing the variable outside of ctx.req.on but none were successful . Can anyone help me ?
export async function postregister(ctx) {
let bodyString = "";
ctx.req.on("data", (chunk) => {
bodyString += chunk;
});
//let collectData = new Array();
ctx.req.on("end", () => {
var post = querystring.parse(bodyString);
var email = post["email"];
var password = post["password"];
var passbestätigen = post["passwort bestä"];
var vorname = post["vorname"];
var nachname = post["nachname"];
var adresse = post["adresse"];
var stadt = post["stadt"];
var telefonnummer = post["telefonnummer"];
var geburtsdatum = post["geburtsdatum"];
var regData = model.add(ctx.db, post);
regData.then(() => console.log("singup successful"))
});
await ctx.render("regOk.html");
}
I'm not very familiar with koa, but I believe your issue is related to the order in which your code is executed.
The event in charge of parsing the data received in the body of the request ends after the synchronic execution of your postregister method, so you never get to see the value of post in the order you'd expect.
One possible solution to go around this issue would be wrapping the parsing of data in a promise, waiting for that promise to complete, and executing then and catch functions once the processing is done.
export async function postregister(ctx) {
await new Promise((resolve) => {
let bodyString = "";
ctx.req.on("data", (chunk) => {
bodyString += chunk;
});
ctx.req.on("end", async () => {
resolve(querystring.parse(bodyString));
});
})
.then(async (post) => {
await model.add(ctx.db, post)
.then(async () => {
console.log("singup successful");
await ctx.render('regOk.html');
});
})
.catch(async (error) => {
console.error(error);
await ctx.render('error.html');
});
}
This way, you handle body parsing inside the Promise, and after that completed you get the result of querystring.parse(bodyString) as a variable named post in your then handler.

Synchronously iterate through firestore collection

I have a firebase callable function that does some batch processing on documents in a collection.
The steps are
Copy document to a separate collection, archive it
Run http request to third party service based on data in document
If 2 was successful, delete document
I'm having trouble with forcing the code to run synchronously. I can't figure out the correct await syntax.
async function archiveOrders (myCollection: string) {
//get documents in array for iterating
const currentOrders = [];
console.log('getting current orders');
await db.collection(myCollection).get().then(querySnapshot => {
querySnapshot.forEach(doc => {
currentOrders.push(doc.data());
});
});
console.log(currentOrders);
//copy Orders
currentOrders.forEach (async (doc) => {
if (something about doc data is true ) {
let id = "";
id = doc.id.toString();
await db.collection(myCollection).doc(id).set(doc);
console.log('this was copied: ' + id, doc);
}
});
}
To solve the problem I made a separate function call which returns a promise that I can await for.
I also leveraged the QuerySnapshot which returns an array of all the documents in this QuerySnapshot. See here for usage.
// from inside cloud function
// using firebase node.js admin sdk
const current_orders = await db.collection("currentOrders").get();
for (let index = 0; index < current_orders.docs.length; index++) {
const order = current_orders.docs[index];
await archive(order);
}
async function archive(doc) {
let docData = await doc.data();
if (conditional logic....) {
try {
// await make third party api request
await db.collection("currentOrders").doc(id).delete();
}
catch (err) {
console.log(err)
}
} //end if
} //end archive
Now i'm not familiar with firebase so you will have to tell me if there is something wrong with how i access the data.
You can use await Promise.all() to wait for all promises to resolve before you continue the execution of the function, Promise.all() will fire all requests simultaneously and will not wait for one to finish before firing the next one.
Also although the syntax of async/await looks synchronous, things still happen asynchronously
async function archiveOrders(myCollection: string) {
console.log('getting current orders')
const querySnapshot = await db.collection(myCollection).get()
const currentOrders = querySnapshot.docs.map(doc => doc.data())
console.log(currentOrders)
await Promise.all(currentOrders.map((doc) => {
if (something something) {
return db.collection(myCollection).doc(doc.id.toString()).set(doc)
}
}))
console.log('copied orders')
}

Resources