Comparing two arrays from async functions? - node.js

Question

I have read a lot of posts on how to solve this problem, but I cannot understand it.
I have a database (psql) and a csv. I have a two functions. One to read a list of domains from psql. And another to read a different list of domains from the csv.
Both functions are async operations that live in separate modules.
Goal: to bring the results of both reader functions (which are arrays)into the same file and compare the files for duplicates.
Currently, I have made progress using Promise.all. However, I cannot seem to isolate the two separate arrays so I can use them.
Solution Function (not working):
This is where I am trying to read in both lists into two separate arrays.
The CSVList variable has a console.log that logs the array when the CSVList.filter is not present. Which leads me to believe that the array is actually there? Maybe?
const allData = async function () {
let [result1, result2] = await Promise.all([readCSV, DBList]);
const DBLists = result2(async (domainlist) => {
return domainlist;
});
const CSVList = result1(async (csv) => {
const csvArr = await csv.map((x) => {
return x[0];
});
console.log(csvArr);
return csvArr;
});
const main = await CSVList.filter((val) => !DBLists.includes(vals)); // this doesn't work. it says that filter is not a function. I understand why filter is not a function. What I do not understand is why the array is not being returned?
};
allData();
psql reader:
const { pool } = require("./pgConnect");
//
const DBList = async (callback) => {
await pool
.query(
`
SELECT website
FROM domains
limit 5
`
)
.then(async (data) => {
const domainList = await data.rows.map((x) => {
return x.website;
});
callback(domainList);
});
};
csv reader:
const { parseFile } = require("#fast-csv/parse");
const path = require("path");
const fs = require("fs");
const domainPath = path.join(__dirname, "domains.csv");
//reads initial domain list and pushes the domains to an array
//on end, calls a callback function with the domain data
const readCSV = async (callback) => {
let domainList = [];
let csvStream = parseFile(domainPath, { headers: false })
.on("data", (data) => {
//push csv data to domainList array
domainList.push(data);
// console.log(data);
})
.on("end", () => {
callback(domainList);
});
};

Answer 1

I took jFriend00 Advice and I updated my code a bit.
The biggest issue was the ReadCSV function. Fast-csv doesn't seem to be asynchronous. I wrapped it in a new promise manually. And then resolved that promise passing the domain list as an argument to resolve.
updated CSV Reader:
const readCSV2 = new Promise((resolve, reject) => {
let domainList = [];
let csvStream = parseFile(domainPath, { headers: false })
.on("data", (data) => {
//push csv data to domainList array
domainList.push(data[0]);
// console.log(data);
})
.on("end", () => {
resolve(domainList);
});
});
Updated Solution for comparing the two lists
const allData = async function () {
// get the values from the DB and CSV in one place
let [result1, result2] = await Promise.all([readCSV2, DBList]);
const CSVDomains = await result1;
const DBDomains = await result2();
//final list compares the two lists and returns the list of non duplicated domains.
const finalList = await CSVDomains.filter(
(val) => !DBDomains.includes(val)
);
console.log("The new list is: " + finalList);
};
Quick aside: I could have accomplished the same result by using psql ON CONFLICT DO NOTHING. This would have ignored duplicates when updating to the database because I have a UNIQUE constraint on the domain column.