Conditional statements in my BigQuery queries are being ignored - node.js

Question

I have a simple BigQuery table with a few columns. One of the columns is named my_id (of type STRING). I'm querying my BigQuery datasets like this:
import * as bq from "#google-cloud/bigquery";
const bqdb = new bq.BigQuery();
// ...
const projectId = 'my_project';
const datasetId = "my_dataset";
const tableId = "my_table";
const dbId = [projectId, datasetId, tableId].join('.');
// myIds is an array of strings
const stringedArray = myIds.map((id) => '\'' + id + '\'');
const sql_select_query = `
SELECT my_id
FROM \`${dbId}\`
WHERE my_id IN (${String(stringedArray)})
LIMIT 1
;
`;
const dataset = bqdb.dataset(datasetId);
const destinationTable = dataset.table(tableId);
console.log("Querying database...");
const queryOptions = {
query: sql_select_query,
destination: destinationTable,
write_disposition: "WRITE_APPEND",
priority: 'BATCH',
};
// Run the query as a job
const [job] = await bqdb.createQueryJob(queryOptions);
// Wait for the job to finish.
const results = await job.getQueryResults({maxResults: 500});
const resutsArray = results[0]
This query brings back the ENTIRE table (all rows, all columns). In other words, the result of this query is the same as if I'd wrote:
const sql_select_query = `
SELECT *
FROM \`${dbId}\`
;
`;
The output is formatted like a successful query: there's no error messages or warnings. But all my conditionals are being ignored, even LIMIT.
Why is BigQuery dumping the entire table into the response?

Answer 1

If your query settings are set to writing results to a destination table and using write_disposition: "WRITE_APPEND", job.getQueryResults() will return the data of the destination table along with the newly appended data which is an expected behavior of BigQuery.
job.getQueryResults() will only return the initially selected result if a destination table is configured and write disposition is either 'write if empty' or 'overwrite table'.
As a workaround you can query two times, first using temporary table to return results with condition and run the query again appending to the destination table.
Using your code, you can create two query options. First query option does not have destination and second query option has the destination along with write_disposition. Then create two jobs that uses the first and second query option.
Code snippet:
const queryOptions = {
query: sql_select_query,
priority: 'BATCH',
};
const queryOptionsWrite = {
query: sql_select_query,
destination: destinationTable,
write_disposition: "WRITE_APPEND",
priority: 'BATCH',
};
const [queryJob] = await bqdb.createQueryJob(queryOptions);
const queryResults = await queryJob.getQueryResults();
console.log("Query result:");
console.log(queryResults[0]);
const [writeJob] = await bqdb.createQueryJob(queryOptionsWrite);
const writeResults = await writeJob.getQueryResults();
console.log("\nUpdated table values:");
console.log(writeResults[0]);
Test done: