Joins with 8 tables in YugabyteDB - yugabytedb

Question

[Question posted by a user on YugabyteDB Community Slack]
I have a dataset and a query with 8 joins on tables that have between 10 and 17,000 rows, the query returns 4,000 rows. My problem is it is very slow. There's no obvious way to rewrite the query in a more optimal way. It only does equi-joins on indexed columns. I tried giving it hints but they are not used. As a comparison, I load the same set in PostgreSQL and it returns in < 1 second. The query plan of YB vs Postgres on this is completely different. I used a single instance postgres vs 3 masters and 3 tservers, latest version, all VMs have 1 cpu and 3 GB memory. I realize the cpu/memory are on the low side but they are not fully used, I would run it on bigger instances if that were the bottleneck. Can you give some tips? Where do I look to optimise this query? What's your experience with queries that do many joins?
Query plan below:
explain (costs off, analyze, verbose) SELECT
a.ptflio_clstr_mstr_key,
a.ptflio_clstr_nm,
a.reference_id,
pku.kpc_usr_nm,
a.chng_usr,
clstrhist.chng_dttm,
a.ptflio_clstr_desc,
b.ptflio_bld_blck_grp_mstr_key,
b.ptflio_bld_blck_grp_nm,
f.prd_offr_mstr_key,
f.prd_offr_nm,
d.atmc_prd_offr_type_ind,
offrhist.chng_dttm
FROM product_store.pm_ptflio_clstr a
LEFT JOIN product_store.pm_ptflio_clstr_hist clstrhist ON a.ptflio_clstr_mstr_key = clstrhist.ptflio_clstr_mstr_key AND clstrhist.row_seq=1
LEFT JOIN product_store.pm_kpc_usr pku on a.kpc_usr_id = pku.kpc_usr_id
JOIN product_store.pm_ptflio_bld_blck_grp b ON a.ptflio_clstr_mstr_key = b.ptflio_clstr_mstr_key
JOIN product_store.pm_ptflio_bld_blck c ON b.ptflio_bld_blck_grp_mstr_key = c.ptflio_bld_blck_grp_mstr_key
LEFT JOIN product_store.pm_atmc_prd_offr d ON c.ptflio_bld_blck_mstr_key = d.ptflio_bld_blck_mstr_key
LEFT JOIN product_store.pm_prd_offr f ON f.prd_offr_mstr_key = d.atmc_prd_offr_mstr_key
LEFT JOIN product_store.pm_prd_offr_hist offrhist ON f.prd_offr_mstr_key = offrhist.prd_offr_mstr_key AND offrhist.row_seq = 1
WHERE a.ptflio_clstr_mstr_key='b3000fbe-65d5-4c68-9758-0954c7f9a0f1';
Nested Loop Left Join (actual time=12.719..10427.074 rows=3835 loops=1)
Output: a.ptflio_clstr_mstr_key, a.ptflio_clstr_nm, a.reference_id, pku.kpc_usr_nm, a.chng_usr, clstrhist.chng_dttm, a.ptflio_clstr_desc, b.ptflio_bld_blck_grp_mstr_key, b.ptflio_bld_blck_grp_nm, f.prd_offr_mstr_key, f.prd_offr_nm, d.atmc_prd_offr_type_ind, offrhist.chng_dttm
Inner Unique: true
-> Nested Loop Left Join (actual time=12.338..8896.474 rows=3835 loops=1)
Output: a.ptflio_clstr_mstr_key, a.ptflio_clstr_nm, a.reference_id, a.chng_usr, a.ptflio_clstr_desc, clstrhist.chng_dttm, pku.kpc_usr_nm, b.ptflio_bld_blck_grp_mstr_key, b.ptflio_bld_blck_grp_nm, d.atmc_prd_offr_type_ind, f.prd_offr_mstr_key, f.prd_offr_nm
Inner Unique: true
-> Nested Loop (actual time=11.903..7197.184 rows=3835 loops=1)
Output: a.ptflio_clstr_mstr_key, a.ptflio_clstr_nm, a.reference_id, a.chng_usr, a.ptflio_clstr_desc, clstrhist.chng_dttm, pku.kpc_usr_nm, b.ptflio_bld_blck_grp_mstr_key, b.ptflio_bld_blck_grp_nm, d.atmc_prd_offr_type_ind, d.atmc_prd_offr_mstr_key
-> Nested Loop Left Join (actual time=1.755..1.759 rows=1 loops=1)
Output: a.ptflio_clstr_mstr_key, a.ptflio_clstr_nm, a.reference_id, a.chng_usr, a.ptflio_clstr_desc, clstrhist.chng_dttm, pku.kpc_usr_nm
Inner Unique: true
-> Nested Loop Left Join (actual time=1.298..1.301 rows=1 loops=1)
Output: a.ptflio_clstr_mstr_key, a.ptflio_clstr_nm, a.reference_id, a.chng_usr, a.ptflio_clstr_desc, a.kpc_usr_id, clstrhist.chng_dttm
Inner Unique: true
Join Filter: (a.ptflio_clstr_mstr_key = clstrhist.ptflio_clstr_mstr_key)
-> Index Scan using xpkportfolio_cluster on product_store.pm_ptflio_clstr a (actual time=0.764..0.767 rows=1 loops=1)
Output: a.ptflio_clstr_mstr_key, a.row_seq, a.ptflio_clstr_nm, a.ptflio_clstr_desc, a.mstr_stat_cd, a.chng_usr, a.chng_dttm, a.chng_rmrk, a.reference_id, a.kpc_usr_id
Index Cond: (a.ptflio_clstr_mstr_key = 'b3000fbe-65d5-4c68-9758-0954c7f9a0f1'::uuid)
-> Index Scan using pm_ptflio_clstr_hist_pkey on product_store.pm_ptflio_clstr_hist clstrhist (actual time=0.529..0.529 rows=0 loops=1)
Output: clstrhist.ptflio_clstr_mstr_key, clstrhist.row_seq, clstrhist.ptflio_clstr_nm, clstrhist.ptflio_clstr_desc, clstrhist.mstr_stat_cd, clstrhist.chng_usr, clstrhist.chng_dttm, clstrhist.chng_rmrk, clstrhist.reference_id, clstrhist.kpc_usr_id
Index Cond: ((clstrhist.ptflio_clstr_mstr_key = 'b3000fbe-65d5-4c68-9758-0954c7f9a0f1'::uuid) AND (clstrhist.row_seq = 1))
-> Index Scan using xpkkpc_user on product_store.pm_kpc_usr pku (actual time=0.453..0.453 rows=0 loops=1)
Output: pku.kpc_usr_id, pku.ruisnaam, pku.kpc_usr_nm, pku.kpc_usr_act_ind, pku.chng_usr, pku.chng_dttm
Index Cond: (a.kpc_usr_id = pku.kpc_usr_id)
-> Nested Loop (actual time=10.131..7193.091 rows=3835 loops=1)
Output: b.ptflio_bld_blck_grp_mstr_key, b.ptflio_bld_blck_grp_nm, b.ptflio_clstr_mstr_key, d.atmc_prd_offr_type_ind, d.atmc_prd_offr_mstr_key
Inner Unique: true
-> Hash Right Join (actual time=9.673..73.707 rows=16878 loops=1)
Output: c.ptflio_bld_blck_grp_mstr_key, d.atmc_prd_offr_type_ind, d.atmc_prd_offr_mstr_key
Inner Unique: true
Hash Cond: (d.ptflio_bld_blck_mstr_key = c.ptflio_bld_blck_mstr_key)
-> Seq Scan on product_store.pm_atmc_prd_offr d (actual time=8.124..45.709 rows=16865 loops=1)
Output: d.atmc_prd_offr_mstr_key, d.row_seq, d.ptflio_bld_blck_mstr_key, d.lcm_phase_cd, d.lcm_phase_start_dttm, d.lcm_phase_end_dttm, d.atmc_prd_offr_type_ind, d.chng_usr, d.chng_dttm, d.lcm_phase_desc, d.lcm_phase_alert_dttm, d.lcm_phase_approved_by, d.lcm_phase_master_key
-> Hash (actual time=1.534..1.534 rows=43 loops=1)
Output: c.ptflio_bld_blck_grp_mstr_key, c.ptflio_bld_blck_mstr_key
Buckets: 1024 Batches: 1 Memory Usage: 11kB
-> Seq Scan on product_store.pm_ptflio_bld_blck c (actual time=0.531..1.523 rows=43 loops=1)
Output: c.ptflio_bld_blck_grp_mstr_key, c.ptflio_bld_blck_mstr_key
-> Index Scan using xpkportfolio_building_block_gr on product_store.pm_ptflio_bld_blck_grp b (actual time=0.403..0.403 rows=0 loops=16878)
Output: b.ptflio_bld_blck_grp_mstr_key, b.row_seq, b.ptflio_bld_blck_grp_nm, b.ptflio_bld_blck_grp_desc, b.ptflio_clstr_mstr_key, b.mstr_stat_cd, b.chng_usr, b.chng_dttm, b.chng_rmrk, b.reference_id, b.kpc_usr_id
Index Cond: (b.ptflio_bld_blck_grp_mstr_key = c.ptflio_bld_blck_grp_mstr_key)
Filter: (b.ptflio_clstr_mstr_key = 'b3000fbe-65d5-4c68-9758-0954c7f9a0f1'::uuid)
Rows Removed by Filter: 1
-> Index Scan using xpkproduct_offering on product_store.pm_prd_offr f (actual time=0.419..0.419 rows=1 loops=3835)
Output: f.prd_offr_mstr_key, f.row_seq, f.prd_offr_nm, f.prop_mod_mstr_key, f.clstr_dsct_prd_offr_grp_cd, f.trgt_ptflio_ind, f.price_brd_nmbr, f.price_brd_stat_cd, f.prd_offr_lnup, f.pm_prd_offr_type_cd, f.comm_prd_id, f.chng_usr, f.chng_dttm, f.reference_id, f.version, f.kpc_usr_id, f.generation
Index Cond: (f.prd_offr_mstr_key = d.atmc_prd_offr_mstr_key)
-> Index Scan using pm_prd_offr_hist_pkey on product_store.pm_prd_offr_hist offrhist (actual time=0.375..0.375 rows=0 loops=3835)
Output: offrhist.prd_offr_mstr_key, offrhist.row_seq, offrhist.prd_offr_nm, offrhist.prop_mod_mstr_key, offrhist.clstr_dsct_prd_offr_grp_cd, offrhist.trgt_ptflio_ind, offrhist.price_brd_nmbr, offrhist.price_brd_stat_cd, offrhist.prd_offr_lnup, offrhist.pm_prd_offr_type_cd, offrhist.comm_prd_id, offrhist.chng_usr, offrhist.chng_dttm, offrhist.reference_id, offrhist.version, offrhist.kpc_usr_id, offrhist.generation
Index Cond: ((f.prd_offr_mstr_key = offrhist.prd_offr_mstr_key) AND (offrhist.row_seq = 1))
Planning Time: 0.777 ms
Execution Time: 10655.916 ms

Answer 1

The general approach to query tuning is called ‘query tuning by eliminating throwaway” http://docplayer.net/20177036-Query-tuning-by-eliminating-throwaway.html, which is the name of a paper that was written by Martin Berg from Denmark (for the Oracle database, but the general methodology holds true for all databases that use query plans with rowsources). On top of that, there are some specifics for distributed databases like YugabyteDB, where rowsources like nested loops can cause higher overhead than a monolithic database, because in a monolith, everything is local.
The pg_hint_plan extension required for making hints work is not enabled in YugabyteDB by default, but it is installed in the YugabyteDB server software. This means to enable it, you need to run create extension pg_hint_plan; in one YugabyteDB database to enable it for the cluster.
Of the ~ 10 seconds, ~ 6 seconds sits in this join:
JOIN product_store.pm_ptflio_bld_blck c ON b.ptflio_bld_blck_grp_mstr_key = c.ptflio_bld_blck_grp_mstr_key
using the index xpkportfolio_building_block_gr.
Reading product_store.pm_ptflio_bld_blck_grp b 16878 times to find no rows. Probably better to start with it (I guess the predicate on b.ptflio_clstr_mstr_key = 'b3000fbe-65d5-4c68-9758-0954c7f9a0f1'::uuid is highly selective):
/*+ Leading( (b c) ) */ may be a good start but may not be possible because of outer join.
So maybe /*+ Leading( (c b) ) HashJoin(c b) */.
Depends on your data. The goal is to start with the most selective table, then join to those that reduces the number of rows, then the others.