Get length of Impala table varchar columns from Spark - apache-spark

Question

Try to get in Spark length of varchar columns, declared in following Impala table:
show create table dev_dm_rco.f102
Result:
CREATE TABLE dev_dm_rco.f102 (
rep_id INT,
msr_prd_id INT,
...
bank_code VARCHAR(9),
tu_cd VARCHAR(2),
pl_stmt_num VARCHAR(5)'
)
PARTITIONED BY (
report_date VARCHAR(50)
)
STORED AS PARQUET
LOCATION ...
TBLPROPERTIES ('spark.sql.create.version'='2.2 or prior', 'spark.sql.sources.schema.numPartCols'='1', 'spark.sql.sources.schema.numParts'='1'...)
First try- using Spark API:
String f102SqlQuery = "select * from dev_dm_rco.f102";
Dataset<Row> f102Dataset = SparkService.sql(f102SqlQuery);
for(StructField sf:f102Dataset.schema().fields())
log.info("f102dataset sf.name()="+sf.name()+" sf.dataType().typeName()="+sf.dataType().typeName());
Varchar columns returned as string without length:
f102dataset sf.name()=rep_id sf.dataType().typeName()=integer
f102dataset sf.name()=msr_prd_id sf.dataType().typeName()=integer
...
f102dataset sf.name()=bank_code sf.dataType().typeName()=string
...
f102dataset sf.name()=tu_cd sf.dataType().typeName()=string
f102dataset sf.name()=pl_stmt_num sf.dataType().typeName()=string
f102dataset sf.name()=report_date sf.dataType().typeName()=string
Second try- using show create table:
String f102CreateTable = "show create table dev_dm_rco.f102";
Dataset<Row> crtF102Stmt = sqlContext.sql(f102CreateTable);
log.info(f102CreateTable+": "+ crtF102Stmt.collectAsList());
Again all varchar columns returned as string with no length:
show create table dev_dm_rco.f102:
[[CREATE TABLE `dev_dm_rco`.`f102`
(`rep_id` int,
`msr_prd_id` int,
...
`bank_code` string,
`tu_cd` string,
`pl_stmt_num` string')
PARTITIONED BY (`report_date` string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
WITH SERDEPROPERTIES (
)
STORED AS
INPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
TBLPROPERTIES (
'transient_lastDdlTime' = '1596083577'
)
]]
Is there any way to get length of varchar fields declared in Impala table, using Spark?
Spark version: 2.3.1
Thanks!

Answer 1

you could try:
show column stats dev_dm_rco.f102
for each column, it should return col name, data type, #distinct values, #nulls, max size/varchar length, and avg size/varchar length
hope this helps.