Teradata has a function called ZEROIFNULL, which does what the name suggests, if the value of a column is NULL, it returns zero. On similar lines, there's a function called NULLIFZERO as well.
I want to imitate/mock these functionalities in SparkSQL(not using the dataframe or RDD APIs, instead, I want to use them in SparkSQL, where you directly pass the SQLs.)
Any ideas?
Try
sqlContext.sql("select COALESCE(column,0)")
Returns zero if column is NULL.
To mimic NULLIFZERO, you could use case when
select case when col=0 then NULL end from tbl
Related
I try to translate a legacy query into a standard SQL query in bigquery, but I don't find the function position() in standard SQL.
You may be looking for the bracket operator. For example,
SELECT array_column[OFFSET(0)]
FROM dataset.table
This selects the first element of an array column for each row. If you want to flatten an array and get the offset of each element, you can do so like this:
SELECT x, x_offset
FROM dataset.table,
UNNEST(array_column) AS x WITH OFFSET x_offset
See also the working with arrays documentation.
I am a newbie in Spark.I want to write the dataframe data into hive table. Hive table is partitioned on mutliple column. Through, Hivemetastore client I am getting the partition column and passing that as a variable in partitionby clause in write method of dataframe.
var1="country","state" (Getting the partiton column names of hive table)
dataframe1.write.partitionBy(s"$var1").mode("overwrite").save(s"$hive_warehouse/$dbname.db/$temp_table/")
When I am executing the above code,it is giving me error partiton "country","state" does not exists.
I think it is taking "country","state" as a string.
Can you please help me out.
The partitionBy function takes a varargs not a list. You can use this as
dataframe1.write.partitionBy("country","state").mode("overwrite").save(s"$hive_warehouse/$dbname.db/$temp_table/")
Or in scala you can convert a list into a varargs like
val columns = Seq("country","state")
dataframe1.write.partitionBy(columns:_*).mode("overwrite").save(s"$hive_warehouse/$dbname.db/$temp_table/")
I know, that Cassandra allows to GROUP BY and can run UDF on that data.
Is there any default function to get the first row of each aggregated set?
(How) Can I stop processing data and return result from my UDF immediately (E.G. After 1 or few rows processed)?
Now I'm using ... COUNT(1) ... as workaround.
Actualy You fon't need any UDF. It works as described out of the box.
Jusr GROUP BY fields you need.
I'm using Cognos Framework Manager and I'm creating a Data Item for a dynamic sort. I'm creating the Data Item using a CASE WHEN, here's my sample code:
CASE #prompt('SortOrder', 'string')#
WHEN 'Date' THEN <Date Column>
WHEN 'ID' THEN <String Column>
END
I'm getting this error QE-DEF-0405 Incompatible data types in case statement. Although I can cast the date column into a string wouldn't that make sort go wrong for the 'date' option? Should I cast the date column in a different way, cast the whole case, or am I barking at the wrong tree? In line with my question, should there be a general rule when creating dynamic columns via CASE with multiple column data types?
Column in Framework Manager should have datatype. Only one datatype.
So you need to cast your date column to correctly sortable string.
E.g. 'yyyy-mm-dd' format.
You are using the two different types of data format, so in prompt function use token instead of string (#prompt('sortorder','token')#)
Maybe I'm missing something simple, but is there a way to write a nested query in AX? I tried some syntax I thought would work, but with no luck.
The following standard SQL statement would accomplish what I'm trying to do, but I need to do this in AX, not SQL.
SELECT table1.column1A, table1.column1B,
(SELECT Top 1 column2B FROM table2
WHERE table1.column1A = table2.column2A
ORDER BY table2.column1A)
AS lookupResult
FROM table1
My problem is that table1 has a one-to-many relationship with table2, and since AX doesn't have a DISTINCT function that I'm aware of, I receive many copies of each record when using a JOIN statement.
Thanks
Nested queries are not supported in AX.
One way to bypass the missing distinct is to use group by (assuming max value of column2B is interesting):
while select column1A, column1B from table1
group column1A, column1B
join max-of(column2B) from table2
where table2.column2A == table1.column1A
{
...
}
Another method would be use a display method on table1 in the form or report.
display ColumnB column2B()
{
return (select max-of(column2B) from table2
where table2.column2A == this.column1A).column2A;
}
The performance is inferior to the first solution, but it may be acceptable.
As mentioned in the previous reply, group-by is the closest you can get to a distinct function. If you need a simpler query for some reason, or if you need a table or query object to use as a datasource on a form or report, you may entertain the idea of creating a view in the AOT, which contains the group-by. You can then use that view to easily join to on a query object or form datasource etc...
Ax2012 has support of computed columns in views, you can use the SysComputedColumn class to build query you want