I came across this interesting date conversion table for postgresql and am wondering if this syntax for date patterns can also be used in presto. It might or might not work, anyone familiar with this?
Equivalent functionality is available through the date_format() Trino (formerly Presto) function
The to_char function does not exist in Presto.
Documentation contains the list of builtin Presto functions. When in doubt, you can get a list of all the functions available using the SHOW FUNCTIONS statement.
Related
We store very complex json in one of our table columns. I would like to write a parser for this. I was reading thru table functions and functions but I never saw a great guide that would tell me how to create a function / deploy it to our cluster. Does anyone have any good pointers.
I'm working with data in AWS Athena, and I'm trying to match the structure of some input data. This involves a nested structure where "from" is a key. This consistently throws errors.
I've narrowed the issue down to the fact that Athena queries don't work when you try to use reserved keywords as keys in rows. The following examples demonstrate this behavior.
This simple case, SELECT CAST(ROW(1) AS ROW("from" INTEGER)), fails with the following error: GENERIC_INTERNAL_ERROR: Unable to create class com.facebook.presto.execution.TaskInfo from JSON response: [io.airlift.jaxrs.JsonMapperParsingException: Invalid json for Java type
This simple case runs successfully: SELECT CAST(ROW(1) AS ROW("work" INTEGER))
The Athena documentation says to enclose reserved keywords in double quotes to use them in SELECT statements, but the examples above show that queries still fail when using keywords as keys in rows.
I know I have other options, but this way is by far the most convenient. Is there a way to use reserved keywords in this scenario?
As Piotr mentions in a comment, this is a Presto bug and given that it was posted just days ago it's unlikely to be fixed in Athena anytime soon. When the bug is fixed in Presto it might find its way into Athena, I know the Athena team sometimes apply upstream patches even though Athena is based on an old version of Presto. This one might not be significant enough to appear on their radar, but if you open a support ticket with AWS it might happen (be sure to be clear that you don't need any workaround just report a bug, otherwise you'll have a support person spending way too much time trying to help you to turn things off and on again).
I just got introduced to Spark SQL higher order functions transform(), filter() etc. I searched online, but couldn't find much advanced use-cases leveraging these functions.
Can anyone please explain transform() with a couple of advanced real-life use-case using sql query. Does it always need to work on nested complex types (arrays, struct etc) ? Or can it be used to process simple data-type records as well ?
Any help is appreciated.
Thanks
The following online resource vividly demonstrates in %sql mode :
https://docs.databricks.com/delta/data-transformation/higher-order-lambda-functions.html
The Spark version 1.5+ has windowing functions. I believe there were a comprehensive documentation for SQL somewhere but have been unsuccessful to find it .
Here is the docs for spark dataframe and sql: it does NOT have the content sought:
http://spark.apache.org/docs/latest/streaming-programming-guide.html#dataframe-and-sql-operations
I have googled a number of different ways and unable to find the comprehensive guide to available sql functions. The closest I could find is "spark 1.5 new Dataframe operations" here:
https://databricks.com/blog/2015/09/16/spark-1-5-dataframe-api-highlights-datetimestring-handling-time-intervals-and-udafs.html
Update I am looking specifically for a SQL reference - not an API (/scaladoc) reference. I.e. a reference showing the provided sql functions, what their arguments are, semantics, and maybe example usage.
There is a page about Windowing and analytics in the Wiki which covers the window specification, aggregate functions, and it contains some examples.
How about this? This is spark2.4.0
https://spark.apache.org/docs/2.4.0/api/sql/index.html#last_value
Databricks had a good introduction to window functions at https://databricks.com/blog/2015/07/15/introducing-window-functions-in-spark-sql.html but the definitive documentation should always be the API docs, scroll right to the bottom
Dataframe functions API documentation
I am currently working with Presto 0.80. I have to write a user defined function to convert degree celsius to degree fahrenheit during select query. I did the same using Hive QL but was wondering if we can replicate the same in Facebook Presto.
Any help would be highly appreciated.
Thanks!!
Here is a guide for writing a new function in presto.
https://trino.io/docs/current/develop/functions.html
After writing your function, add the plugin to the plugin directory as explained in SPI Overview.
There is another example for writing presto UDF from Qubole blog.
http://www.qubole.com/blog/product/plugging-in-presto-udfs/
You can try it if you can make it work.