Apache Spark SQL: COALESCE NULL array into empty struct array - apache-spark

I have query
SELECT foo FROM bar;
Foo in this case is an array of structs that can be NULL. I would like to coalesce it into an empty array.
SELECT COALESCE(foo, array()) FROM bar;
When I do that I get an error that there is a mismatch.
cannot resolve 'coalesce(foo, array())' due to data type mismatch: input to function coalesce should all be the same type, ...
The struct has about 25 fields so if possible I don't want to manually define them unless there is no other way. The reason why I don't want NULL is because when I write it to JSON I want the field to be an empty array and with NULL it's missing completely.

Related

How to convert data type if Variant.Type is known?

How do I convert the data type if I know the Variant.Type from typeof()?
for example:
var a=5;
var b=6.9;
type_cast(b,typeof(a)); # this makes b an int type value
How do I convert the data type if I know the Variant.Type from typeof()?
You can't. GDScript does not have generics/type templates, so beyond simple type inference, there is no way to specify a type without knowing the type.
Thus, any workaround to cast the value to a type only known at runtime would have to be declared to return Variant, because there is no way to specify the type.
Furthermore, to store the result on a variable, how do you declare the variable if you don't know the type?
Let us have a look at variable declarations. If you do not specify a type, you get a Variant.
For example in this code, a is a Variant that happens to have an int value:
var a = 5
In this other example a is an int:
var a:int = 5
This is also an int:
var a := 5
In this case the variable is typed according to what you are using to initialized, that is the type is inferred.
You may think you can use that like this:
var a = 5
var b := a
Well, no. That is an error. "The variable type can't be inferred". As far as Godot is concerned a does not have a type in this example.
I'm storing data in a json file: { variable:[ typeof(variable), variable_value ] } I added typeof() because for example I store an int but when I reassign it from the file it gets converted to float (one of many other examples)
It is true that JSON is not good at storing Godot types. Which is why many authors do not recommend using JSON to save state.
Now, be aware that we can't get a variable with the right type as explained above. Instead we should try to get a Variant of the right type.
If you cannot change the serialization format, then you are going to need one big match statement. Something like this:
match type:
TYPE_NIL:
return null
TYPE_BOOL:
return bool(value)
TYPE_INT:
return int(value)
TYPE_REAL:
return float(value)
TYPE_STRING:
return str(value)
Those are not all the types that a Variant can hold, but I think it would do for JSON.
Now, if you can change the serialization format, then I will suggest to use str2var and var2str.
For example:
var2str(Vector2(1, 10))
Will return a String value "Vector2( 1, 10 )". And if you do:
str2var("Vector2( 1, 10 )")
You get a Variant with a Vector2 with 1 for the x, and 10 for the y.
This way you can always store Strings, in a human readable format, that Godot can parse. And if you want to do that for whole objects, or you want to put them in a JSON structure, that is up to you.
By the way, you might also be interested in ResourceFormatSaver and ResourceFormatLoader.

Initiialize an empty tuple scala

I want to create a mutable tuple in scala (String, Set[String]).
And before I want to initialize it to an empty value first before adding to it
the set can not be initialized to empty set. The same for String.
I'm getting an error saying class java.lang.String is not a value
Is there a simple way to do that ? and What am I doing wrong
Empty Set can be defined like so
Set.empty[String] // because assert(Set.empty[String].isEmpty)
Empty String can be defined like so
"" // because assert("".isEmpty)
Empty tuple is not something Scala models as TupleN, I think. Instead it uses Unit type for it. Perhaps you are after something like so
val ta: (String, Set[String]) = ("", Set.empty)
val tb: (Option[String], Option[Set[String]]) = (None, None)
val tc: Option[(String, Set[String])] = None
The error message
class java.lang.String is not a value
means you are using a type where value is expected, for example, consider the difference between
Set[String] // ok
Set(String) // error

Big query update struct to null in repeated field

In Google bigquery, I'm trying to do an update on a repeated field.
For comparison, this works (or at least is flagged as valid), but of course isn't actually updating the field.
UPDATE my.table t
SET my_field = ARRAY(
SELECT AS STRUCT g.foo, g.bar, g.struct_to_set_null
FROM unnest(t.groups) as g
), ... FROM ... etc
Setting struct_to_set_null to null gives an error:
UPDATE my.table t
SET my_field = ARRAY(
SELECT AS STRUCT g.foo, g.bar, null as struct_to_set_null
FROM unnest(t.groups) as g
), ... FROM ... etc
Value of type ARRAY<STRUCT<... (really long and cut off) cannot be assigned to groups, which has type <ARRAY,STRUCT<... (same, really long, cut off)
I can see that the field in question is of type RECORD and NULLABLE, so I would think setting it to null is allowed. Is there a trick to getting this to work?
The problem is that BigQuery isn't inferring the type of the struct field just from the NULL literal; you need to be a bit more explicit. Here is an example:
CREATE TABLE tmp_elliottb.UpdateTable (
my_field ARRAY<STRUCT<foo INT64, bar INT64, struct_to_set_null STRUCT<x STRING, y BOOL, z INT64>>>
);
UPDATE tmp_elliottb.UpdateTable
SET my_field = ARRAY(
SELECT AS STRUCT foo, bar, NULL AS struct_to_set_null FROM UNNEST(my_field)
)
WHERE true;
This gives me:
Value of type ARRAY<STRUCT<foo INT64, bar INT64, struct_to_set_null INT64>> cannot be assigned to my_field, which has type ARRAY<STRUCT<foo INT64, bar INT64, struct_to_set_null STRUCT<x STRING, y BOOL, z INT64>>> at [4:16]
What I can do instead is to use an IF expression that produces NULL, but has struct_to_set_null on one of the branches in order to force the output type that I want:
UPDATE tmp_elliottb.UpdateTable
SET my_field = ARRAY(
SELECT AS STRUCT
foo, bar,
IF(false, struct_to_set_null, NULL) AS struct_to_set_null
FROM UNNEST(my_field)
)
WHERE true;
Or, alternatively, I can use SELECT * REPLACE:
UPDATE tmp_elliottb.UpdateTable
SET my_field = ARRAY(
SELECT AS STRUCT * REPLACE (IF(false, struct_to_set_null, NULL) AS struct_to_set_null )
FROM UNNEST(my_field)
)
WHERE true;
Repeated is an Array type, so it cannot be set to NULL.
Currently, BigQuery has two following limitations with respect to NULLs and ARRAYs:
BigQuery raises an error if query result has ARRAYs which contain NULL elements, although such ARRAYs can be used inside the query.
BigQuery translates NULL ARRAY into empty ARRAY in the query result, although inside the query NULL and empty ARRAYs are two distinct values.

How do I get an Option<T> instead of an Option<Vec<T>> from a Diesel query which only returns 1 or 0 records?

I'm querying for existing records in a table called messages; this query is then used as part of a 'find or create' function:
fn find_msg_by_uuid<'a>(conn: &PgConnection, msg_uuid: &Uuid) -> Option<Vec<Message>> {
use schema::messages::dsl::*;
use diesel::OptionalExtension;
messages.filter(uuid.eq(msg_uuid))
.limit(1)
.load::<Message>(conn)
.optional().unwrap()
}
I've made this optional as both finding a record and finding none are both valid outcomes in this scenario, so as a result this query might return a Vec with one Message or an empty Vec, so I always end up checking if the Vec is empty or not using code like this:
let extant_messages = find_msg_by_uuid(conn, message_uuid);
if !extant_messages.unwrap().is_empty() { ... }
and then if it isnt empty taking the first Message in the Vec as my found message using code like
let found_message = find_msg_by_uuid(conn, message_uuid).unwrap()[0];
I always take the first element in the Vec since the records are unique so the query will only ever return 1 or 0 records.
This feels kind of messy to me and seems to take too many steps, I feel as if there is a record for the query then it should return Option<Message> not Option<Vec<Message>> or None if there is no record matching the query.
As mentioned in the comments, use first:
Attempts to load a single record. Returns Ok(record) if found, and Err(NotFound) if no results are returned. If the query truly is optional, you can call .optional() on the result of this to get a Result<Option<U>>.
fn find_msg_by_uuid<'a>(conn: &PgConnection, msg_uuid: &Uuid) -> Option<Message> {
use schema::messages::dsl::*;
use diesel::OptionalExtension;
messages
.filter(uuid.eq(msg_uuid))
.first(conn)
.optional()
.unwrap()
}

Is there a way to set a null value to decimal datatype in C# without using Nullable<decimal> or 'decimal?'?

public decimal Value { get; set; }
I want to set a null value to Value property without using Nullable or '?'
You are dealing with a Value Type. Value Types can't have NULL values. Imagine a value type like being an array of integers.
For example, a Point having two properties X=12 and Y=15 will be conceptually represented in memory like:
[12,15]
If you add let's say a "string" property labeled name "Name" then in the array will be stored the memory address pointing in Heap to that string:
[12,15, 65535] where 65535 represents the heap address.
Taking this in consideration, a Value Type obviously can't be null, in the best case you will have an array full of zero values.
a Nullable is a value type that has actually two properties: the represented T value type and an additional boolean value that says whether conceptually we have a value or not. So we still have a structure but this time we have an additional flag that indicates the "nullability" property.

Resources