I've started using the play-json/play-json-compat libraries with reactivemongo 0.20.11.
So I can use JSON Play reads/writes while importing the 'reactivemongo.play.json._' package and then easily fetch data from a JSONCollection instead of a BSONCollection.
For most cases, this works great but for Long fields, it doesn't :(
For example:
case class TestClass(name: String, age: Long)
object TestClass {
implicit val reads = Json.reads[TestClass]
}
If I try querying using the following func:
def getData: Map[String, TestClass] = {
val res = collection.find(emptyDoc)
.cursor[TestClass]()
.collect[List](-1, Cursor.ContOnError[List[TestClass]] { case (_, t) =>
failureLogger.error(s"Failed deserializing TestClass from Mongo", t)
})
.map { items =>
items map { item =>
item.name -> item.age
} toMap
}
Await.result(res, 10 seconds)
}
Then I get the following error:
play.api.libs.json.JsResultException: JsResultException(errors:List((/age,List(ValidationError(List(error.expected.jsnumber),WrappedArray())))))
I've debugged the reading of the document and noticed that when it first converts the BSON to a JsObject, then the long field is as following:
"age": {"$long": 1526389200000}
I found a way to make this work but I really don't like it:
case class MyBSONLong(`$long`: Long)
object MyBSONLong {
implicit val longReads = Json.reads[MyBSONLong]
}
case class TestClass(name: String, age: Long)
object TestClass {
implicit val reads = (
(__ \ "name").read[String] and
(__ \ "age").read[MyBSONLong].map(_.`$long`)
) (apply _)
}
So this works, but it's a very ugly solution.
Is there a better way to do this?
Thanks in advance :)
Related
I have this code for deferring the execution of a function
export type DeferredFunction<T> = () => T | PromiseLike<T>;
export class Deferrable<T> {
protected df: DeferredFunction<T>;
constructor(df: DeferredFunction<T>) {
this.df = df;
}
public async execute(): Promise<T> {
return this.df();
}
}
export const defer = <T>(df: DeferredFunction<T>): Deferrable<T> => new Deferrable<T>(df);
That works fine and I can run code like
await defer(() => someFunction('foo', 'bar')).execute();
but I what I want to do is type DeferredFunction in a way that I can specify the inner function's signature but I can't get it working. In generic cases the above works but when I want to limit the arguments such that they are specific to a certain type of function I don't have that kind of control.
For clarity, I want to be able to type the inner function's inputs like (as an example)
export type InnerDeferredFunction<T> = (a: string, b: number, c: SomeObjectType) => T | PromiseLike<T>
Any help would be greatly appreciated!
What "inner function" are you talking about? Is it someFunction? If so then the type of DeferredFunction<T> has no handle on it, since it's a function called by the implementation of DeferredFunction<T>. There's no way in TypeScript to specify "a function whose implementation must call a function of type (x: string, y: number, z: boolean) => string". Implementation details are not part of a function's call signature.
The only way I can imagine to begin to approach this would be for DeferredFunction<T> to accept as a parameter the inner function you want to call, along with the list of arguments to call it with. This might not be what you're looking for, but it's the closest that the type system can represent.
Something like this:
export type InnerDeferredFunction<T, A extends any[]> = (...args: A) => T | PromiseLike<T>;
export type ZeroArgDeferredFunction<T> = InnerDeferredFunction<T, []>
Here I'm keeping A generic but you can specify it to some hardcoded list of arguments. I've renamed your DeferredFunction to ZeroArgDeferredFunction to be explicit that it doesn't need arguments.
But now Deferrable needs to know about T and A:
export class Deferrable<T, A extends any[]> {
protected df: ZeroArgDeferredFunction<T>;
constructor(df: InnerDeferredFunction<T, A>, ...args: A) {
this.df = () => df(...args);
}
public async execute(): Promise<T> {
return this.df();
}
}
And you can see that you have to construct one by passing it the inner function and its arguments, and the ZeroArgDeferredFunction is built inside the constructor and is not passed in.
There are different ways to define defer(). It could be a thin wrapper around new Deferrable the way you had it, or you could imagine splitting it up so that the args come first:
export const defer = <A extends any[]>(...args: A) => <T>(
df: InnerDeferredFunction<T, A>): Deferrable<T, A> => new Deferrable<T, A>(df, ...args);
And then you can test it like this:
function someFunction(x: string, y: string) {
return (x + y).length;
}
function anotherFunction(x: number, y: number) {
return (x * y).toFixed()
}
const deferFooBar = defer('foo', 'bar');
await deferFooBar(someFunction).execute(); // okay
await deferFooBar(anotherFunction); // error! string is not assignable to number
Once you call deferFooBar('foo', 'bar'), the returned value will only accept functions that can be safely called with the arguments foo and 'bar'. That means someFunction will be accepted and anotherFunction will be rejected.
Okay, hope that helps; good luck!
Playground link to code
I have the next method:
fun get(browsePlayerContext: BrowsePlayerContext): Single<List<Conference>>
Which returns a Single> with the next structure for the Conference object:
data class Conference(
val label: String,
val uid: UID?,
val action: BrowsePlayerAction?,
val image: String
)
But I need to transfor this response in a:
Single<List<EntityBrowse>>
The entity browse has the same structure I mean:
data class EntityBrowse(
val label: String,
val uid: UID?,
val action: BrowsePlayerAction?,
val image: String
)
I am doing the transformation manually, but I need a more sophisticated way, because I am gona get different kind of objects and I will have to do the same transformation to EntityBrowse.
Any ideas?
You could use the .map function to transform the Conference objects into EntityBrowse objects
val conferences: List<Conference> = getConferences()
val entities: List<Entities> = conferences.map {conference ->
EntityBrowse(conference.label, conference.uid, conference.action, conference.image)
}
You can use map function on Single object to transform Single<List<Conference>> to Single<List<EntityBrowse>>:
val result: Single<List<EntityBrowse>> = get(context).map { conferences: List<Conference> ->
// transform List<Conference> to List<EntityBrowse> using `conferences` variable
conferences.map { EntityBrowse(it.label, it.uid, it.action, it.image) }
}
Working on an internal website where the URL contains the source reference from other systems. This is a business requirement and cannot be changed.
i.e. "http://localhost:9000/source.address.com/7808/project/repo"
"http://localhost:9000/build.address.com/17808/project/repo"
I need to remove these strings from the "project/repo" string/variables using a trait so this can be used natively from multiple services. I also want to be able to add more sources to this list (which already exists) and not modify the method.
"def normalizePath" is the method accessed by services, 2 non-ideal but reasonable attempts so far. Getting stuck on a on using foldLeft which I woudl like some help with or an simpler way of doing the described. Code Samples below.
1st attempt using an if-else (not ideal as need to add more if/else statements down the line and less readable than pattern match)
trait NormalizePath {
def normalizePath(path: String): String = {
if (path.startsWith("build.address.com/17808")) {
path.substring("build.address.com/17808".length, path.length)
} else {
path
}
}
}
and 2nd attempt (not ideal as likely more patterns will get added and it generates more bytecode than if/else)
trait NormalizePath {
val pattern = "build.address.com/17808/"
val pattern2 = "source.address.com/7808/"
def normalizePath(path: String) = path match {
case s if s.startsWith(pattern) => s.substring(pattern.length, s.length)
case s if s.startsWith(pattern2) => s.substring(pattern2.length, s.length)
case _ => path
}
}
Last attempt is to use an address list(already exists elsewhere but defined here as MWE) to remove occurrences from the path string and it doesn't work:
trait NormalizePath {
val replacements = (
"build.address.com/17808",
"source.address.com/7808/")
private def remove(path: String, string: String) = {
path-string
}
def normalizePath(path: String): String = {
replacements.foldLeft(path)(remove)
}
}
Appreciate any help on this!
If you are just stripping out those strings:
val replacements = Seq(
"build.address.com/17808",
"source.address.com/7808/")
replacements.foldLeft("http://localhost:9000/source.address.com/7808/project/repo"){
case(path, toReplace) => path.replaceAll(toReplace, "")
}
// http://localhost:9000/project/repo
If you are replacing those string by something else:
val replacementsMap = Seq(
"build.address.com/17808" -> "one",
"source.address.com/7808/" -> "two/")
replacementsMap.foldLeft("http://localhost:9000/source.address.com/7808/project/repo"){
case(path, (toReplace, replacement)) => path.replaceAll(toReplace, replacement)
}
// http://localhost:9000/two/project/repo
The replacements collection can come from elsewhere in the code and will not need to be redeployed.
// method replacing by empty string
def normalizePath(path: String) = {
replacements.foldLeft(path){
case(startingPoint, toReplace) => startingPoint.replaceAll(toReplace, "")
}
}
normalizePath("foobar/build.address.com/17808/project/repo")
// foobar/project/repo
normalizePath("whateverPath")
// whateverPath
normalizePath("build.address.com/17808build.address.com/17808/project/repo")
// /project/repo
A very simple replacement could be made as follows:
val replacements = Seq(
"build.address.com/17808",
"source.address.com/7808/")
def normalizePath(path: String): String = {
replacements.find(path.startsWith(_)) // find the first occurrence
.map(prefix => path.substring(prefix.length)) // remove the prefix
.getOrElse(path) // if not found, return the original string
}
Since the expected replacements are very similar, have you tried to generalize them and use regex matching?
There are a million and one ways to extract /project/repo from a String in Scala. Here are a few I came up with:
val list = List("build.address.com/17808", "source.address.com/7808") //etc
def normalizePath(path: String) = {
path.stripPrefix(list.find(x => path.contains(x)).getOrElse(""))
}
Output:
scala> normalizePath("build.address.com/17808/project/repo")
res0: String = /project/repo
val list = List("build.address.com/17808", "source.address.com/7808") //etc
def normalizePath(path: String) = {
list.map(x => if (path.contains(x)) {
path.takeRight(path.length - x.length)
}).filter(y => y != ()).head
}
Output:
scala> normalizePath("build.address.com/17808/project/repo")
res0: Any = /project/repo
val list = List("build.address.com/17808", "source.address.com/7808") //etc
def normalizePath(path: String) = {
list.foldLeft(path)((a, b) => a.replace(b, ""))
}
Output:
scala> normalizePath("build.address.com/17808/project/repo")
res0: String = /project/repo
Depends how complicated you want your code to look (or how silly you want to be), really. Note that the second example has return type Any, which might not be ideal for your scenario. Also, these examples aren't meant to be able to just take the String out of the middle of your path... they can be fairly easily modified if you want to do that though. Let me know if you want me to add some examples just stripping things like build.address.com/17808 out of a String - I'd be happy to do so.
How to convert RDD[Array[Row]] to RDD[Row]?
Details:
I have some use case where my parsing function returns type Array[Row] for some data and Row for some data. How will I convert both of these to RDD[Row] for further use?
CODE SAMPLE
private def getRows(rdd: RDD[String], parser: Parser): RDD[Row] = {
var processedLines = rdd.map { line =>
map(p => parser.processBeacon(line) }
val rddOfRowsList = processedLines.map { x =>
x match {
case Right(obj) => obj.map { p =>
MyRow.getValue(p)
}//I can use flatmap here
case Left(obj) =>
MyRow.getValue(obj)
}//Cant use flatmap here
}
// Here I have to convert rddOfRowsList to RDD[Row]
//?????
val rowsRdd =?????
//
rowsRdd
}
def processLine(logMap: Map[String, String]):Either[Map[String, Object], Array[Map[String, Object]]] =
{
//process
}
Use flatMap;
rdd.flatMap(identity)
You ca use flatmap to get new rdd, and then use union to compose them.
use flatMap to flattern the contents of RDD
I'm using Spark 2.0.2. I have a DataFrame that has an alias on it, and I'd like to be able to retrieve that. A simplified example of why I'd want that is below.
def check(ds: DataFrame) = {
assert(ds.count > 0, s"${df.getAlias} has zero rows!")
}
The above code of course fails because DataFrame has no getAlias function. Is there a way to do this?
You can try something like this but I wouldn't go so far to claim it is supported:
Spark < 2.1:
import org.apache.spark.sql.catalyst.plans.logical.SubqueryAlias
import org.apache.spark.sql.Dataset
def getAlias(ds: Dataset[_]) = ds.queryExecution.analyzed match {
case SubqueryAlias(alias, _) => Some(alias)
case _ => None
}
Spark 2.1+:
def getAlias(ds: Dataset[_]) = ds.queryExecution.analyzed match {
case SubqueryAlias(alias, _, _) => Some(alias)
case _ => None
}
Example usage:
val plain = Seq((1, "foo")).toDF
getAlias(plain)
Option[String] = None
val aliased = plain.alias("a dataset")
getAlias(aliased)
Option[String] = Some(a dataset)
Disclaimer: as stated above, this code relies on undocumented APIs subject to change. It works as of Spark 2.3.
After much digging into mostly undocumented Spark methods, here is the full code to pull the list of fields, along with the table alias for a dataframe in PySpark:
def schema_from_plan(df):
plan = df._jdf.queryExecution().analyzed()
all_fields = _schema_from_plan(plan)
iterator = plan.output().iterator()
output_fields = {}
while iterator.hasNext():
field = iterator.next()
queryfield = all_fields.get(field.exprId().id(),{})
if not queryfield=={}:
tablealias = queryfield["tablealias"]
else:
tablealias = ""
output_fields[field.exprId().id()] = {
"tablealias": tablealias,
"dataType": field.dataType().typeName(),
"name": field.name()
}
return list(output_fields.values())
def _schema_from_plan(root,tablealias=None,fields={}):
iterator = root.children().iterator()
while iterator.hasNext():
node = iterator.next()
nodeClass = node.getClass().getSimpleName()
if (nodeClass=="SubqueryAlias"):
# get the alias and process the subnodes with this alias
_schema_from_plan(node,node.alias(),fields)
else:
if tablealias:
# add all the fields, along with the unique IDs, and a new tablealias field
iterator = node.output().iterator()
while iterator.hasNext():
field = iterator.next()
fields[field.exprId().id()] = {
"tablealias": tablealias,
"dataType": field.dataType().typeName(),
"name": field.name()
}
_schema_from_plan(node,tablealias,fields)
return fields
# example: fields = schema_from_plan(df)
For Java:
As #veinhorn mentioned, it is also possible to get the alias in Java. Here is a utility method example:
public static <T> Optional<String> getAlias(Dataset<T> dataset){
final LogicalPlan analyzed = dataset.queryExecution().analyzed();
if(analyzed instanceof SubqueryAlias) {
SubqueryAlias subqueryAlias = (SubqueryAlias) analyzed;
return Optional.of(subqueryAlias.alias());
}
return Optional.empty();
}