I am using mongodb to store time series data in following way
values: {
0: { 0: 999999, 1: 999999, …, 59: 1000000 },
1: { 0: 2000000, 1: 2000000, …, 59: 1000000 },
…,
58: { 0: 1600000, 1: 1200000, …, 59: 1100000 },
59: { 0: 1300000, 1: 1400000, …, 59: 1500000 }
}
But now i want to update specific range value
FOR EXAMPLE. i want to set all values to 0 greater than values.30.40 where VALUES is the main node, 30 is MINUTES and 40 is SECOND
Would it be possible to change the inner Json to an array ?
That way, you can access and modify using indexes.
It would also help when you do not have values at some indexes.
values: {
0:{[12,25,36,........,12]},
.
.
.
}
Related
There is a list of documents P having two timestamps representing the time range [ P ] in which the document is valid. An index over these intervals was created:
function (doc) {
emit([doc.start, doc.end], someStuff(doc));
}
We want to receive documents P wich start before some end timestamp E and stop after some start timestamp S:
P(S, E) = { P | P_s <= E && P_e >= S }
For instance, in a picture like this
<-- TIME -->
..------------------S-------------------------------------E----------------------..
.. P0 ][ P1 ][ P2 ][ P3 ][ P4 ][ P5 ][ P6 ..
we expected the subset {P1, P2, P3, P4} as the result. We try to get the desired result using the following key-range
_view/range?descending=false&startkey=[0,S]&endkey=[E,{}]
The result P(A, E) = {P0, P1, P2, P3, P4} is wrong which makes sense when checking the following example for S=17 and E=30:
key startkey endkey accept
_________________________________________________________________________
[10,15] [0,17] <= [10,15] <= [30, {}] -> True <- This is wrong
[15,25] [0,17] <= [15,25] <= [30, {}] -> True OK
[25,30] [0,17] <= [25,30] <= [30, {}] -> True OK
[25,50] [0,17] <= [25,50] <= [30, {}] -> True OK
[35,50] [0,17] <= [35,50] <= [30, {}] -> False OK
Is it possible to define a range such that we get the desired result?
This is much easier to achieve using the POST /db/_find endpoint. You can express your query as a selector:
{
"selector": {
"start": { "$lt": 100 },
"end": { "$gt": 300 }
}
"sort": ["start"]
}
This is the equivalent to the SQL SELECT * FROM db WHERE start<100 AND end > 300 SORY BY start.
You will almost certainly need an index on "start" too to speed things up.
I have a collection which holds more than 15 million documents. Out of those 15 million documents I update 20k records every hour. But update query takes a long time to finish (30 min around).
Document:
{ "inst" : "instance1", "dt": "2015-12-12T00:00:000Z", "count": 10}
I have an array which holds 20k instances to be updated.
My Query looks like this:
For h in hourly filter h.dt == DATE_ISO8601(14501160000000)
For i in instArr
filter i.inst == h.inst
update h with {"inst":i.inst, "dt":i.dt, "count":i.count} in hourly
Is there any optimized way of doing this. I have hash indexing on inst and skiplist indexing on dt.
Update
I could not use 20k inst in the query manually so following is the execution plan for just 2 inst:
FOR r in hourly FILTER r.dt == DATE_ISO8601(1450116000000) FOR i IN
[{"inst":"0e649fa22bcc5200d7c40f3505da153b", "dt":"2015-12-14T18:00:00.000Z"}, {}] FILTER i.inst ==
r.inst UPDATE r with {"inst":i.inst, "dt": i.dt, "max":i.max, "min":i.min, "sum":i.sum, "avg":i.avg,
"samples":i.samples} in hourly OPTIONS { ignoreErrors: true } RETURN NEW.inst
Execution plan:
Id NodeType Est. Comment
1 SingletonNode 1 * ROOT
5 CalculationNode 1 - LET #6 = [ { "inst" : "0e649fa22bcc5200d7c40f3505da153b", "dt" : "2015-12-14T18:00:00.000Z" }, { } ] /* json expression */ /* const assignment */
13 IndexRangeNode 103067 - FOR r IN hourly /* skiplist index scan */
6 EnumerateListNode 206134 - FOR i IN #6 /* list iteration */
7 CalculationNode 206134 - LET #8 = i.`inst` == r.`inst` /* simple expression */ /* collections used: r : hourly */
8 FilterNode 206134 - FILTER #8
9 CalculationNode 206134 - LET #10 = { "inst" : i.`inst`, "dt" : i.`dt`, "max" : i.`max`, "min" : i.`min`, "sum" : i.`sum`, "avg" : i.`avg`, "samples" : i.`samples` } /* simple expression */
10 UpdateNode 206134 - UPDATE r WITH #10 IN hourly
11 CalculationNode 206134 - LET #12 = $NEW.`inst` /* attribute expression */
12 ReturnNode 206134 - RETURN #12
Indexes used:
Id Type Collection Unique Sparse Selectivity Est. Fields Ranges
13 skiplist hourly false false n/a `dt` [ `dt` == "2015-12-14T18:00:00.000Z" ]
Optimization rules applied:
Id RuleName
1 move-calculations-up
2 move-filters-up
3 move-calculations-up-2
4 move-filters-up-2
5 remove-data-modification-out-variables
6 use-index-range
7 remove-filter-covered-by-index
Write query options:
Option Value
ignoreErrors true
waitForSync false
nullMeansRemove false
mergeObjects true
ignoreDocumentNotFound false
readCompleteInput true
I assume the selection part (not the update part) will be the bottleneck in this query.
The query seems problematic because for each document matching the first filter (h.dt == DATE_ISO8601(...)), there will be an iteration over the 20,000 values in the instArr array. If instArr values are unique, then only one value from it will match. Additionally, no index will be used for the inner loop, as the index selection has happened in the outer loop already.
Instead of looping over all values in instArr, it will be better to turn the accompanying == comparison into an IN comparison. That would already work if instArr would be an array of instance names, but it seems to be an array of instance objects (consisting of at least attributes inst and count). In order to use the instance names in an IN comparison, it would be better to have a dedicated array of instance names, and a translation table for the count and dt values.
Following is an example for generating these with JavaScript:
var instArr = [ ], trans = { };
for (i = 0; i < 20000; ++i) {
var instance = "instance" + i;
var count = Math.floor(Math.random() * 10);
var dt = (new Date(Date.now() - Math.floor(Math.random() * 10000))).toISOString();
instArr.push(instance);
trans[instance] = [ count, dt ];
}
instArr would then look like this:
[ "instance0", "instance1", "instance2", ... ]
and trans:
{
"instance0" : [ 4, "2015-12-16T21:24:45.106Z" ],
"instance1" : [ 0, "2015-12-16T21:24:39.881Z" ],
"instance2" : [ 2, "2015-12-16T21:25:47.915Z" ],
...
}
These data can then be injected into the query using bind variables (named like the variables above):
FOR h IN hourly
FILTER h.dt == DATE_ISO8601(1450116000000)
FILTER h.inst IN #instArr
RETURN #trans[h.inst]
Note that ArangoDB 2.5 does not yet support the #trans[h.inst] syntax. In that version, you will need to write:
LET trans = #trans
FOR h IN hourly
FILTER h.dt == DATE_ISO8601(1450116000000)
FILTER h.inst IN #instArr
RETURN trans[h.inst]
Additionally, 2.5 has a problem with longer IN lists. IN-list performance decreases quadratically with the length of the IN list. So in this version, it will make sense to limit the length of instArr to at most 2,000 values. That may require issuing multiple queries with smaller IN lists instead of just one with a big IN list.
The better alternative would be to use ArangoDB 2.6, 2.7 or 2.8, which do not have that problem, and thus do not require the workaround. Apart from that, you can get away with the slightly shorter version of the query in the newer ArangoDB versions.
Also note that in all of the above examples I used a RETURN ... instead of the UPDATE statement from the original query. This is because all my tests revealed that the selection part of the query is the major problem, at least with the data I had generated.
A final note on the original version of the UPDATE: updating each document's inst value with i.inst seems redudant, because i.inst == h.inst so the value won't change.
I have a method that returns a String. I want to substitute it with a default value such as "<empty>" if it returns an empty string or null. Let's assume its name is getSomeString, it is an expensive operation so I can call it only once, and I can't change its return type to Option[String]. For now, I'm doing the following:
val myStr = {
val s = getSomeString
if (s == null || s.isEmpty) "<empty>" else s
}
Is there a simpler way to achieve the same thing?
val myStr = Option(getSomeString).filterNot(_.isEmpty).getOrElse("<empty>")
Updated
I posted this code because I think the intent in this code is clear than if/else or pattern matching version, but I didn't consider the performance issue.
As the others in comments mentioned, this code is much slower than simple if / else or pattern matching(this line will create a lot new objects which is an expensive operation), so please do not use this code when performance is an issue.
Given an expensive function:
scala> def s(i: Int): String = i match { case 0=>null case 1=>"" case 2=>"hi" }
s: (i: Int)String
I think this is easy to read and free of overhead, cf this in the wild:
scala> def q(i: Int) = s(i) match { case ""|null => "<empty>" case x => x }
q: (i: Int)String
scala> q(0)
res3: String = <empty>
scala> q(1)
res4: String = <empty>
scala> q(2)
res5: String = hi
To my eyes, this is not as expressive, even with minimalist punctuation:
scala> Option(s(0)) filterNot (_.isEmpty) getOrElse "<empty>"
res6: String = <empty>
Moreover, contrast the cost in anonfun classes for the closures and additional method invocations:
scala> :javap -
Size 1161 bytes
MD5 checksum 765f5f67b0c574252b059c8adfab1cf0
Compiled from "<console>"
[...]
9: getstatic #26 // Field scala/Option$.MODULE$:Lscala/Option$;
12: getstatic #31 // Field .MODULE$:L;
15: iconst_0
16: invokevirtual #35 // Method .s:(I)Ljava/lang/String;
19: invokevirtual #39 // Method scala/Option$.apply:(Ljava/lang/Object;)Lscala/Option;
22: new #41 // class $anonfun$1
25: dup
26: invokespecial #42 // Method $anonfun$1."<init>":()V
29: invokevirtual #48 // Method scala/Option.filterNot:(Lscala/Function1;)Lscala/Option;
32: new #50 // class $anonfun$2
35: dup
36: invokespecial #51 // Method $anonfun$2."<init>":()V
39: invokevirtual #55 // Method scala/Option.getOrElse:(Lscala/Function0;)Ljava/lang/Object;
42: checkcast #57 // class java/lang/String
45: putfield #17 // Field res6:Ljava/lang/String;
The pattern match is generally just an if-else, smaller and faster (even considering that it doesn't optimise s == "" to s.isEmpty):
scala> :javap -r #q
public java.lang.String q(int);
flags: ACC_PUBLIC
Code:
stack=2, locals=5, args_size=2
0: getstatic #19 // Field $line3/$read$$iw$$iw$.MODULE$:L$line3/$read$$iw$$iw$;
3: iload_1
4: invokevirtual #22 // Method $line3/$read$$iw$$iw$.s:(I)Ljava/lang/String;
7: astore_3
8: ldc #24 // String
10: aload_3
11: invokevirtual #28 // Method java/lang/Object.equals:(Ljava/lang/Object;)Z
14: ifeq 22
17: iconst_1
18: istore_2
19: goto 33
22: aload_3
23: ifnonnull 31
26: iconst_1
27: istore_2
28: goto 33
31: iconst_0
32: istore_2
33: iload_2
34: ifeq 44
37: ldc #30 // String <empty>
39: astore 4
41: goto 47
44: aload_3
45: astore 4
47: aload 4
49: areturn
But inspired by the other answer, even if I would never take this code home to meet my parents (because it incorrectly converts the value "null" if the expensive function returns it -- though maybe it's a feature to do that), here is a regex:
scala> def p(i: Int) = "" + s(i) replaceAll ("^null$|^$", "<empty>")
p: (i: Int)String
The "" + s(i) is a shorthand for String.valueOf, which of course produces the String "null" for a null reference value. I appreciate SO's ability not only to generate quick answers to questions, but to encourage some out-of-the-box thinking.
You could replace null with empty string in first step using Option and then substitute with default text if result is empty (whether because it was empty originally or because it was null):
Option(getSomeString).getOrElse("").replaceAll("^$","<empty>")
You could add a method to String using an implicit value class
object ImplicitClassContainer {
implicit class RichString(val s: String) extends AnyVal {
def getOrDefault(defaultValue: String): String = {
s match {
case null | "" => defaultValue
case x => x
}
}
}
to be used like this
import ImplicitClassContainer._
println("hi".getOrDefault("<empty1>"))
println("".getOrDefault("<empty2>"))
val s: String = null
println(s.getOrDefault("<empty3>"))
so even the method call on null is handled gracefully (Scala 2.10.1).
If you don't need changes:
Option(getSomeString).fold("<empty>")(s => s)
If you need to modify if for nonEmpty result. Simple example:
Option(getSomeString).fold("<empty>")(str => s"xxxxxxxx$str")
val myStr = getSomeString match { case ""|null => "<empty>" case s => s }
Well it might be a dumb question, but I'm unable to find an answer:
The case is simple, I have a function with an integer in entry and two variable to assign depending of that. The problem is that the value assigned to the variables is common to certain case, but those case are not the same for the two variables. (if it's not clear enough, see the example).
I was wondering what was the best practice for such a case.
Is it something like :
function test(a){
var x,y;
switch (a){
case 0:
case 1:
case 7:
y=...;
break;
case 2:
case 6:
y=...;
break;
case 3:
case 4:
case 5:
y=...;
}
switch(a){
case 5:
case 6:
case 7:
x=...;
break;
case 0:
case 4:
x=...;
break;
case 1:
case 2:
case 3:
x=...;
}
...
}
or
function test(a){
var x,y;
switch (a){
case 0:
x=...;
y=...;
break;
case 1:
x=...;
y=...;
break;
case 2:
x=...;
y=...;
break;
case 3:
x=...;
y=...;
break;
case 4:
x=...;
y=...;
break;
case 5:
x=...;
y=...;
break;
case 6:
x=...;
y=...;
break;
case 7:
x=...;
y=...;
}
...
}
Or to use a mix of the two with the value assigned to x in each case and to make groups for the value of y?
Note that there might be more than 8 value, it's just for the example.
Thanks in advance.
If there's no real overlap between the different cases in the switch statements, you're probably better off having them separate.
If there was a majority of commonality between the actions, you could combine them with certain special cases within the individual case blocks but you seem to indicate that's not the case here.
However, if this isn't just a simple example of a much more complex case, you can achieve a more compact solution with something like:
// index: 0 1 2 3 4 5 6 7 8 9 10 11 12
static const int lookupX[] = { 2, 7, 1, 8, 2, 8, 1, 8, 2, 8, 4, 5, 9};
static const int lookupY[] = { 3, 1, 4, 1, 5, 9, 2, 6, 5, 3, 5, 8, 9};
if ((a >=0) && (a < sizeof(lookupX) / sizeof(*lookupX)))
x = lookupX[a];
if ((a >=0) && (a < sizeof(lookupY) / sizeof(*lookupY)))
y = lookupY[a];
This will allow you to keep the values in a much smaller "configuration" section so that you can easily see the intent. The range checking and lookup then become very simple.
That code is tailored to C - I'm not sure what specific language you're using (because of the var statement) but it's basically only doing the lookup if the index will be valid. You'll need to translate that bit to your language of choice.
this is my code but when it shows up on my android display it changes my fractions to a decimal how do i keep it in this format.
public void readyForJavascript(final String arg) {
webView.loadUrl("javascript:SpinningWheel.addSlot({ " +
"1:1, 2: 5/16, 3: 3/8, 4: 7/16, 5: 1/2, 6: 9/16," +
"7: 5/8, 8: 3/4, 9: 7/8, 10: 1, 11: 1/18, 12: 1/4 });");
webView.loadUrl("javascript:SpinningWheel.addSlot({ " +
"1: 'AM', 2: 'PM'});");
webView.loadUrl("javascript:SpinningWheel.addSlot({ " +
"1: 'AM', 2: 'PM'});");
webView.loadUrl("javascript:SpinningWheel.open();");
}
You need to enclose your fractions in single quotes:
"1:1, 2: '5/16', 3: '3/8' .......
I think at some point your Javascript sees 3/8 and evaluates it as a calculation.