Timeseries chart: data as rows instead of columns - apache-spark

Question

I want to use a bar chart as a visualisation, to depict the number of times that one of several applications was run on each date (for simplicity sake, think of applications as excel, word, powerpoint etc. For example, on 15-March, if a user opens Excel once, and Word twice, it will show a height of two for Word, stacked with a height of one for excel etc).
The visualisation shows one year of stacked data (365 bars), where each stacked bar uses a different color for each program. There are 20 programs. The dataseries are stacked - for example Word (if it is depicted) is always at the bottom, followed by Excel, etc.
The traditional way to do this is using an outerjoin that creates 365 rows, and 20 columns (Word, Excel etc) with an integer value for each column for each day. This technique works, but is very compute intensive on databases (not to mention dense SQL) and I wondered if anyone knows of a timeseries charting tool that handles this problem.
This is a data pivoting issue: I want to extract the data in rows, and let the charting product assemble the bars. It needs to read the Data Series name from the data, and create a new Data Series whenever it encounters a new value. I actually have thousands of possible Programs, but each chart is unlikely to need to depict any more than 20.
The charting product should work like this:
minDate=20150101, maxDate=20151231, interval=Day
and then the data is
20150315 "Word" 1
20150315 "Word" 1
20150315 "Excel" 1
20150316 "Powerpoint" 1
20150316 "Word" 1
Alternatively, each row could be a total per day per program:
20150315 "Word" 2
20150315 "Excel" 1
20150316 "Powerpoint" 1
20150316 "Word" 1
I am using C3, with spark and scala. I can use any javascript library.

Answer 1

What you are asking is how do you reshape the data from wide to long format. In Spark, you use DataFrame.explode. Assuming you had something that looked like this:
+--------+----+-----+
| date|word|excel|
+--------+----+-----+
|20150101| 3| 4|
|20150102| 1| 2|
+--------+----+-----+
You could reshape it with this:
df.explode($"word", $"excel"){case row =>
Seq(("Word", row.getInt(0)), ("Excel", row.getInt(1)))
}.select($"date", $"_1" as "app", $"_2" as "count")
Into this:
+--------+-----+-----+
| date| app|count|
+--------+-----+-----+
|20150101| word| 3|
|20150101|excel| 4|
|20150102| word| 1|
|20150102|excel| 2|
+--------+-----+-----+