Slick print list of tables - slick

I want to print a list of tables from the database I connect to via JDBC but cannot figure out how to do this. I have tried using MTable.getTables and defaultTables, thanks
import slick.driver.H2Driver.api._
import scala.concurrent._
import com.typesafe.config.{ ConfigFactory, Config }
object SlickTest {
def main(args: Array[String]): Unit = {
"""
|mydb = {
| driver = com.microsoft.sqlserver.jdbc.SQLServerDriver
| slickDriver = com.typesafe.slick.driver.ms.SQLServerDriver
| url="jdbc:sqlserver://#############"
| properties = {
| databaseName = "######"
| user = "#######"
| password = "########"
| }
| numThreads = 10
|}
""".stripMargin
val db = Database.forConfig("mydb")
???
}
}

There are quite a few different ways of doing this, but following the layout of the code you had in your question, this works for Slick 3.1 (and using freeslick profiles, because I don't have the slick extensions SQLServer lib).
import com.typesafe.config.ConfigFactory
import slick.backend.DatabaseConfig
import slick.driver.JdbcProfile
import scala.concurrent.duration.Duration
import scala.concurrent.{Await, ExecutionContext}
object SlickTest {
def main(args: Array[String]): Unit = {
val config = """
|mydb = {
|
| driver = "freeslick.MSJDBCSQLServerProfile$"
| db {
| driver = com.microsoft.sqlserver.jdbc.SQLServerDriver
| url="jdbc:sqlserver://localhost:2008"
| properties = {
| databaseName = "freeslicktest"
| user = "sa"
| password = "FreeSlick"
| }
| }
|}
""".stripMargin
val dbConfig = DatabaseConfig.forConfig[JdbcProfile]("mydb", ConfigFactory.parseString(config))
import ExecutionContext.Implicits.global
Await.result(dbConfig.db.run(dbConfig.driver.defaultTables), Duration.Inf).foreach(println)
}
}
produces this result for a database with one table called USERS in it.
*** (c.z.hikari.HikariDataSource) HikariCP pool mydb.db is starting.
MTable(MQName(freeslicktest.dbo.USERS),TABLE,null,None,None,None)
Process finished with exit code 0

Related

Sinon Class Constructor

I have an Animal class as follows
Animal.js
export default class Animal {
constructor(type) {
this.type = type
}
getAnimalSound(animal) {
if (animal && animal.type == 'dog') return 'woof'
if (animal && animal.type == 'cat') return 'meow'
}
}
I make a zoo module which has a method for getAnimalSound() as follows
zoo.js
import Animal from './Animal'
export default function getAnimalSound(type) {
let animal = new Animal(type)
let animalSound = animal.getAnimalSound(animal)
return animalSound
}
Now how do i make unit testing for zoo module?
zoo.test.js
import sinon from 'sinon'
import Animal from './Animal'
import getAnimalSound from './zoo'
let animalStub = sinon.createStubInstance(Animal)
let a = animalStub.getAnimalSound.returns('woof')
let sound = getAnimalSound('cat')
console.log(sound)
So the problem is that the 'new' has no effect by the way i have stubbed in test.js
Can i achieve this?
Regards
Bobu P
You could use Link Seams to mock your ./animal.js module and Animal class.
E.g.
animal.ts:
export default class Animal {
type: any;
constructor(type) {
this.type = type;
}
getAnimalSound(animal) {
if (animal && animal.type == 'dog') return 'woof';
if (animal && animal.type == 'cat') return 'meow';
}
}
zoo.ts:
import Animal from './animal';
export default function getAnimalSound(type) {
let animal = new Animal(type);
let animalSound = animal.getAnimalSound(animal);
return animalSound;
}
zoo.test.ts:
import sinon from 'sinon';
import proxyquire from 'proxyquire';
import { expect } from 'chai';
describe('61716637', () => {
it('should pass', () => {
const animalInstanceStub = {
getAnimalSound: sinon.stub().returns('stubbed value'),
};
const AnimalStub = sinon.stub().returns(animalInstanceStub);
const getAnimalSound = proxyquire('./zoo', {
'./animal': { default: AnimalStub },
}).default;
const actual = getAnimalSound('bird');
expect(actual).to.be.equal('stubbed value');
sinon.assert.calledWith(AnimalStub, 'bird');
sinon.assert.calledWith(animalInstanceStub.getAnimalSound, animalInstanceStub);
});
});
unit test results with coverage report:
61716637
✓ should pass (2242ms)
1 passing (2s)
-----------|---------|----------|---------|---------|-------------------
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
-----------|---------|----------|---------|---------|-------------------
All files | 54.55 | 0 | 33.33 | 66.67 |
animal.ts | 16.67 | 0 | 0 | 25 | 4-8
zoo.ts | 100 | 100 | 100 | 100 |
-----------|---------|----------|---------|---------|-------------------

Spark Struct structfield names getting changed in UDF

I am trying to pass a struct in spark to udf. It is changing the field names and renaming to the column position. How do I fix it?
object TestCSV {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("localTest").setMaster("local")
val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)
val inputData = sqlContext.read.format("com.databricks.spark.csv")
.option("delimiter","|")
.option("header", "true")
.load("test.csv")
inputData.printSchema()
inputData.show()
val groupedData = inputData.withColumn("name",struct(inputData("firstname"),inputData("lastname")))
val udfApply = groupedData.withColumn("newName",processName(groupedData("name")))
udfApply.show()
}
def processName = udf((input:Row) =>{
println(input)
println(input.schema)
Map("firstName" -> input.getAs[String]("firstname"), "lastName" -> input.getAs[String]("lastname"))
})
}
Output:
root
|-- id: string (nullable = true)
|-- firstname: string (nullable = true)
|-- lastname: string (nullable = true)
+---+---------+--------+
| id|firstname|lastname|
+---+---------+--------+
| 1| jack| reacher|
| 2| john| Doe|
+---+---------+--------+
Error:
[jack,reacher]
StructType(StructField(i[1],StringType,true), > StructField(i[2],StringType,true))
17/03/08 09:45:35 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 2)
java.lang.IllegalArgumentException: Field "firstname" does not exist.
What you are encountering is really strange. After playing around a bit I finally figured out that it may be related to a problem with the optimizer engine. It seems that the problem is not the UDF but the struct function.
I get it to work (Spark 1.6.3) when I cache the groupedData, without caching I get your reported exception:
import org.apache.spark.sql.Row
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.{SparkConf, SparkContext}
object Demo {
def main(args: Array[String]): Unit = {
val sc = new SparkContext(new SparkConf().setAppName("Demo").setMaster("local[1]"))
val sqlContext = new HiveContext(sc)
import sqlContext.implicits._
import org.apache.spark.sql.functions._
def processName = udf((input: Row) => {
Map("firstName" -> input.getAs[String]("firstname"), "lastName" -> input.getAs[String]("lastname"))
})
val inputData =
sc.parallelize(
Seq(("1", "Kevin", "Costner"))
).toDF("id", "firstname", "lastname")
val groupedData = inputData.withColumn("name", struct(inputData("firstname"), inputData("lastname")))
.cache() // does not work without cache
val udfApply = groupedData.withColumn("newName", processName(groupedData("name")))
udfApply.show()
}
}
Alternatively you can use the RDD API to make your struct, but this is not really nice:
case class Name(firstname:String,lastname:String) // define outside main
val groupedData = inputData.rdd
.map{r =>
(r.getAs[String]("id"),
Name(
r.getAs[String]("firstname"),
r.getAs[String]("lastname")
)
)
}
.toDF("id","name")

How to generate a running sequence number in spark dataframe v1.6

I use spark v1.6. I have the below dataframe.
Primary_key | Dim_id
PK1 | 1
PK2 | 2
PK3 | 3
I would like to create a new dataframe with a new sequence #s whenever a new record comes in. Lets say, I get 2 new records from the source with values PK4 & PK5, I would like to create new dim_ids with the values 4 and 5. So, my new dataframe should look like below.
Primary_key | Dim_id
PK1 | 1
PK2 | 2
PK3 | 3
PK4 | 4
PK5 | 5
How to generate a running sequence number in spark dataframe v1.6 for the new records?
If you have a database somewhere, you can create a sequence in it, and use it with a user defined function (as you, I stumbled upon this problem...).
Reserve a bucket of sequence numbers, and use it (the incrementby parameter must be the same as the one used to create the sequence). As it's an object, SequenceID will be a singleton on each working node, and you can iterate over the bucket of sequences using the atomiclong.
It's far from being perfect (possible connection leaks, relies on a DB, locks on static class, does), comments welcome.
import java.sql.Connection
import java.sql.DriverManager
import java.util.concurrent.locks.ReentrantLock
import java.util.concurrent.atomic.AtomicLong
import org.apache.spark.sql.functions.udf
object SequenceID {
var current: AtomicLong = new AtomicLong
var max: Long = 0
var connection: Connection = null
var connectionLock = new ReentrantLock
var seqLock = new ReentrantLock
def getConnection(): Connection = {
if (connection != null) {
return connection
}
connectionLock.lock()
if (connection == null) {
// create your jdbc connection here
}
connectionLock.unlock()
connection
}
def next(sequence: String, incrementBy: Long): Long = {
if (current.get == max) {
// sequence bucket exhausted, get a new one
seqLock.lock()
if (current.get == max) {
val rs = getConnection().createStatement().executeQuery(s"SELECT NEXT VALUE FOR ${sequence} FROM sysibm.sysdummy1")
rs.next()
current.set(rs.getLong(1))
max = current.get + incrementBy
}
seqLock.unlock()
}
return current.getAndIncrement
}
}
class SequenceID() extends Serializable {
def next(sequence: String, incrementBy: Long): Long = {
return SequenceID.next(sequence, incrementBy)
}
}
val sequenceGenerator = new SequenceID(properties)
def sequenceUDF(seq: SequenceID) = udf[Long](() => {
seq.next("PK_SEQUENCE", 500L)
})
val seq = sequenceUDF(sequenceGenerator)
myDataframe.select(myDataframe("foo"), seq())

Scala Chat Application, separate threads for local IO and socket IO

I'm writing a chat application in Scala, the problem is with the clients, the client reads from StdIn (which blocks) before sending the data to the echo server, so if multiple clients are connected then they don't receive data from the server until reading from StdIn has completed. I'm thinking that local IO, i.e reading from StdIn and reading/writing to the socket should be on separate threads but I can't think of a way to do this, below is the Client singleton code:
import java.net._
import scala.io._
import java.io._
import java.security._
object Client {
var msgAcc = ""
def main(args: Array[String]): Unit = {
val conn = new ClientConnection(InetAddress.getByName(args(0)), args(1).toInt)
val server = conn.connect()
println("Enter a username")
val user = new User(StdIn.readLine())
println("Welcome to the chat " + user.username)
sys.addShutdownHook(this.shutdown(conn, server))
while (true) {
val txMsg = StdIn.readLine()//should be on a separate thread?
if (txMsg != null) {
conn.sendMsg(server, user, txMsg)
val rxMsg = conn.getMsg(server)
val parser = new JsonParser(rxMsg)
val formattedMsg = parser.formatMsg(parser.toJson())
println(formattedMsg)
msgAcc = msgAcc + formattedMsg + "\n"
}
}
}
def shutdown(conn: ClientConnection, server: Socket): Unit = {
conn.close(server)
val fileWriter = new BufferedWriter(new FileWriter(new File("history.txt"), true))
fileWriter.write(msgAcc)
fileWriter.close()
println("Leaving chat, thanks for using")
}
}
below is the ClientConnection class used in conjunction with the Client singleton:
import javax.net.ssl.SSLSocket
import javax.net.ssl.SSLSocketFactory
import javax.net.SocketFactory
import java.net.Socket
import java.net.InetAddress
import java.net.InetSocketAddress
import java.security._
import java.io._
import scala.io._
import java.util.GregorianCalendar
import java.util.Calendar
import java.util.Date
import com.sun.net.ssl.internal.ssl.Provider
import scala.util.parsing.json._
class ClientConnection(host: InetAddress, port: Int) {
def connect(): Socket = {
Security.addProvider(new Provider())
val sslFactory = SSLSocketFactory.getDefault()
val sslSocket = sslFactory.createSocket(host, port).asInstanceOf[SSLSocket]
sslSocket
}
def getMsg(server: Socket): String = new BufferedSource(server.getInputStream()).getLines().next()
def sendMsg(server: Socket, user: User, msg: String): Unit = {
val out = new PrintStream(server.getOutputStream())
out.println(this.toMinifiedJson(user.username, msg))
out.flush()
}
private def toMinifiedJson(user: String, msg: String): String = {
s"""{"time":"${this.getTime()}","username":"$user","msg":"$msg"}"""
}
private def getTime(): String = {
val cal = Calendar.getInstance()
cal.setTime(new Date())
"(" + cal.get(Calendar.HOUR_OF_DAY) + ":" + cal.get(Calendar.MINUTE) + ":" + cal.get(Calendar.SECOND) + ")"
}
def close(server: Socket): Unit = server.close()
}
You can add concurrency by using Scala Akka Actors. As of this writing the current Scala version is 2.11.8. See Actor documentation here:
http://docs.scala-lang.org/overviews/core/actors.html
This chat example is old but demonstrates a technique to handle in the neighborhood of a million simultaneous clients using Actors:
http://doc.akka.io/docs/akka/1.3.1/scala/tutorial-chat-server.html
Finally you can also Google the Twitter Finagle project which uses Scala and provides servers with concurrency. A lot of work to learn it I think...

Slick 3 for scala update query not functioning

First of I would like to state that I am new to slick and am using version 3.1.1 . I been reading the manual but i am having trouble getting my query to work. Either something is wrong with my connection string or something is wrong with my Slick code. I got my config from here http://slick.typesafe.com/doc/3.1.1/database.html and my update example from here bottom of page http://slick.typesafe.com/doc/3.1.1/queries.html . Ok so here is my code
Application Config.
mydb= {
dataSourceClass = org.postgresql.ds.PGSimpleDataSource
properties = {
databaseName = "Jsmith"
user = "postgres"
password = "unique"
}
numThreads = 10
}
My Controller -- Database table is called - relations
package controllers
import play.api.mvc._
import slick.driver.PostgresDriver.api._
class Application extends Controller {
class relations(tag: Tag) extends Table[(Int,Int,Int)](tag, "relations") {
def id = column[Int]("id", O.PrimaryKey)
def me = column[Int]("me")
def following = column[Int]("following")
def * = (id,me,following)
}
val profiles = TableQuery[relations]
val db = Database.forConfig("mydb")
try {
// ...
} finally db.close()
def index = Action {
val q = for { p <- profiles if p.id === 2 } yield p.following
val updateAction = q.update(322)
val invoker = q.updateStatement
Ok()
}
}
What could be wrong with my code above ? I have a separate project that uses plain JDBC and this configuration works perfectly for it
db.default.driver=org.postgresql.Driver
db.default.url="jdbc:postgresql://localhost:5432/Jsmith"
db.default.user="postgres"
db.default.password="unique"
You did not run your action yet. db.run(updateAction) executes your query respectively your action on a database (untested):
def index = Action.async {
val q = for { p <- profiles if p.id === 2 } yield p.following
val updateAction = q.update(322)
val db = Database.forConfig("mydb")
db.run(updateAction).map(_ => Ok())
}
db.run() returns a Future which will be eventually completed. It is then simply mapped to a Result in play.
q.updateStatement on the other hand just generates a sql statement. This can be useful while debugging.
Look the code from my project:
def updateStatus(username: String, password: String, status: Boolean): Future[Boolean] = {
db.run(
(for {
user <- Users if user.username === username
} yield {
user
}).map(_.online).update(status)
}

Resources