I'm trying to be a bit fancy in a database table class. I'd like the user to be able to just pass a row interface (or class if necessary) and have the table class infer how to create the table.
interface Row {
rowid: number
}
class Table<T extends Row> {
getRowDefinition(){
// magically use T to create col definitions
}
createTable(){
const defs = this.getRowDefinition();
const cols = defs.map(def => {
switch(def.type){
case 'string':
return `${def.name} TEXT`;
case 'number':
return `${def.name} INTEGER;
}
});
const sql = 'CREATE TABLE ...'; // use cols, etc
}
}
interface MyRow extends Row {
name: string
age: number
}
class MyTable extends Table<MyRow> {}
Typescript types are removed at compile time and so they are not available in run-time.
You can use instanceof if you create some classes to determine each row type, and use typeof to determine the type of a variable in run-time.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/instanceof
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/typeof
I use spark v1.6. I have the below dataframe.
Primary_key | Dim_id
PK1 | 1
PK2 | 2
PK3 | 3
I would like to create a new dataframe with a new sequence #s whenever a new record comes in. Lets say, I get 2 new records from the source with values PK4 & PK5, I would like to create new dim_ids with the values 4 and 5. So, my new dataframe should look like below.
Primary_key | Dim_id
PK1 | 1
PK2 | 2
PK3 | 3
PK4 | 4
PK5 | 5
How to generate a running sequence number in spark dataframe v1.6 for the new records?
If you have a database somewhere, you can create a sequence in it, and use it with a user defined function (as you, I stumbled upon this problem...).
Reserve a bucket of sequence numbers, and use it (the incrementby parameter must be the same as the one used to create the sequence). As it's an object, SequenceID will be a singleton on each working node, and you can iterate over the bucket of sequences using the atomiclong.
It's far from being perfect (possible connection leaks, relies on a DB, locks on static class, does), comments welcome.
import java.sql.Connection
import java.sql.DriverManager
import java.util.concurrent.locks.ReentrantLock
import java.util.concurrent.atomic.AtomicLong
import org.apache.spark.sql.functions.udf
object SequenceID {
var current: AtomicLong = new AtomicLong
var max: Long = 0
var connection: Connection = null
var connectionLock = new ReentrantLock
var seqLock = new ReentrantLock
def getConnection(): Connection = {
if (connection != null) {
return connection
}
connectionLock.lock()
if (connection == null) {
// create your jdbc connection here
}
connectionLock.unlock()
connection
}
def next(sequence: String, incrementBy: Long): Long = {
if (current.get == max) {
// sequence bucket exhausted, get a new one
seqLock.lock()
if (current.get == max) {
val rs = getConnection().createStatement().executeQuery(s"SELECT NEXT VALUE FOR ${sequence} FROM sysibm.sysdummy1")
rs.next()
current.set(rs.getLong(1))
max = current.get + incrementBy
}
seqLock.unlock()
}
return current.getAndIncrement
}
}
class SequenceID() extends Serializable {
def next(sequence: String, incrementBy: Long): Long = {
return SequenceID.next(sequence, incrementBy)
}
}
val sequenceGenerator = new SequenceID(properties)
def sequenceUDF(seq: SequenceID) = udf[Long](() => {
seq.next("PK_SEQUENCE", 500L)
})
val seq = sequenceUDF(sequenceGenerator)
myDataframe.select(myDataframe("foo"), seq())
First of I would like to state that I am new to slick and am using version 3.1.1 . I been reading the manual but i am having trouble getting my query to work. Either something is wrong with my connection string or something is wrong with my Slick code. I got my config from here http://slick.typesafe.com/doc/3.1.1/database.html and my update example from here bottom of page http://slick.typesafe.com/doc/3.1.1/queries.html . Ok so here is my code
Application Config.
mydb= {
dataSourceClass = org.postgresql.ds.PGSimpleDataSource
properties = {
databaseName = "Jsmith"
user = "postgres"
password = "unique"
}
numThreads = 10
}
My Controller -- Database table is called - relations
package controllers
import play.api.mvc._
import slick.driver.PostgresDriver.api._
class Application extends Controller {
class relations(tag: Tag) extends Table[(Int,Int,Int)](tag, "relations") {
def id = column[Int]("id", O.PrimaryKey)
def me = column[Int]("me")
def following = column[Int]("following")
def * = (id,me,following)
}
val profiles = TableQuery[relations]
val db = Database.forConfig("mydb")
try {
// ...
} finally db.close()
def index = Action {
val q = for { p <- profiles if p.id === 2 } yield p.following
val updateAction = q.update(322)
val invoker = q.updateStatement
Ok()
}
}
What could be wrong with my code above ? I have a separate project that uses plain JDBC and this configuration works perfectly for it
db.default.driver=org.postgresql.Driver
db.default.url="jdbc:postgresql://localhost:5432/Jsmith"
db.default.user="postgres"
db.default.password="unique"
You did not run your action yet. db.run(updateAction) executes your query respectively your action on a database (untested):
def index = Action.async {
val q = for { p <- profiles if p.id === 2 } yield p.following
val updateAction = q.update(322)
val db = Database.forConfig("mydb")
db.run(updateAction).map(_ => Ok())
}
db.run() returns a Future which will be eventually completed. It is then simply mapped to a Result in play.
q.updateStatement on the other hand just generates a sql statement. This can be useful while debugging.
Look the code from my project:
def updateStatus(username: String, password: String, status: Boolean): Future[Boolean] = {
db.run(
(for {
user <- Users if user.username === username
} yield {
user
}).map(_.online).update(status)
}
I need to collect report data from a master-detail relations. Here is a simplified example:
case class Person(id: Int, name: String)
case class Order(id: String, personId: Int, description: String)
class PersonTable(tag: Tag) extends Table[Person](tag, "person") {
def id = column[Int]("id")
def name = column[String]("name")
override def * = (id, name) <>(Person.tupled, Person.unapply)
}
class OrderTable(tag: Tag) extends Table[Order](tag, "order") {
def id = column[String]("id")
def personId = column[Int]("personId")
def description = column[String]("description")
override def * = (id, personId, description) <>(Order.tupled, Order.unapply)
}
val persons = TableQuery[PersonTable]
val orders = TableQuery[OrderTable]
case class PersonReport(nameToDescription: Map[String, Seq[String]])
/** Some complex function that cannot be expressed in SQL and
* in slick's #join.
*/
def myScalaCondition(person: Person): Boolean =
person.name.contains("1")
// Doesn't compile:
// val reportDbio1:DBIO[PersonReport] =
// (for{ allPersons <- persons.result
// person <- allPersons
// if myScalaCondition(person)
// descriptions <- orders.
// filter(_.personId == person.id).
// map(_.description).result
// } yield (person.name, descriptions)
// ).map(s => PersonReport(s.toMap))
val reportDbio2: DBIO[PersonReport] =
persons.result.flatMap {
allPersons =>
val dbios = allPersons.
filter(myScalaCondition).map { person =>
orders.
filter(_.personId == person.id).
map(_.description).result.map { seq => (person.name, seq) }
}
DBIO.sequence(dbios)
}.map(ps => PersonReport(ps.toMap))
It looks far away from straightforward. When I need to collect master-detail data with 3 levels, it becomes incomprehensible.
Is there a better way?
I'm trying to change the save() method, but I don't find where I can overload it. I use typescript and node.js.
For the moment, I have a UserModel that contains a mongoose.Schema and a mongoose.Model.
When I call UserModel.getModel() I retrieve the mongoose.Model from the UserModel.
I basically use a DAO to retrieve the Model class object.
user = message.getDataByKey('user');
user.save(function(err, data) {
// do stuff
});
I want to overload automatically the user object to use my own .save() method to check if there is error and always handle them by the same way.
When I set the Model, I do it like that:
public static model: any = model.Models.MongooseModel.getNewInstance(UserModel.modelName, UserModel._schema);
And in the parent:
public static getNewInstance(modelName, schema){
var Model: any = mongoose.model(modelName, schema);
// Overload methods.
//console.log(new Model());
// Return overloaded Model class.
return Model;
}
I would like to know if there is any way to overload the Model to make sure that each new instance from it will have my own .save method.
I thought use the statics/methods (methods actually, I guess) but it's empty or I know that the final object will have save/remove/update methods. So I don't know why it's not already into the object, I tried to console.log(Model and new Model()) but no save() method.
So I'm a little desappointed, maybe I missed something.
The fact is, I can't update directly the new Model() because they will be created later, in another context, I need to update the Model directly to make sure that the new instance from this model will have my extra function.
And I don't want to rewrite the basic .save() method, I just want to overload it to add extra validation.
Any idea? I'm kinda lost here, it's not that easy. Thx.
I found a solution to do so, I'm using typescript so I'll post both .ts and .js to everybody understand.
I use CommonJs compilation
Model.ts (Super model, parent of all models)
///<reference path='./../../lib/def/defLoader.d.ts'/>
/**
* Package that contains all Models used to interact with the database.
* TODO Use options http://mongoosejs.com/docs/guide.html
*/
export module Models {
/**
* Interface for all Models, except the parent class.
*/
export interface IModel{
/**
* Name of the model.
* It's a helper to always get the name, from instance or static.
* MUST start by uppercase letter!
*/
modelName: string;
/**
* Contains the static value of the public schema as object.
* It's a helper to always get the schema, from instance or static.
*/
schema: mongoose.Schema;
/**
* Contains the static value of the object used to manipulate an instance of the model.
* It's a helper to always get the model, from instance or static.
*/
model: any;
}
/**
* Parent class for all models.
* A model contains a mongoose schema and a mongoose model and other things.
*/
export class Model{
/**
* Suffix used to load automatically models.
*/
public static suffix: string = 'Model';
/**
* Suffix used to load automatically models.
* It's a helper to always get the schema, from instance or static.
*/
public suffix: string;
/**
* Name of the model.
* MUST start by uppercase letter!
*/
public static modelName: string = '';
/**
* Readable schema as object.
*/
public static schema: any;
/**
* Schema as mongoose Schema type.
*/
public static Schema: mongoose.Schema;
/**
* The mongoose model that uses the mongoose schema.
*/
public static model: any;
/**
* Use static values as instance values.
*/
constructor(){
// Use static values as instance values.
this.suffix = Model.suffix;
}
/**
* Returns a new mongoose.Schema customized instance.
* #param ChildModel Child model that made the call.
* #returns {*}
* #see http://mongoosejs.com/docs/2.7.x/docs/methods-statics.html
*/
public static getNewSchemaInstance(ChildModel): mongoose.Schema{
var schema: any = new mongoose.Schema(ChildModel.schema, {collection: ChildModel.modelName.toLowerCase()});
// Overload methods.
//schema.methods.toObject = function(callback){}
// Return overloaded instance.
return schema;
}
/**
* Retrieves a new Model instance and overload it to add statics methods available for all Models.
* #param ChildModel
* #returns {*}
* #see http://mongoosejs.com/docs/2.7.x/docs/methods-statics.html
*/
public static getNewModelInstance(ChildModel): any{
// Get the Model class.
var Model: any = mongoose.model(ChildModel.modelName, ChildModel.Schema);
/**
**************************************************************************************************
************************ Extended Model static methods for all Models ****************************
**************************************************************************************************
*/
/**
* Handler for all database/mongoose errors.
* #param err Error.
* #param data Data. Contains the model and the emitter. (+ more)
* #param callback Callback function to execute.
*/
Model.errorHandler = (err: any, data: any, callback: (message: (any) => any) => any) => {
// Extract data.
var _Model = data.model;
var __function = data.__function;
var __line = data.__line;
// Will contains the error.
var message:any = [];
// Mongo error.
if(err && err.name && err.name == 'MongoError'){
var _err = MongoError.parseMongoError(err);
if(err.code == 11000){
// Duplicate key on create.
message[0] = '__19';
message[1] = [_err.value, _err.field];
}else if(err.code == 11001){
// Duplicate key on update.
message[0] = '__20';
message[1] = [_err.value, _err.field];
}else{
// Non-managed mongo error.
if(dev()){
// Return not only the message but also some information about the error.
message[0] = [];
// Message. [0][1] could be args.
message[0][0] = '__21';
// Data.
message[1] = {
err: err,
model: _Model.modelName
};
}else{
message = '__21';
}
}
fs.appendFile(__config.path.base + __config.mongo.error.log, new Date() + ': ' + JSON.stringify({error: err, model: _Model.modelName, _err: _err}) + '\n');
}else if(err && err.name && err.name == 'ValidationError'){
// Validation error from mongoose.
var _err = MongoError.parseValidationError(err);
message[0] = [];
// Message. [0][1] could be args.
message[0][0] = '__24';
message[0][1] = [_err[0].value, _err[0].field, _err[0].type];
if(dev()){
// Will be send as args but not displayed in the message.
message[1] = {
err: _err,
model: _Model.modelName
};
}
fs.appendFile(__config.path.base + __config.mongo.error.log, new Date() + ': ' + JSON.stringify({error: err, model: _Model.modelName, _err: _err}) + '\n');
}else{
// Another error? I don't know if that could happens, but manage it anyway.
message[0] = '__22';
if(dev()){
message[1] = [err, _Model.modelName];// Will be send as args but not displayed in the message.
}
fs.appendFile(__config.path.base + __config.mongo.error.log, new Date() + ': ' + JSON.stringify({error: err, model: _Model.modelName}) + '\n');
}
callback(message);// return an error.
};
/**
* Check if the object exists and returns it in this case.
* #param object Object to find.
* #param callback Callback to execute.
* #return
* err Error if it happens. [null]
* found Found object or false.
*/
Model.exists = (object, callback): any => {
// If object is null or false or empty or whatever, don't do the research, the result could be wrong!
if(!object){
callback (null, false);
}else{
Model.findOne(object, function (err, found) {
if (err){
Model.errorHandler(err, ChildModel, callback);
}else if (found){
callback(null, found);
}else{
callback (null, false);
}
});
}
};
// Return overloaded instance.
return Model;
}
}
/**
* Class that manage MongoDb errors, used statically.
*/
export class MongoError{
/**
* Parse a mongo error to returns data from it because Mongo returns really bad errors.
* #param err The mongo error.
* #returns {*}
*/
public static parseMongoError(err): any{
var _err: any = {};
var _message: string = err.err;
if(err.code == 11000 || err.code == 11001){
var message = _message.split(':');
// Get the table where the error was generated.
_err.table = message[1].split('.')[1];
// Get the field name where the error was generated.
_err.field = message[1].split('.')[2].split(' ')[0].replace('$', '');
_err.field = _err.field.substr(0, _err.field.lastIndexOf('_'));
// Get the
_err.value = message[3].split('"')[1].replace('\\', '');
}
return _err;
}
/**
* Parse a mongoose validation error, probably generated during a save/update function.
* #param err The mongoose error.
* #returns {*}
*/
public static parseValidationError(err): any{
var _errors: any = new Array();
var i = 0;
for(var error in err.errors){
_errors[i] = [];
_errors[i]['field'] = err.errors[error]['path'];
_errors[i]['value'] = err.errors[error]['value'];
_errors[i]['type'] = err.errors[error]['type'];
i++;
}
return _errors;
}
}
}
The JS version:
http://pastebin.com/xBTr1ZVe
Error messages (__21, etc.) are:
"__19": "Unable to add the element, the value **_$0** for the field _$1 already exists, it cannot be duplicated.",
"__20": "Unable to update the element, the value **_$0** for the field _$1 already exists, it cannot be duplicated.",
"__21": "Unable to execute the requested operation. Database error 21. Please report to an administrator.",
"__22": "Unable to execute the requested operation. Database error 22. Please report to an administrator.",
"__23": "Validation error, the requested operation was aborted. Please check that all your data are correct and retry. Please report to an administrator. (code 23)",
"__24": "Unable to perform the operation, the value ***_$0** for the field _$1 didn't pass successfully the validation. Error: _$2",
Basically all my models should manage by themself these exception, of course. But if I forgot to do it, I'll get a managed exception, better.
Now I'll post a real Model, UserModel inheriting the parent Model.
///<reference path='./../../lib/def/defLoader.d.ts'/>
import model = require('./Model');
export module Models {
/**
* Model used to manage users.
* The model is primary static, but, to make it easy to use, some things are also stored for each instance.
* That allows the code to use both Model or instance of Model such as:
* Model.schema
* model.Schema
*/
export class UserModel extends model.Models.Model implements model.Models.IModel{
/**
*************************************************************************************************
****************************** Public methods & attributes **************************************
*************************************************************************************************
*/
/**
* Name of the model.
* MUST start by uppercase letter!
*/
public static modelName: string = 'User';
/**
* Readable schema as object.
*/
public static schema: any = require('../schemas/userSchema.js');
/**
* Schema as mongoose Schema type.
*/
public static Schema: mongoose.Schema = model.Models.Model.getNewSchemaInstance(UserModel);
/**
* The mongoose Model that uses the mongoose schema.
*/
public static model: any = model.Models.Model.getNewModelInstance(UserModel);
/**
* Helpers to always get the property, from instance or static.
*/
public modelName: string = UserModel.modelName;
public schema: mongoose.Schema = UserModel.schema;
public model: mongoose.Model<any> = UserModel.model;
/**
*************************************************************************************************
***************************** Extended methods & attributes **************************************
*************************************************************************************************
*/
/**
* These fields are protected, the user password is required to access to them.
* These fields are basically shared between applications.
* #private
*/
private static _protectedFields: string[] = [
'login',
'email'
];
/**
* Method to use to hash the user password.
*/
private static _passwordHashMethod: string = 'sha256';
/**
* Digest to use to hash the user password.
*/
private static _passwordDigest: string = 'hex';
/**
* Returns the protected fields.
* #returns {string[]}
*/
public static getProtectedFields(): string[]{
return this._protectedFields;
}
/**
* Hash a user password depending on the password hash configuration. Currently SHA256 in hexadecimal.
* Assuming crypto is global.
* #param password User password.
* #returns {string} Hashed password.
*/
public static hashPassword(password: string): string{
return crypto
.createHash(UserModel._passwordHashMethod)
.update(password)
.digest(UserModel._passwordDigest)
}
}
/**
* Don't forget that some methods such as exists() are written in the Model class and available for all Models.
* The following methods belong ONLY to the mongoose model instance, not to the Model class itself!
*
*************************************************************************************************
******************************** Extended Model methods *****************************************
*************************************************************************************************
*/
/**
* Connect a user to the game.
* #param user User to check. {}
* #param callback Callback to execute.
*/
UserModel.model.checkAuthentication = (user, callback) => {
// Force to provide login and password.
UserModel.model.exists({login: user.login, password: UserModel.hashPassword(user.password)}, function(err, userFound){
// Load public profile.
UserModel.model._getProtectedInformation(userFound, function(userPublic){
// Provides only public fields.
callback(new __message("__17", {err: err, user: userPublic}, !err && userFound ? true: false));
});
});
};
/**
* Get the protected fields for the found user.
* #param user User to find.
* #param callback Callback to execute.
*/
UserModel.model.getProtectedInformation = (user, callback) => {
// We are looking for an unique user.
UserModel.model.exists(user, function(err, userFound){
if(err){
UserModel.model.errorHandler(err, UserModel, callback);
}else{
// Load public profile.
UserModel.model._getProtectedInformation(userFound, function(userPublic){
// Provides only public fields.
callback(new __message('', {err: err, user: userPublic}, err ? false: true));
});
}
});
};
/**
* Get the protected fields of a user.
* #param user Instance of model.
* #param callback Callback to execute.
* #private
*/
UserModel.model.hashPassword = (user, callback): any => {
var err = false;
if(user && user.password){
user.password = UserModel.hashPassword(user.password);
}else{
err = true;
}
callback(new __message(err ? '__18': '', {user: user}, err ? false: true));
};
/**
*************************************************************************************************
*************************** Methods to use only locally (private) *******************************
*************************************************************************************************
*/
/**
* Get the protected fields of a user.
* #param user Instance of model.
* #param callback Callback to execute.
* #private
*/
UserModel.model._getProtectedInformation = (user, callback): any => {
var userPublic = {};
// Get fields to share.
var publicFields = UserModel.getProtectedFields();
// Fill the userPublic var with public fields only.
for(var field in publicFields){
userPublic[publicFields[field]] = user[publicFields[field]];
}
callback(userPublic);
};
}
The JS version:
http://pastebin.com/0hiaMH25
The schema:
/**
* Schema ued to create a user.
* #see http://mongoosejs.com/docs/2.7.x/docs/schematypes.html
*/
module.exports = userSchema = {
/**
* User Login, used as id to connect between all our platforms.
*/
login: {
type: 'string',
//match: /^[a-zA-Z0-9_-]{'+userSchema.login.check.minLength+','+userSchema.login.check.maxLength+'}$/,
trim: true,
required: true,
notEmpty: true,
unique: true,
check: {
minLength: 4,
maxLength: 16
}
},
/**
* User email.
*/
email: {
type: 'string',
lowercase: true,
match: /^[a-zA-Z0-9._-]+#[a-zA-Z0-9.-]+\.[a-zA-Z]{2,4}$/,
required: true,
notEmpty: true,
unique: true,
check: {
minLength: 6,
maxLength: 30
}
},
/**
* User private password, the one hashed in SHA512 and stored on the database.
*/
password: {
type: 'string',
required: true,
check: {
length: 128
}
},
/**
* Salt to use to decrypt the password.
*/
passwordSalt: {
type: 'string',
check: {
length: 64
}
},
/**
* Password sent from user interface but hashed before be send on the network.
* Used to basically connect an user or generate the final password.
* Not stored in the DB.
*/
passwordProtected: {
type: 'string',
check: {
length: 64
}
},
/**
* Password wrote by the user on the GUI, not hashed or encrypted.
* Will be encrypted to respect the "passwordProtected" rules.
* Not stored in the DB.
*/
passwordPublic: {
type: 'string',
check: {
minLength: 8,
maxLength: 25
}
},
/**
* User banned status (Temporary of Definitive)
*/
banned: {
temporary : {
type : "number",
default : Date.now
},
definitive: {
type: 'boolean',
default: false
}
},
/**
* User right
*/
right : {
admin : {
type : "boolean",
default : false,
required: true
},
moderator : {
type : "boolean",
default : false,
required: true
}
}
};
So, what the code does?
Basically, in the Model.getNewModelInstance() I bind to the created model the errorHandler method that I will call if I found a DB error in the controller.
**UserController.js**
User.exists({email: user.email}, function(err, emailFound){
// If we got an err => Don't find couple User/pass
if (err) {
User.errorHandler(err, {model: User, __filename: __filename,__function: __function || 'subscription#exists', __line: __line}, function(err){
res.json(__format.response(err));
});
)
});
The __filename and so on are global functions that I use to get the current data, useful to debug. I'm still looking for a way to add this automatically but so far I couldn't. The __function doesn't exists when the function is anonymous. But it helps me to debug.
Any suggestion? That's a lot of piece of code.