My team uses Opaleye to query Postgres from Haskell.
However, we also use raw SQL to do such things as:
Initialize the database. Including commands create database and create table
Perform database migrations when we change our schema, including varied commands.
Is it possible to entirely replace this raw SQL code with Opaleye?
Can Opaleye replace any SQL?
According to the Opaleye tutorial, creating tables and databases is currently unsupported:
Opaleye assumes that a Postgres database already exists. Currently
there is no support for creating databases or tables, though these
features may be added later according to demand.
Related
Currently, migration scripts (node-pg-migrate) build the entire database from zero to everything. It's the database of a new system that is going to replace the legacy system. Data needs to be imported from the legacy system to the new database. They have completely different data structures.
The migrations scripts build the import schema with raw data imported from the legacy system (using its web service). Then all other schemas with its tables, functions and everything is created. Primarily the data schema with data transformed from the import schema usable for the new system. The api schema with views and functions exposed through postgREST working on data from the data schema. And some more helper schemas and stuff.
Now, the data to be imported is not final yet, so I need to re-import often. To do that, I need to migrate all the way down, dropping all other schemas as it goes down to get to the migration steps that remove all imported data and drops the import schema. Then go up again to import the data all the way up again to build all the schemas to have a working api again.
I'm getting to my question now shortly... I'm quite sure I need to move the import data scripts away from the migration so I don't need to deconstruct and reconstruct the entire database and all its schemas. Ideally, I want to run import scripts and schema scripts independently. Using node-pg-migrate though is really convenient, also for importing the data.
How can I use node-pg-migrate with independent lanes of migrations? One for imports (or dml changes) and one for schema changes (or ddl changes).
Related:
Migration scripts for Structure and Data
https://dba.stackexchange.com/questions/64632/how-to-create-a-versioning-ddl-dml-change-script-for-sql-server
Update: I just found out the solution may lie in the area of scope as implemented by node-db-migrate. I'm using node-pg-migrate though...
node-pg-migrate does not support lanes as e.g. db-migrate. But you can emulate them IMO in a way, that you will use separate migrations folder and table:
node-pg-migrate -m migrations/dll -t pgmigrations_dll
node-pg-migrate -m migrations/dml -t pgmigrations_dml
Is there any way i can maintain table relationship in Hive, Spark SQL and Azure U SQL. Does any of these support creating relationships like in Oracle or SQL Server. In oracle i can get the relationships using user_constraints table. Looking for something similar to that for processing in ML.
I don't believe any of the big data platforms support integrity constraints. They are designed to load data as quickly as possible. Having constraints will only slow down the import. Hope this makes sense
None of the big data tools maintain constraints. If we just consider hive it doesn't even bother while you are writing data to the table whether the table schema is maintained or not as Hive follows schema on read approach.
While reading the data if you want to establish any relation, one has to work with joins.
I have gone through this chapter and all the examples mentioned there are making the database schema initially and then operating over them.
But what to do if there is already an existing database schema with data ? Is there any way to create schema from it ?
You can use the sql= attributes in the models file to specify the table and column names in the existing schema. In this setup, you likely want to avoid using Persistent's automatic migration capabilities as well.
Database.Persist seems to be index-agnostic. This is okay, I can create my own indices, but generic SQL migration seems to create and drop tables when adding/removing fields. This has the effect of dropping the index as well.
Is there a recommended way to make sure they survive database migrations?
Only the SQLite3 backend should be dropping tables, PostgreSQL and MySQL both provide powerful enough ALTER TABLE commands to avoid that. So indices should only be lost for SQLite3. If you're using SQLite3 in production (not really recommended), you have two choices:
Disable automatic migrations and handle the schema yourself.
Add some code after the migrations are run to replace any missing indices.
In http://www.yesodweb.com/book/persistent there is no mention of SQL views.
I (even in imperative languages) have been very fond of immutable database schema design. i.e. only INSERTs and SELECTs - UPDATEs and DELETEs are not used.
This has the advantage of preserving all history, at the expense of making the current 'state' a relatively expensive pure function of the history in the DB.
e.g. there is not a 'user' table, just 'user_created', 'user_password_updated' and 'user_deleted' tables, which are unified in a 'user' SQL VIEW, showing the current state of users.
How should I work with VIEWs in Persistent? Should I use Persistent at all - is it (ironically for Haskell) too tightly focused on a mutable DB for my use-case?
It's been a long time since the question was asked, but I thought it was worth
responding because seven years later the answer has not changed and I really like
your idea about keeping the old versions of tables around and reading them with
views! One drawback of this approach is that using Template Haskell in persistent
will slow down compile times a lot. Once I had a database of about 50 tables in
persistent Template Haskell and it took over half an hour to compile if it was
ever changed.
Yesod's persistent does not support
SQL views and I doubt it ever will because it intends to be database agnostic.
Currently it looks like it supports CouchDB, MongoDB, MySQL , PostgreSQL, Redis
and SQLite. Not all of these databases support SQL style views so it would be
hard to abstract over all of them.
Where persistent excels at is at providing an easy way to create a set of
Haskell types that serialize to and
from different databases. It provides with type class instances and functions to
do single table queries and these work really well. If you want to do join queries on an SQL database that
you are interfacing with persistent, then you can use esqueleto,
a type safe EDSL for SQL join queries.
As far as handling SQL Views in Haskell, I have not come across any tool yet.
You can either use rawQuery which will work but be harder to maintain or
you can build your own tool around one the Haskell DB interfaces like postgresql-simple,
which is what persistent does. In fact, you can even start with the persistent
source code of whatever database you are using and build an SQL View EDSL as you
need. In a closed-source project I helped build a custom PostgreSQL
interface based on some of persistent's ideas and types, but without using
an Template Haskell because the compile time was too slow.