How to query this with ORM? - kohana

I've using Kohana for a couple of weeks. One thing I noticed is that Kohana is missing eager loading (as far as I know). Let's say I have the following tables.
Subjects
id
name
Chapters
id
subject_id
name
Videos
id
chapter_id
name
When a user opens a subject page, I want to display all the chapters and videos. With ORM, I can do
$tutorials = ORM::factory('subject')->where('id','=', 1)->find();
foreach($tutorials as $tutorial)
{
$chapters = $tutorial->chapters->find_all();
foreach($chapters as $chapter)
{
$videos = $chapter->videos->find_all();
}
}
The above code is not efficient since it makes too many queries.
I thought about using join or database query builder, but both of them do not return a model object as their results. I also looked into with(), but it seems like it only works with one-to-one relationship.
using join on an ORM object returns an OPM object, but it doesn't return the data from the joining tables.
What would be my best option here? I would like to minimize # of queries and also want to get ORM objects a result. Whatever it would be, should return all the columns from tutorials, chapters, and videos.

First of all, your code is excess. ORM method find() returns 1 Model_Subject object. See
$chapters = ORM::factory('subject', 1)->chapters->find_all();
foreach($chapters as $chapter)
{
$videos = $chapter->videos->find_all();
}
With DB builder you can make just 2 requests. First get array of all chapters ids:
$chapters = DB::select('id')
->from('chapters')
->where('subject_id', '=', '1')
->execute()
->as_array(NULL, 'id');
Second - get all videos by ids as Model_Video object
$videos = DB::select('id')
->from('videos')
->where('chapter_id', 'IN', $chapters)
->as_object('Model_Video')
->execute()
->as_array();

So I guess you want something like this.
$videos = ORM::factory('Video')
->join(array('chapters', 'chapter'), 'LEFT')->on('video.chapter_id', '=', 'chapter.id')
->join(array('subjects', 'subject'), 'LEFT')->on('chapter.subject_id', '=', 'subject.id')
->where('subject.id', '=', $id)
->find_all();
Come to think of it, if the video belongs_to chapter belongs_to subject, try the following using with():
$videos = ORM::factory('Video')
->with('chapter:subject') // These are the names of the relationships. `:` is separator
// equals $video->chapter->subject;
->where('subject.id', '=', $id)
->find_all();
With things like this it often helps to think 'backwards'. You need the videos on that subject so start with the videos instead of the subject. :)
EDIT: The drawback of the second function is that it is going to preload all the data, it might be shorter to write but heavier on the server. I'd use the first one unless I need to know the subject and chapter anyway.

Related

ArangoDB populate relation as field over graph query

I recently started using Arango since I want to make use of the advantages of graph databases. However, I'm not yet sure what's the most elegant and efficient approach to query an item from a document collection and applying fields to it that are part of a relation.
I'm used to make use of population or joins in SQL and NoSQL databases, but I'm not sure how it works here.
I created a document collection called posts. For example, this is a post:
{
"title": "Foo",
"content": "Bar"
}
And I also have a document collection called tags. A post can have any amount of tags, and my goal is to fetch either all or specific posts, but with their tags included, so for example this as my returning query result:
{
"title": "Foo",
"content": "Bar",
"tags": ["tag1", "tag2"]
}
I tried creating those two document collections and an edge collection post-tags-relation where I added an item for each tag from the post to the tag. I also created a graph, although I'm not yet sure what the vertex field is used for.
My query looked like this
FOR v, e, p IN 1..2 OUTBOUND 'posts/testPost' GRAPH post-tags-relation RETURN v
And it did give me the tag, but my goal is to fetch a post and include the tags in the same document...The path vertices do contain all tags and the post, but in separate arrays, which is not nice and easy to use (and probably not the right way). I'm probably missing something important here. Hopefully someone can help.
You're really close - it looks like your query to get the tags is correct. Now, just add a bit to return the source document:
FOR post IN posts
FILTER post._key == 'testPost'
LET tags = (
FOR v IN 1..2 OUTBOUND post
GRAPH post-tags-relation
RETURN v.value
)
RETURN MERGE(
post,
{ tags }
)
Or, if you want to skip the FOR/FILTER process:
LET post = DOCUMENT('posts/testPost')
LET tags = (
FOR v IN 1..2 OUTBOUND post
GRAPH post-tags-relation
RETURN v.value
)
RETURN MERGE(
post,
{ tags }
)
As for graph definition, there are three required fields:
edge definitions (an edge collection)
from collections (where your edges come from)
to collections (where your edges point to)
The non-obvious vertex collections field is there to allow you to include a set of vertex-only documents in your graph. When these documents are searched and how they're filtered remains a mystery to me. Personally, I've never used this feature (my data has always been connected) so I can't say when it would be valuable, but someone thought it was important to include.

TYPO3: Performance issue with pagination

I am currently building some kind of video channel based on an extension I created. It consists of videos and playlist that contains videos (obviously).
I have to create a page which contains a list of videos AND playlist by category. You also can sort those items by date. Finally, the page is paginated with an infinite scrolling that should load items 21 by 21.
To do so, I created on both Video and Playlist repositories a "findByCategory" function which is really simple :
$query = $this->createQuery();
return $query->matching($query->equals('categorie.uid',$categoryUid))->execute()->toArray();
Once I requested the items I need, I merge them in one array and do my sorting stuff. Here is my controller show action :
if ($this->request->hasArgument('sort'))
$sort = $this->request->getArgument('sort');
else
$sort = 'antechrono';
//Get videos in repositories
$videos = $this->videoRepository->findByCategorie($categorie->getUid());
$playlists = $this->playlistRepository->findByCategorie($categorie->getUid());
//Merging arrays then sort it
if ($videos && $playlists)
$result = array_merge($videos, $playlists);
else if ($videos)
$result = $videos;
else if ($playlists)
$result = $playlists;
if ($sort == "chrono")
usort($result, array($this, "sortChrono"));
else if ($sort == "antechrono" || $sort == null)
{
usort($result, array($this, "sortAnteChrono"));
$sort="antechrono";
}
$this->view->assignMultiple(array('categorie' => $categorie, 'list' => $result, 'sort' => $sort));
Here is my view :
<f:widget.paginate objects="{list}" as="paginatedList" configuration="{addQueryString: 'true', addQueryStringMethod: 'GET,POST', itemsPerPage: 21}">
<div class="videos row">
<f:for each="{paginatedList}" as="element">
<f:render partial="Show/ItemCat" arguments="{item: element}"/>
</f:for>
</div>
</f:widget.paginate>
The partial render shows stuff including a picture used as a cover. So I need at least this relation in the view.
This works fine and shows only the items from the category that is requested. Unfortunatly I have a huge performance issue : I tried to show a category that contains more than 3000 records and It takes about one minute to load. It's a little bit long.
By f:debugging my list variable, I see that it contains every records even through it shouldn't be the case (that's the point of pagination...). So the first question is : is there something wrong in the way I did my pagination ?
I tried to simplify my requests by enabling the rawQuery thing ($query->execute(true)) : I get way better performance, but I can't get the link for the pictures (in my view, I get 1 or 0 but not the picture's uid...). Second question : is there a way to fix this issue ?
I hope my description is clear enough. Thanks for your help :-)
When you execute a query, it will not actually fetch the data from the database until the results are accessed. If the paginate widget gets a query result it will add limits and offset to the query and then fetch the data from the database, so you will only get the records that are shown on a page.
In your case you added toArray() after execute(), which accesses the results, so the data is fetched from the database and you get all records. The best solution I can think of is to combine the 2 tables into 1 so you can do it with a single query and don't have to merge and order them in PHP.
As long as you sort the data after the query you have to handle all data (request all records and especially resolve all relations).
Try to sort the data in the query itself (order by), so you could restrict the data to only those records which are needed for the current 'page' (limit <offset>,<number>).
Here a complex query with join and limit could be faster than a full query and filtering in PHP.

Flask-AppBuilder equivalent of SQLite WHERE clause to filter column data

I'm new to Flask and have started designing a front end for an inventory management database using Flask-AppBuilder.
I have created several models and have have managed to display my sqlite data in tables using Flask-AppBuilder's views.
However, I don't seem to be able to find the equivalent of SQLite WHERE clause to filter or "restrict" column data. I've been reading a lot about sqlalchemy, filters, queries but this has left me more confused that anything else and the explanations seem to be extremely elaborate and complicated to do something which is extremely simple.
Assuming we reproduce the following SQLite query in Flask-AppBuilder:
SELECT Field_A
FROM Table_A
WHERE Field_A = 'some text'
with:
result = session.query(Table_A).filter_by(Field_A = 'some text').all()
Where does the above line of code go in my app?
Considering I have the following Class:
class Table_A(Model):
id = Column(Integer, primary_key=True)
Field_A = Column(String)
def __repr__(self):
return self
and View:
class Table_AView(ModelView):
datamodel = SQLAInterface(Table_AView)
label_columns = {'Field_A':'A'}
list_columns = ['Field_A']
After much digging flask-appbuilder uses it's own filterclass in order to enable you to filter your views.
All the classes are referenced here on GitHub:
Flask Filter Clases List
Also not the difference between FilterEqual and FilterEqualFunction here:
What is the difference between : FilterEqual and FilterEqualFunction?
For other customisation and first port of call of Flask-appbuilder go straight to the API Reference where you'll find a couple of examples of the filterclass in action.
In essence it is extremely simple. In your views.py code within the ModelView class you want to filter simply add base_filters = [['field_A', FilterEqual, 'abc']] like so:
`class Table_AView(ModelView):
datamodel = SQLAInterface(Table_AView)
label_columns = {'Field_A':'A'}
list_columns = ['Field_A']
base_filters = [['field_A', FilterEqual, 'abc']]`
This will only show the lines where the field_A variable is equal to abc.
Hope this helps someone as it took me nearly (sigh) two weeks to figure it out...
SQLALchemy is an ORM (Object-Relational Mapping), it mean that you dont have to deal with raw SQL, you will call a function that you "build" (by adding filters in your case). It will transparently generate an SQL query, execute it, and return the result as python objects.
I would suggest you to read closely at sqlalchemy documentation about filters again, especially filter_by :
http://docs.sqlalchemy.org/en/latest/orm/query.html#sqlalchemy.orm.query.Query.filter_by
It is the easiest way to apply a WHERE with sqlalchemy.
If you have declared correctly the model for Table_A, you should be able to use it so:
result = session.query(Table_A).filter_by(Field_A = 'some text').all()
Here session.query(Table_A).filter_by(Field_A = 'some text') will generate the SQL, and .all() will execute it.

loopback relational database hasManyThrough pivot table

I seem to be stuck on a classic ORM issue and don't know really how to handle it, so at this point any help is welcome.
Is there a way to get the pivot table on a hasManyThrough query? Better yet, apply some filter or sort to it. A typical example
Table products
id,title
Table categories
id,title
table products_categories
productsId, categoriesId, orderBy, main
So, in the above scenario, say you want to get all categories of product X that are (main = true) or you want to sort the the product categories by orderBy.
What happens now is a first SELECT on products to get the product data, a second SELECT on products_categories to get the categoriesId and a final SELECT on categories to get the actual categories. Ideally, filters and sort should be applied to the 2nd SELECT like
SELECT `id`,`productsId`,`categoriesId`,`orderBy`,`main` FROM `products_categories` WHERE `productsId` IN (180) WHERE main = 1 ORDER BY `orderBy` DESC
Another typical example would be wanting to order the product images based on the order the user wants them to
so you would have a products_images table
id,image,productsID,orderBy
and you would want to
SELECT from products_images WHERE productsId In (180) ORDER BY orderBy ASC
Is that even possible?
EDIT : Here is the relationship needed for an intermediate table to get what I need based on my schema.
Products.hasMany(Images,
{
as: "Images",
"foreignKey": "productsId",
"through": ProductsImagesItems,
scope: function (inst, filter) {
return {active: 1};
}
});
Thing is the scope function is giving me access to the final result and not to the intermediate table.
I am not sure to fully understand your problem(s), but for sure you need to move away from the table concept and express your problem in terms of Models and Relations.
The way I see it, you have two models Product(properties: title) and Category (properties: main).
Then, you can have relations between the two, potentially
Product belongsTo Category
Category hasMany Product
This means a product will belong to a single category, while a category may contain many products. There are other relations available
Then, using the generated REST API, you can filter GET requests to get items in function of their properties (like main in your case), or use custom GET requests (automatically generated when you add relations) to get for instance all products belonging to a specific category.
Does this helps ?
Based on what you have here I'd probably recommend using the scope option when defining the relationship. The LoopBack docs show a very similar example of the "product - category" scenario:
Product.hasMany(Category, {
as: 'categories',
scope: function(instance, filter) {
return { type: instance.type };
}
});
In the example above, instance is a category that is being matched, and each product would have a new categories property that would contain the matching Category entities for that Product. Note that this does not follow your exact data scheme, so you may need to play around with it. Also, I think your API query would have to specify that you want the categories related data loaded (those are not included by default):
/api/Products/13?filter{"include":["categories"]}
I suggest you define a custom / remote method in Product.js that does the work for you.
Product.getCategories(_productId){
// if you are taking product title as param instead of _productId,
// you will first need to find product ID
// then execute a find query on products_categories with
// 1. where filter to get only main categoris and productId = _productId
// 2. include filter to include product and category objects
// 3. orderBy filter to sort items based on orderBy column
// now you will get an array of products_categories.
// Each item / object in the array will have nested objects of Product and Category.
}

Kohana 3.2 ORM Does not contain model info

I'm working with Kohana 3.2 and have the following code in my controller.
$Blog_Post = new Model_Blogpost();
$Blog_Post->where('id', '=', 1);
$Blog_Post->find();
$content = $Blog_Post->content;
I Currently have 3 records in my db with id's 1, 2, and 3.
$Blog_Post->content, or any other field return null. and I'm not sure why.
Use ORM::factory('blogpost', $id) or new Model_Blogpost($id) if you need an object with PK == $id.
Check your model after loading.
if $Blog_Post->loaded()
{
// it works!
}
else
{
// record not found
}
If record not found, you can see last DB query with $Blog_Post->last_query()
UPD. From comments. Your model will not work with this modifications. Note that ORM data stored in $_object property, and $Blog_Post->content is just a shortcut for $Blog_Post->_object['content'] via __get() method. Of course, if you define public $content property, $Blog_Post->content will return NULL value instead of using DB data.
There is no reason for defining model fields as properties. If you need IDE hints, just use PHPDOC.
At the firm I work for we were looking into upgrading to 3.2 very recently. However, in our evaluation I don't recall seeing a difference in ORM handling methods.
Yours above looks like it should be something like this:
$Blog_Post = ORM::factory('blogpost')->where('id', '=', 1)->find();
$content = $Blog_Post->content;
Assuming your table is called blogposts, of course. I may be wrong about that and if I am, can you link to the documentation that shows this kind of model interaction?

Resources