For example one of my entities has two sets of IDs.
One that is continuous (which apparently is necessary to create the EntitySet), and one to use as a foreign key when merging with my other table.
This results in featuretools including the ID in the set of features to aggregate. SUM(ID) isn't a feature I am interested in though.
Is there a way to include certain feature when running deep feature synthesis?
There are three ways to exclude features when calling ft.dfs.
Use the ignore_variables to specify variables in an entity that should not be used to create features. It is a dictionary mapping an entity id to a list of variable names to ignore.
Use drop_contains to drop features that contain any of the strings
listed in this parameter.
Use drop_exact to drop features that exactly match any of the strings listed in this parameter.
Here is a example usage of all three in a ft.dfs call
ft.dfs(target_entity="customers"],
ignore_variables={
"transactions": ["amount"],
"customers": ["age", "gender", "date_of_birth"]
}, # ignore these variables
drop_contains=["customers.SUM("], # drop features that contain these strings
drop_exact=["STD(transactions.quanity)"], # drop features named exactly this
...
)
These 3 parameters are all documented here.
The final thing to consider if you are getting features you don't want is the variable types of the variables in your entity set. If you are seeing the sum of an ID variable that must mean that featuretools thinks the ID variable is a numeric value. If you tell featuretools it is an ID it will not apply a numeric aggregation to it.
Related
I have two lists which need to be joined before being used in for_each. Can the setunion be used directly with lists or each list needs to be casted using toset before setunion can be used?
The setunion function knows that its arguments are supposed to be sets and so it will automatically convert from list to set if you pass in a list value.
In practice the following two expressions are equivalent:
toset(concat(list_a, list_b))
setunion(list_a, list_b)
The difference is only in the method Terraform takes to calculate the result. In the first case it will first construct a new list which is the concatenation of list_a and list_b, and then convert that result to a set. In the second case it'll convert list_a and list_b to sets first and then calculate the union of those two sets. Because of the definition of setunion, these two methods will always produce the same result.
I think you may be asking this question because for for_each Terraform does not automatically convert from a list to a set.
The for_each argument has a special rule to disable Terraform's normal automatic conversion rule in this case because during the early alpha/beta testing period for for_each the team observed that people were often confused about what assigning a list to for_each would mean:
Some people expecting it to behave like count = length(list), while other people expected it to behave like for_each = toset(list).
Because the difference between these is subtle and may cause problems with later updates if the wrong interpretation were used, the Terraform team chose to force module authors to be explicit about which of these two interpretations they mean, either by explicitly converting the list to a set or by using the count argument instead to indicate that they want to use numeric indices for instance keys.
However, Terraform doesn't actually care how you produce the set of strings assigned to for_each, so you can use any expression whose result is of that type. setunion when given sets of strings (or anything that can convert to sets of strings) will always produce a set of string as a result, and so a call to that function is also a valid thing to assign to for_each:
for_each = setunion(a, b)
No explicit conversion is needed in this case because setunion inherently returns a list as part of its definition.
I want to map a timestamp t and an identifier id to a certain state of an object. I can do so by mapping a tuple (t,id) -> state_of_id_in_t. I can use this mapping to access one specific (t,id) combination.
However, sometimes I want to know all states (with matching timestamps t) of a specific id (i.e. id -> a set of (t, state_of_id_in_t)) and sometimes all states (with matching identifiers id) of a specific timestamp t (i.e. t -> a set of (id, state_of_id_in_t)). The problem is that I can't just put all of these in a single large matrix and do linear search based on what I want. The amount of (t,id) tuples for which I have states is very large (1m +) and very sparse (some timestamps have many states, others none etc.). How can I make such a dict, which can deal with accessing its contents by partial keys?
I created two distinct dicts dict_by_time an dict_by_id, which are dicts of dicts. dict_by_time maps a timestamp t to a dict of ids, which each point to a state. Similiarly, dict_by_id maps an id to a dict of timestamps, which each point to a state. This way I can access a state or a set of states however I like. Notice that the 'leafs' of both dicts (dict_by_time an dict_by_id) point to the same objects, so its just the way I access the states that's different, the states themselves however are the same python objects.
dict_by_time = {'t_1': {'id_1': 'some_state_object_1',
'id_2': 'some_state_object_2'},
't_2': {'id_1': 'some_state_object_3',
'id_2': 'some_state_object_4'}
dict_by_id = {'id_1': {'t_1': 'some_state_object_1',
't_2': 'some_state_object_3'},
'id_2': {'t_1': 'some_state_object_2',
't_2': 'some_state_object_4'}
Again, notice the leafs are shared across both dicts.
I don't think it is good to do it using two dicts, simply because maintaining both of them when adding new timestamps or identifiers result in double work and could easily lead to inconsistencies when I do something wrong. Is there a better way to solve this? Complexity is very important, which is why I can't just do manual searching and need to use some sort of HashMap magic.
You can always trade add complexity with lookup complexity. Instead of using a single dict, you can create a Class with an add method and a lookup method. Internally, you can keep track of the data using 3 different dictionaries. One uses the (t,id) tuple as key, one uses t as the key and one uses id as the key. Depending on the arguments given to lookup, you can return the result from one of the dictionaries.
I have a situation where I need to store some intermediate values so I can reuse them in other parts of the root module. I know about local values and I know about null_data_source except I do not know which one is the recommended option for holding re-usable values. Both descriptions look somewhat similar to me
local values (https://www.terraform.io/docs/configuration/locals.html)
Local values can be helpful to avoid repeating the same values or expressions multiple times in a >configuration, but if overused they can also make a configuration hard to read by future >maintainers by hiding the actual values used.
and null_data_source (https://www.terraform.io/docs/providers/null/data_source.html)
The primary use-case for the null data source is to gather together collections of intermediate >values to re-use elsewhere in configuration:
So both appear to be a valid choice for this scenario.
Here is my example code
locals {
my_string_A = "This is string A"
}
data "null_data_source" "my_string_B" {
inputs = {
my_string_B = "This is string B"
}
}
output "my_output_a" {
value = "${local.my_string_A}"
}
output "my_output_b" {
value = "${data.null_data_source.my_string_B.outputs["my_string_B"]}"
}
Could you suggest on when to use the one over the other for holding intermediate values and what is the pros/cons of each approach?
Thank you
The null_data_source data source was introduced prior to the local values mechanism as an interim solution to meet that use-case before that capability became first-class in the language. It continues to be supported only for backward-compatibility with existing configurations using it.
All new configurations should use the Local Values mechanism instead. It's fully integrated into the Terraform language, supports values of any type (while null_data_source can support only strings), and has a much more concise/readable syntax.
I'm using mutual_info_classif to determine the most important words for a binary text-classification task as:
mi_score = mutual_info_classif(X, y)
but the above gives an array of feature scores without reference to the corresponding classes
Is there a way to get the most important features per class using MI?
P.s., I've already tried Chi2 but it gives the same feature rank for both classes
Mutual information is a measure of dependence between 2 variables. In your case, between each of the attribute variables and the "Class" variable. The mutual information will give a higher score, when the attribute variable creates a better split of the target variable. This means you only get one score that describes the strength between the attribute and the class. The most important feature is the one that best distinguishes between all of the classes.
If you have a class with multiple labels (not a binary class), you can create a new class variable for each label by using dummy variables.
For example, assume your class name is CLASS, and it holds 3 different labels: "Red", "Green" and "Blue". Create 3 new target variables, the first will be called "Is_Red", and it will hold "Yes" if CLASS=="Red" or "No" Otherwise. In this manner you can see which attribute best distinguish between each specific instance of the class. You will have to run the mutual information per new class variable.
If you have ever worked with the metamodel of UML, you propably know the concepts of unions and subsets - As far as I understand it:
Attributes and associations of an element/class marked as "derived union" cannot be used directly. In more specific sub-classes, you can possibly find subsets of them that can be used, as long as they are not marked as derived unions themselves.
"derived" (without union) attributes and associations have also subsets in more specific classes, but unlike above you can use them directly without having to look for subsets in more specific classes
My questions:
Does this make sense or am I on the wrong track here?
What is the meaning of the "/" (slash) you can find in front of some
attributes/associations, that they have subsets in child-classes?
E.g. /general : Classifier[*]
An union property is a property that consists of multiple other properties. You can only understand the union, when you combine all subsets. A list is almost by definition an union.
Almost, because it might be uninitialized.
A derived union is a property requiring a specific collection of subsets. I would not talk about accessing them directly, but about how direct you can understand them. You need all information before you can make an interpretation.
The difference between the two that a derived union requires a specific subset and an union might have a subset and might have different subsets in different contexts. A very simple example being the fields on a form. All required fields show the definition of a derived union. All other fields are part of the complete union.
Derived unions can contain derived unions in their subsets. It directs the creation of classes and their instances, it does not make them impossible.
All derived features require other features to be known. Temperature can be read directly, but to know if someone has fever requires more knowledge, like the time of the day, the place of collecting information etc..
The slash implies that it is being derived.