I'm a complete newbie to Reinforcement Learning and have been searching for a framework/module to easily navigate this treacherous terrain. In my search I've come across two modules keras-rl & OpenAI GYM.
I can get both of them two work on the examples they have shared on their WIKIs but they come with predefined environments and have little or no information on how to setup my own custom environment.
I would be really thankful if anyone could point me towards a tutorial or just explain it to me on how can i setup a non-game environment?
I've been working on these libraries for some time and can share some of my experiments.
Let us first consider as an example of custom environment a text environment, https://github.com/openai/gym/blob/master/gym/envs/toy_text/hotter_colder.py
For a custom environment, a couple of things should be defined.
Constructor__init__ method
Action space
Observation space (see https://github.com/openai/gym/tree/master/gym/spaces for all available gym spaces (it's a kind of data structure))
_seed method (not sure that it's mandatory)
_step method accepting action as a param and returning observation (state after action), reward (for transition to new observational state), done (boolean flag), and some optional additional info.
_reset method that implements logic of fresh start of episode.
Optionally, you can create a _render method with something like
def _render(self, mode='human', **kwargs):
outfile = StringIO() if mode == 'ansi' else sys.stdout
outfile.write('State: ' + repr(self.state) + ' Action: ' + repr(self.action_taken) + '\n')
return outfile
And also, for better code flexibility, you can define logic of your reward in _get_reward method and changes to observation space from taking action in _take_action method.
Related
I have a hierarchy of data that i would like to build using classes instead of hard coding it in. The structure is like so:
Unit (has name, abbreviation, subsystems[5 different types of subsystems])
Subsystem ( has type, block diagram(photo), ParameterModel[20 different sets of parameterModels])
ParameterModel (30 or so parameters that will have [parameter name, value, units, and model index])
I'm not sure how to do this using classes but what i have made kindof work so far is creating nested dictionaries.
{'Unit':{'Unit1':{'Subsystem':{'Generator':{Parameter:{'Name': param1, 'Value':1, 'Units': 'seconds'}
like this but with 10-15 units and 5-6 subsystems and 30 or so parameters per subsystem. I know using dictionaries is not the best way to go about it but i cannot figure out the class sharing structure or where to start on building the class structure.
I want to be able to create, read, update and delete, parameters in a tkinter gui that i have built as well as export/import these system parameters and do calculations on them. I can handle the calculations and the import export but i need to create classes that will build out this structure and be able to reference each individual unit/subsystem/parameter/value/etc
I know thats alot but any advice? ive been looking into the factory and abstract factory patterns in hope to try and figure out how to create the code structure but to no avail. I have experience with matlab, visual basic, c++, and various arduio projects so i know most basic programming but this inheritance class structure is something i cannot figure out how to do in an abstract way without hardcoding each parameter with giant names like Unit1_Generator_parameterName_parameter = ____ and i really dont want to do that.
Thanks,
-A
EDIT: Here is one way I've done the implementation using a dictionary but i would like to do this using a class that can take a list and make a bunch of empty attributes and have those be editable/callable generally like setParamValue(unit, susystem, param) where i can pass the unit the subsystem and then the parameter such as 'Td' and then be able to change the value of the key,value pair within this hierarchy.
def create_keys(list):
dict = {key: None for key in list}
return dict
unit_list = ['FL','ES','NN','SF','CC','HD','ND','TH'] #unit abbreviation
sub_list = ['Gen','Gov','Exc','PSS','Rel','BlkD']
params_GENROU = ["T'do","T''do","T'qo","T''qo",'H','D','Xd','Xq',"Xd'","Xq'","X''d=X''q",'Xl','S(1.0)','S(1.2)','Ra'] #parameter names
dict = create_keys(unit_list)
for key in dict:
dict[key] = create_keys(sub_list)
dict[key]['Gen'] = create_keys(params_GENROU)
and inside each dict[unit][Gen][ParamNames] there should be a dict containing Value, units(seconds,degrees,etc), description and CON(#basically in index for another program we use)
I am working on Verigy 93K test program and I have a logic that I would like to know if there's an equivalent code in Origen.
I am working on Verigy 93K test program and I have this logic (IF condition) that I need to insert in my flow.
Basically, I have a variable called 'INSERTION' and this will have different values like 'GCORR', 'VCORR' and others.
I would like to know if there's an equivalent code like this in Origen.
I attached a snapshot, hope that it can help clarify my question more.
In this logic, I would like to check the INSERTION value and if the value is not equal to GCORR or VCORR, the logic should pass, else, fail.
Here is the screenshot:
This pull-request adds an official API for this.
This example would be implemented as:
whenever_any ne(:INSERTION, 'GCORR'), ne(:INSERTION, 'VCORR') do
# Your tests in here
end
That would produce something logically equivalent and which can be re-targeted to other platforms.
If you don't care about that and want to produce exactly as you have it in the above example, then this should work too (where the OR is hard-coded for V93K syntax):
whenever ne(:INSERTION, 'GCORR|VCORR') do
# Your tests in here
end
Here is the preliminary documentation of this feature from the above PR - https://github.com/Origen-SDK/origen_testers/blob/66345c9422d9fa6b2577af20110259e45c2bdd26/templates/origen_guides/program/flowapi.md.erb#L71
I couldn't find api support on flow control or variable values beyond "if/unless_enable" support which can help check for 1 or zero. One way is to use render.
render 'if #INSERTION != "GCORR|VCORR" then'
render '{'
# your code for non-GCORR_VCORR flow
render "} \n else \n { \n } "
I am exploring convolution layer in keras from:
https://github.com/fchollet/keras/blob/master/keras/layers/convolutional.py#L233
everywhere i found following type of code lines:
#interfaces.legacy_conv1d_support
#interfaces.legacy_conv2d_support
what is working and role of these lines. I searched in google but not find answer anywhere. please explain.
These lines starting with # are called decorators in python. Check out this page to read a brief summary about them. The basic function of this decorators is, that they wrap the following function into another function which has some kind of "wrapper" functions, like preprocessing the arguments, changing the accessibility of the function etc.
Taking a look at the interfaces.py file you will see this:
legacy_conv1d_support = generate_legacy_interface(
allowed_positional_args=['filters', 'kernel_size'],
conversions=[('nb_filter', 'filters'),
('filter_length', 'kernel_size'),
('subsample_length', 'strides'),
('border_mode', 'padding'),
('init', 'kernel_initializer'),
('W_regularizer', 'kernel_regularizer'),
('b_regularizer', 'bias_regularizer'),
('W_constraint', 'kernel_constraint'),
('b_constraint', 'bias_constraint'),
('bias', 'use_bias')],
preprocessor=conv1d_args_preprocessor)
So, the use of this function is basicly to rename parameters. Why is this? The keras API changed the names of some arguments of some functions (like W_regularizer -> kernel_regularizer). To allow users to be able to run old code, they added this decorator, which will just replace the names of old arguments with the corresponding new parameter name before calling the real function. This allows you to run "old" keras 1 code, even though you have installed keras 2.
Tl;dr: These lines are just used to for compatibility reasons. As this are just internal aspects of keras there is nothing you have to worry about or to take care of.
I am able to create the lda model and save it. Now I am trying load the model, and pass a new document
lda = LdaModel.load('..\\models\\lda_v0.1.model')
doc_lda = lda[new_doc_term_matrix]
print(doc_lda )
On printing the doc_lda I am getting the object. <gensim.interfaces.TransformedCorpus object at 0x000000F82E4BB630>
However I want to get the topic words associated with it. What is the method I have to use. I was referring to this.
Not sure if this is still relevant, but have you tried get_document_topics()? Though I assume that would only work if you've updated your LDA model using update().
I don't think there is anything wrong with your code - the "Usage example" from the documentation link you posted uses doc2bow which returns a sparse vector - I don't know what new_doc_term_matrix consists of, but I'll assume it worked fine.
You might want to look at this stackoverflow question: you want to print an "object" - that isn't printable, the data you want is somewhere in the object, and that in itself is printable.
Alternatively, you can also use your IDE's capabilities - the Variable explorer in Spyder, for example - to click yourself into the objects and get the info you need.
For more info on similarity analysis with gensim, see this tutorial.
Use this
doc_lda.print_topics(-1)
I always have difficulty in finding all available alternative ways to produce a specific graph, either one that I have already decided to use (looking for different variations) or one that I have not yet thought of.
The R Graphical Manual site provides a complete list of samples of R's graphics functions, however it's easier for me to search providing a package name (how else -for example- can I get a resultset including superbarplot function, when I want to look for barplots?. Let alone that the superbarplot graph does not appear in the results even if I try searching for it's package: UsingR)
The R-SAS-SPSS Add-on Module Comparison - and especially on topic Graphics, Static in the table provided - gave me the idea that it would be nice to have a place where all relevant packages are listed by topic.
Do you have any idea about something like that?
If you're interested in learning about all the possible graphics you can make, you should learn about the grammar of graphics, and (my) implementation of it in R: ggplot2.
Your question, or the general pattern anyway, was clearly a primary use case for the design of the sos package.
sos actually goes one step further that your question requires by identifying particular functions with packages; in addition, it ranks the results by relevance (by default, you can change the default behavior via the "sortby" parameter, e.g., sortby="Date")
Here's how it works:
most of this package's functionality is exposed via the "findFn" command
for instance, if you want a list of all functions and the parent package related to scatter plots:
findFn("scatter plot", maxPages=2, sortby="TotalScore")
This returns a dataframe formatted as HTML table and delivered in your default browser (if you don't want it to pop-up immediately, then just bind the function call to a variable and then call the variable when you're ready)
The right-most column of the dataframe/HTML page is "Description and Link". Clicking an entry in that column opens another tab in your browser (according to the user-set preferences set in your browser) with the complete R help page for that function.
The results from the function call above show, for instance, that the functions for plotting data in a 'scatter plot' format are found in the following packages:
ade4 (function: scatter)
IDPmisc (functions: ipairs, iplots)
GGally (function: ggally_points)
PerformanceAnalytics (function:
chart.Scatter)
mclust (function: clPairs)
Another example:
findFn("boxplot", maxPages=2, sortby="TotalScore")
identifies these (among others) packages/functions for plotting boxplots:
sfsmisc (function: boxplot.matrix)
aplpack (function: boxplot2D)
NADA (function: boxplot-methods)
StatDA (function: rg.boxplot)
plotrix (function: gap.boxplot)
gplots (function: boxplot.n)
multcompView (function:
multcompBoxplot)
oligo (function: boxplot)
Have you seen the R Graph Gallery ?
Other than that, you may have to index all the source code of CRAN packages to search efficiently...
these are good memory-joggers. I second the ggplot2 recommend, also recommend looking thru CRAN views:
http://cran.r-project.org/web/views/
http://cran.fhcrc.org/web/views/Graphics.html
(this mirror seems faster in west coast US)
http://dataspora.com/archive/2009/seminar/Survey_of_R_Graphics_by_Driscoll_Dataspora_Jun2009.pdf
http://zoonek2.free.fr/UNIX/48_R/04.html
(possibly world's longest webpage)
http://www.stat.auckland.ac.nz/~ihaka/120/lectures.html
Ihaka's lectures notes