I'm trying to get some insights into reinforcement learning while using openAI gym as a learning environment. I do this by reading the book Hands-on reinforcement learning with Python. In this book, some code is provided. Often, the code doesn't work, because I have to unwrap it first, as shown in: openai gym env.P, AttributeError 'TimeLimit' object has no attribute 'P'
However, I personally am still interested in the WHY of this unwrapping. Why do you need to unwrap? What does this do exactly? And why isn't it coded like that in the book? Is it outdated software as Giuliov assumed?
Thanks in advance.
Open AI Gym offers many different environments. Each of them with their own set of parameters and methods. Nevertheless they generally are wrapped by a single Class (like an interface on real OOPLs) called Env. This class exposes the common most essential methods of any environment, like step, reset and seed. Having this “interface” class is great, because it allows your code to be environment agnostic. It is also makes things easier if you want to test a single agent on different environments.
However, if you want to access the behind-the.scenes dynamics of a specific environment, then you use the unwrapped property.
Related
I am using PyTorch to carry out vision tasks, but would like to use some of what fast.ai provides since it has a lot of useful functionality. I'd prefer to work mostly in PyTorch since it's easier for me to understand what's going on, it's easier for me to find information on it online, and I want to maintain flexibility.
In https://docs.fast.ai/migrating_pytorch it's written that after I use the following imports: from fastai.vision.all import * and from migrating_pytorch import *, I should be able to start "Incrementally adding fastai goodness to your PyTorch models", which sounds great.
But when I run the second import I get ModuleNotFoundError: No module named 'migrating_pytorch'. Searching in https://github.com/fastai/fastai I also don't find any code mention of migrating_pytorch.py, nor did I manage to find something online.
(I'm using fast.ai version 2.3.1)
I'd like to know if this is indeed the way to go, and if so how to get it working. Or if there's a better way then how I should use that approach instead.
As an example, it would be nice if I could use the EarlyStoppingCallback, SaveModelCallback, and add some metrics from fast.ai instead of writing them myself, while still having everything in mostly "native" PyTorch.
Preferably the solution isn't specific to vision only, but that's my current need.
migrating_pytorch is an example script. It's in the fast.ai repo at: https://github.com/fastai/fastai/blob/master/nbs/examples/migrating_pytorch.py
The notebook that shows how to use it is at: https://github.com/fastai/fastai/blob/827e7cc0fad2db06c40df393c9569309377efac0/nbs/examples/migrating_pytorch.ipynb
For the callback example. Your training code would end up looking something like:
cbs = [EarlyStoppingCallback(), SaveModelCallback()]
learner = Learner(dls, simple_cnn(), loss_func=F.cross_entropy, cbs=cbs)
learner.fit(1)
Those two callbacks probably need some arguments, e.g. save path, etc.
I'm using Alloy to model a system. I would like to check the implemented system matches the Alloy model by comparing log traces from a concrete execution of the actual system with the model.
The way I see this working is:
Add logs to the implemented system at points that correspond to the high level concepts modelled in Alloy, such as "Receptionist checks in guest G1"
Pre-process these into a form understood by Alloy
Give this to Alloy (or some other tool) and say 'Does this model admit this trace?' (this question)
This would be run over the operational logs of the system (or maybe subsets if performance is a problem) and continuously validate that the system was operating 'to spec'.
Is that possible / reasonable?
Possible yes.
Reasonable I'm not quite sure.
To me, Alloy shine at finding unknown unknowns, i.e. pitfalls, in your specifications.
Once the specification is fool-proofed using Alloy analysis, I don't see the point of encumbering your program with unnecessary translations and analysis steps. It's not only error prone, but you might also find yourself limited with the scalability of the analyzer if the traces you want to validate are substantial...
But again, it's doable. So if you want it, sure, do it ... :-)
I'm using Alloy to model a system. I would like to check the implemented system matches the Alloy model by comparing log traces from a concrete execution of the actual system with the model.
Yes, I think that is a bit of work but it should be doable. I would be very interested in getting this to work. I have been thinking about this for a long time.
Loïc argues correctly that Alloy shines in finding solutions but to keep this manageable, Alloy must keep the scope small. Although this is true, Alloy is also a specification language. The timing issue is only in finding a solution. However, the problem you sketch is different, you already have the solution in the log. Each event specifies a transition in the state.
If you're familiar with the Alloy Evaluator then you should be aware that once you have a solution, you can run any Alloy code on that instance. Inside Alloy, there is a full set of classes to simulate an instance and run Alloy code against it.
So I think you can start with an initial instance and use your log event to create a secondary instance and then use Alloy to verify this is a valid transition. This will be very fast and I do not see why this could not handle a very large number of objects. Surely thousands and with a bit of caching wizardry millions.
We are currently working hard on Alloy 6, which will be integrate Electrum, where we will have full temporal logic that will make the rules easier to express.
I've been looking for a customer for a long time that would like to develop the necessary code to bridge Alloy & the trenches. If this works as I think it can work, it would be very interesting for the software industry.
I am looking at a two step approach for a optimization problem. My first step is using a MILP formulation of the problem and the second step involves using the solution from the first step as an initial solution but now with a MIQP formulation. I have been able to apply this concept in MATLAB using CPLEX. However, I am now trying the same using CVXPY with CPLEX as the solver. Now I know about the warm_start option but this does not work with the CPLEX solver. I am able to set CPLEX parameters but I am not sure how to initialize my solution. I am thinking of setting the ADVANCE START SWITCH parameter for CPLEX to 1, but now I need to set the initial solution. According to this page: http://www-eio.upc.es/lceio/manuals/cplex-11/html/usrcplex/solveMIP17.html, I need to use the method setVectors in a Concert Technology application, or by using CPXcopymipstart in a Callable Library application to set the initial solution. I am unsure of how to use this along with CVXPY.
The functionality you are looking for does not currently exist in CVXPY. CVXPY is a generic modeling layer that wraps around several solvers and it does not expose the CPLEX-specific CPXreadcopymipstarts nor CPXaddmipstarts functionality.
The fact that setting the value property of variables and using the warm_start option, as suggested in this answer, doesn't work, is a CVXPY issue. It looks like there is an open github issue for this here. In the future, this will likely be the intended solution to your general question.
For now, you'll have to use one of the CPLEX APIs directly. As you mentioned in the comments of this related stackoverflow question, you do not like the idea of using the lower-level CPLEX Python API. That leaves you with docplex as a viable option.
I've just found that PyTorch docs expose something that is called Torch Scripts. However, I do not know:
When they should be used?
How they should be used?
What are their benefits?
Torch Script is one of two modes of using the PyTorch just in time compiler, the other being tracing. The benefits are explained in the linked documentation:
Torch Script is a way to create serializable and optimizable models from PyTorch code. Any code written in Torch Script can be saved from your Python process and loaded in a process where there is no Python dependency.
The above quote is actually true both of scripting and tracing. So
You gain the ability to serialize your models and later run them outside of Python, via LibTorch, a C++ native module. This allows you to embed your DL models in various production environments like mobile or IoT. There is an official guide on exporting models to C++ here.
PyTorch can compile your jit-able modules rather than running them as an interpreter, allowing for various optimizations and improving performance, both during training and inference. This is equally helpful for development and production.
Regarding Torch Script specifically, in comparison to tracing, it is a subset of Python, specified in detail here, which, when adhered to, can be compiled by PyTorch. It is more laborious to write Torch Script modules instead of tracing regular nn.Module subclasses, but it allows for some extra features over tracing, most notably flow control like if statements or for loops. Tracing treats such flow control as "constant" - in other words, if you have an if model.training clause in your module and trace it with training=True, it will always behave this way, even if you change the training variable to False later on.
To answer your first question, you need to use jit if you want to deploy your models outside Python and otherwise you should use jit if you want to gain some execution performance at the price of extra development effort (as not every model can be straightforwardly made compliant with jit). In particular, you should use Torch Script if your code cannot be jited with tracing alone because it relies on some features such as if statements. For maximum ergonomy, you probably want to mix the two on a case-by-case basis.
Finally, for how they should be used, please refer to all the documentation and tutorial links.
I have an Alloy model. The model is of some decision making logic in software I wrote. In that model I have a few predicates that create examples. The predicate creates instances that are expected behavior and outside expected behavior. I would love to take those examples as inputs to a unit test for my code.
Does anyone have an example of some software that interacts with Alloy to dump many examples generated to a single file? I would love to run a program, get a file with many instances in it, and then use that file as input to my test program.
This interests me because the examples and counter examples created are often not what I would think to do when hand writing my test inputs.
Thoughts?
You can export an instance in the File/Export To menu.
If you can work in Java, then maybe it is interesting to know we're setting up an open source repo on github: https://github.com/AlloyTools/
I think it is quite easy to link your code with this code and generate your test cases or provide them from proper files.
I am extremely interested in this kind of applications for Alloy so please keep us posted on https://groups.google.com/forum/#!forum/alloytools