I edited Keras .optimizer and .layers modules locally, but Colab uses its own Keras & TensorFlow libraries. Uploading then using the edited libs would be rather involved per pathing and package interactions, and an overkill for a few small edits.
The closest I've got to accessing a module is keras.optimizers.__file__, which gives a relative path I don't know what to do with: '/usr/local/lib/python3.6/dist-packages/keras/optimizers.py'
Can Colab libraries be edited? Permanently (not per-runtime)?
Colab now allows direct access to system files from the GUI itself. There one can view and edit all the installed libraries like one would have done on their pc itself.
Go to the Files icon in the left Sidebar. Go to the Up Folder. From there go to the path
usr/local/lib/python3.6/dist-packages
Here, find the package and make your edit.
Then restart the runtime, from Runtime/Restart Runtime option in the menu.
You could fork the libraries on GitHub, push your changes to a new branch and then do.
!pip install git+https://github.com/your-username/keras.git#new-branch
Or even a specific commit
!pip install git+https://github.com/your-username/keras.git#632560d91286
You will need to restart your runtime for the changes to work.
More details here.
Per-runtime solution
import keras.optimizers
with open('optimizers.txt','r') as writer_file:
contents_to_write = writer_file.read()
with open(keras.optimizers.__file__,'w') as file_to_overwrite:
file_to_overwrite.write(contents_to_write)
>>Restart runtime (do not 'Reset all runtimes')
To clarify, (1) save edited module of interest as a .txt, (2) overwrite Colab module with the saved module via .__file__, (3) 'Reset all runtimes' restores Colab modules - use if module breaks
Considering its simplicity, it's as good as a permanent fix. For possibly better scalability, see fizzybear's solution.
Related
This is the scenario:
I am using Python 3 (3.6 through 3.8 on Windows 10, using Pipenv and vanilla Python) to create an SQLite file with Spatial support and several triggers based on Spatial Indices.
Creating the database and adding records works just fine after loading Spatialite
conn.enable_load_extension(True)
conn.load_extension("mod_spatialite")
However, adding spatial links with code such as below
SELECT CreateSpatialIndex( 'nodes' , 'geometry' );"""
returns the following error
updateTableTriggers: "no such module: rtree"
I tried compiling the rtree extension following some recommendation from
Compiling SQLite RTREE in MSVC?
and using VS 2016 (16.4.2).
But I get all sorts of errors when trying to load that in SQL (Might not be compiling it properly, but I tried multiple things and nothing worked). My best attempt was a successful compilation using pretty much the instructions I referred to above, but when I attempted
p.conn.load_extension("libSqliteRtree.dll")
I got
sqlite3.OperationalError: The specified procedure could not be found.
I am really at loss here, as there seems to be very little discussion on this topic everywhere I looked. A few questions that come to mind are:
Are the specific compilation instructions/tricks/compiler versions that I should be using?
Is it even possible to compile and load rtree in Python 3 using the standard sqlite3 library?
Is this particular to Windows?
Are there alternative SQLite Python packages that could do the job (I didn't find any on PyPI)?
It is critical, however, that the solution works across different platforms.
I was just having the exact same problem, with Python 3.8 x64. I believe the problem was that the sqlite3.dll file inside my Python's installation DLLs folder had been compiled without RTREE enabled (https://sqlite.org/rtree.html).
To resolve this I visited the SQLite website, downloaded the .zip with the latest sqlite3.dll for Windows x64 (hoping RTREE would be enabled on that version, because I tried compiling it on my own and it didn't work), and swapped the old DLL in the DLLs folder with the newly downloaded DLL from the website. The RTREE error was gone! Detailed steps below:
Access https://sqlite.org/download.html and choose the "Precompiled Binaries for Windows" of your system. Mine was x64 because I was running Python x64. Download the .zip and unzip it.
Find your Python's installation folder (the folder which contains the python.exe you're running), and open the DLLs folder. You'll see there is a file there called sqlite3.dll. That's the one that comes with the Python installation.
Copy the sqlite3.dll from the unzipped folder in step 1 and paste into the DLLs folder, click Yes to substitute the file in the DLLs folder for the new one. That should solve the problem.
I'm trying out Freeling's API for python. The installation and test were ok, they provide a sample.py file that works perfectly (I've played around a little bit with it and it works).
So I was trying to use it on some other python code I have, in a different folder (I'm kind of guessing this is a path issue), but whenever I import freeling (like it shows on the sample.py):
import freeling
FREELINGDIR = "/usr/local";
DATA = FREELINGDIR+"/share/freeling/";
LANG="es";
freeling.util_init_locale("default");
I get this error:
ModuleNotFoundError: No module named 'freeling'.
The sample.py is located on the ~/Freeling-4.0/APIs/Python/ folder, while my other file is located in ~/project/, I dont know if that can be an issue.
Thank you!
A simple solution is to have a copy of freeling.py in the same directory as your code, since python will look there.
A better solution is to either paste it in one of the locations where it usually checks (like the lib folder in its install directory), or to tell it that the path where your file is should be scanned for a module.
You can check out this question to see how it can be done on Windows. You are basically just setting the PYTHONPATH environment variable, and there will only be minor differences in how to do so for other OSes. This page gives instructions that should work on Linux systems.
I like this answer since it adds the path at runtime in the script itself, doesn't make persistent changes, and is largely independent of the underlying OS (apart from the fact that you need to use the appropriate module path of course).
You need to set PYTHONPATH so python can find the modules if they are not in the same folder.
I'm working with some new libraries and I'm afraid that my script might show some troubles in the future with unexpected updates of packages. So I want to create a new environment but I don't want to manually install all the basic packages like numpy, pandas, etc. So, does it makes sense to create a new environment using conda which is the exact copy of my base environment or could it create some sort of conflict?
Copying using conda works, but if you used only virtualenv, you should manually build requirements.txt, create a new virtual environment, activate it, and then simply use pip install -r requirements.txt. Note the key word - manually.
For example if you needed requests, numpy and pandas, your requirements.txt would look like this:
requests==2.20.0
numpy==1.15.2
pandas==0.23.4
You could actually exclude numpy in this case, but you still keep it as you are using it and if you removed pandas you'd still need it. I build it by installing a new package and then using pip freeze to find the module I just installed and put it into the requirements.txt with current version. Of course if I ever get to the state where I will share it with someone, I replace == with >=, most of the time that's enough, if it conflicts, you need to check what the conflicting library requires, and adjust if possible, e.g. you put in latest numpy version as requirement, but older library needs specifically x.y.z version and your library is perfectly fine with that version too (ideal case).
Anyway, this is how much you have to keep around to preserve your virtual environment, also helps if you are going to distribute your project, as anyone can drop this file into a new folder with your source and create their own environment without any hassle.
Now, this is why you should build it manually:
$ pip freeze
certifi==2018.10.15
chardet==3.0.4
idna==2.7
numpy==1.15.2
pandas==0.23.4
python-dateutil==2.7.3
pytz==2018.5
requests==2.20.0
six==1.11.0
urllib3==1.24
virtualenv==16.0.0
six? pytz? What? Other libraries use them but we don't even know what they are for unless we look it up, and they shouldn't be listed as project dependencies, they will be installed if they depend on it.
This way you ensure that there won't be too many problems only in very rare cases where one library you are using needs a new version of another library but the other library wants an ancient version of the library of which the version is conflicting and in that case it's a big mess, but normally it doesn't happen.
I am trying to run the PumpingSystem Example in the openmodelica Fluid library using the nightly build 1.9.1+dev (r21018). Unfortunately the simulation crashes saying it failed to solve NLS at initialization.
I tried to modify the model either by creating a new one extending it (which only permits the modification of the parameters but not the structure, I understand that this is probably what extension means) or by copying the text view of the model to a new file, but then OMEdit crashes.
Will you please advise how I can create a copy that I can modify?
Thank you.
PS: I am running this on Linux, the Windows version seems to translate all libraries,creates an infinite amount of translation errors "expected package to have within ; but got ..." and then terminates with a translation error "C:/OpenModelica1.9.1Nightly/lib/omlibrary/Modelica 3.2.1/Blocks/Continuous/Internal/Filter/Utilities/normalizationFactor.mo:14:3-42:27] Error: An element with name normalizationResidue is already declared in this scope."
You can not modify the system libraries. You can only extend them.
Creating a copy of the model is currently not supported. You can follow this ticket https://trac.openmodelica.org/OpenModelica/ticket/2190
If you just copy paste the text, as you already did, you need to manually update the relative paths used in the model.
So far I've seen the answer for Python2 however it doesn't work on Python3, I want to be able to always get the latest changes in a module that lives in a package every time the code runs without reopening a new interpreter every time. Since modules seems to be loaded just once for performance purposes as specified in documentation,I would like to be able to force a load in the modules programatically right before starting my program. Thanks in advance...
Although I'm not a fan of answering my own questions, I think in this case totally worth to mention it since seems to be quiet useful, in order to reload a module that you previously modified without having to restart the whole interpreter, just programatically forcing the modules (contained within a package) to get the latest changes this is the way to go:
import com.your.package.YourModule as MyModuleInPackage
import imp
imp.reload(MyModuleInPackage)
Notice that trying to use imp.reload(com.your.package.YourModule) causes an error, so the way to go is by having an Alias for the fully quialified name of the module and use it in the reload function to work properly...
Hope this helps.
Regards!