importing ecoinvent 3.2 with brightway - brightway

I am having some trouble importing Ecoinvent 3.2 with Brightway2, I was following the example notebook:
from brightway2 import *
fp = 'D:\LCAdb\e_3.2_cutoff_lci\datasets'
projects.set_current("myproject")
bw2setup()
ei = SingleOutputEcospold2Importer(fp, "ecoinvent 3.2 cutoff")
I get a warning message that seems to block the import process
C:\Users\###\Anaconda3\envs\bw3\lib\site-packages\bw2data\project.py:157:
UserWarning:
Read only project
This project is being used by another process and no writes can be
made until:
1. You close the other program, or switch to a different project, and
2. You call projects.enable_writes and get the response True.
If you are **sure** that this warning is incorrect, call
`projects.enable_writes(force=True)` to enable writes.
warnings.warn(READ_ONLY_PROJECT)
if I run
projects.enable_writes(force=True)
I get another a persmission error
PermissionError Traceback (most recent call
last) in ()
----> 1 projects.enable_writes(force=True)
C:\Users\###\Anaconda3\envs\bw3\lib\site-packages\bw2data\project.py
in enable_writes(self, force)
234 """Enable writing for the current project."""
235 if force:
--> 236 os.remove(os.path.join(self.dir, "write-lock"))
237 self.read_only = not self._lock.acquire(timeout = 0.05)
238 if not self.read_only:
PermissionError: [WinError 32] El proceso no tiene acceso al archivo
porque está siendo utilizado por otro proceso:
'C:\Users\###\AppData\Local\pylca\Brightway3\myproject.4da39212894ad06eb7c95810f8a2a6b0\write-lock'
the winerror translated would be something like "the process does not have access to the file because the file is being used by other process"
I do not have other Brightway environments running at the same time and I have recently updated Brightway2 so I do not know where the problem may be. Any ideas?
thanks!
UPDATE1: I have installed brightway2 in a different computer and I have found the same warning message. Despite the message, the import seems to be correct. once the database is loaded and written, if I open the project again the database is still there.
In the previous laptop the process of importing seems to be too much for the machine (an ASUS S56CB with windows 10 and 6 GB RAM). After 40 min waiting for the result I usually despair and kill it. I will give a try reinstalling python...

See how do I resolve errors about read-only projects?.
The locking in Brightway2 is done with the fasteners library. If you have two python interpreters running, or don't exit a ipython notebook correctly (you should shutdown or restart a kernel before closing the process completely), there will be a lock file in the project directory, and you will get errors about read-only projects. By the way, just closing a ipython notebook is not enough to shut down the python kernel or release the lock.
It is difficult to diagnose why these errors could be occurring with what you have reported. Try the following:
1) See if you can import ecoinvent into a new project.
2) Try restarting your computer.
3) If you can't get anything to work, you can delete the file C:\Users\###\AppData\Local\pylca\Brightway3\myproject.4da39212894ad06eb7c95810f8a2a6b0\write-lock in the command line or in an explorer window.
4) Do what #Mpa suggest in the next answer, and set projects.read_only = False.

I have run into this in the past, surely because of the reasons #Chris evoked.
You can use projects.read_only = False to force-write data. Please make sure that this is really what you want to do. You will not want to do this, for example, if you are accessing the same project through two different kernels that may try to write data at the same time.

Related

Setting up Visual Studio Code to run models from Hugging Face

I am trying to import models from hugging face and use them in Visual Studio Code.
I installed transformers, tensorflow, and torch.
I have tried looking at multiple tutorials online but have found nothing.
I am trying to run the following code:
from transformers import pipeline
classifier = pipeline('sentiment-analysis')
result = classifier("I hate it when I'm sitting under a tree and an apple hits my head.")
print(result)
However, I get the following error:
No model was supplied, defaulted to distilbert-base-uncased-finetuned-sst-2-english and revision af0f99b (https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english).
Using a pipeline without specifying a model name and revision in production is not recommended.
Traceback (most recent call last):
File "c:\Users\user\Desktop\Artificial Intelligence\transformers\Workshops\workshop_3.py", line 4, in <module>
classifier = pipeline('sentiment-analysis')
File "C:\Users\user\Desktop\Artificial Intelligence\transformers\src\transformers\pipelines\__init__.py", line 702, in pipeline
framework, model = infer_framework_load_model(
File "C:\Users\user\Desktop\Artificial Intelligence\transformers\src\transformers\pipelines\base.py", line 266, in infer_framework_load_model
raise ValueError(f"Could not load model {model} with any of the following classes: {class_tuple}.")
ValueError: Could not load model distilbert-base-uncased-finetuned-sst-2-english with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForSequenceClassification'>, <class 'transformers.models.auto.modeling_tf_auto.TFAutoModelForSequenceClassification'>, <class 'transformers.models.distilbert.modeling_distilbert.DistilBertForSequenceClassification'>, <class 'transformers.models.distilbert.modeling_tf_distilbert.TFDistilBertForSequenceClassification'>).
I have already searched online for ways to set up transformers to use in Visual Studio Code but nothing is helping.
Do you know how to fix this error, or if someone knows how to successfully use models from Hugging Face into my code, it would be appreciated?
This question is a little less about Hugging Face itself and likely more about installation and the installation steps you took (and potentially your program's access to the cache file where the models are automatically downloaded to.).
From what I am seeing either:
1/ your program is unable to access the model
2/ your program is throwing specific value errors in a bit of an edge case
If 1/ Take a look here: [https://huggingface.co/docs/transformers/installation#cache-setup][1]
Notice that it the docs walks through where the pre-trained models are downloaded. Check that it was downloaded here: C:\Users\username\.cache\huggingface\hub (of course with your own username on your computer instead. Check in the cache location to make sure it was downloaded? (You can check in the cache locations mentioned.)
Second, if for some reason, there is an issue with downloading, you can try downloading manually and doing it via offline mode (this is more to get it up and running): https://huggingface.co/docs/transformers/installation#offline-mode
Third, if it is downloaded, do you have the right permissions to access the .cache? (Try running your program (if it is a program that you trust) on Windows Terminal as an administrator.). Various ways - find one that you're comfortable with, here are a couple hints from Stackoverflow/StackExchange: Opening up Windows Terminal with elevated privileges, from within Windows Terminal or this: https://superuser.com/questions/1560049/open-windows-terminal-as-admin-with-winr
If 2/ I have seen people bring up very specific issues on not finding specific values (not the same as yours but similar) and the issue was solved by installing PyTorch because some models only exist as PyTorch models. You can see the full response from #YokoHono here: Transformers model from Hugging-Face throws error that specific classes couldn t be loaded

How can I use Brightway2 with US LCI database?

Short version:
I am trying to upload US LCI database to Brightway2 and I am failing miserably. Has anyone succeeded? If so, could you share it with me? :D
Long version:
I am following the notebook IO - Importing the US LCI database notebook and I am having a lot of problems. I am aware that, as the notebook indicates, it is a work in progress. Anyhow, I wanted to give it a try:
I tried uploading every ecospold version database found here, following the method from the notebook. The only one that gave me a similar results was version FY20.Q3.02. However, right off the bat I get the following differences/errors:
Same as the notebook, I get this error: Couldn't apply strategy link_technosphere_by_activity_hash: Object in source database can't be uniquely linked to target database. And two activities that are linked. When I follow the instructions of ignoring these datasets, it throws me that error over and over again.
Trying to move on with the tutorial, I get more errors and at the end I end up with all exchanges unlinked:
633 datasets
37513 exchanges
37505 unlinked exchanges
Finally, after running the code in line [15]:
import functools
f = functools.partial(link_iterable_by_fields,
other=Database(config.biosphere),
kind='biosphere'
)
sp.apply_strategy(f)
sp.statistics(f)
I end up with:
0 datasets
0 exchanges
0 unlinked exchanges
Which is hilarious and sad at the same time. Since I am new with Python and BW, my troubleshooting is clumpsy and probably erroneous (I promise I googled a lot and went through the code). And concluded I am failing and it is time to ask questions:
Has anybody succeeded uploading the US LCI database to Brightway2?
If so, how? Which file did you use?
Thank you!!!!
This is an excellent question. I have added text to the offending notebook to note that it is obsolete.
In general, I think trying to import the ecospold files is a fools errand, as though they are labeled ecospold2, they are actually ecospold1 (which is a totally different format):
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ecoSpold xmlns="http://www.EcoInvent.org/EcoSpold01">
The most recent export also raises an error when I try the ecospold1 importer:
AttributeError: no such child: {http://www.EcoInvent.org/EcoSpold01}modellingAndValidation
This is a required attribute in ecospold1.
I think the best way forward would be to consume the JSON-LD directly. Note that it is important not to run bw2setup(), as you would also want to use their list of elementary flows and LCIA methods. Currently the experimental JSON-LD importer fails because the provided datasets need allocation, but don't provide a set of consistent allocation methods. When I use the git checkout of bw2io and do the following:
uslci = JSONLDImporter(
"/Users/cmutel/Downloads/National_Renewable_Energy_Laboratory-USLCI_Database/",
"US LCI",
preferred_allocation="CAUSAL_ALLOCATION"
)
uslci.apply_strategies()
I get the following error:
UnallocatableDataset: We currently only support exchange-specific CAUSAL_ALLOCATION
This is fixable, but someone would need to step through this and fix the allocation procedure, and I don't have the time to do that now.

Import of Ecoinvent 2.2 and Ecoinvent 3.x fail with Brightway

The import of Ecoinvent 2.2 and 3.x does not happen. I do not understand what is the issue here.
I have downloaded both 2 and 3 versions from Ecoinvent.org but both the Ecospold1 and 2 importers show me the same (or the lack of) results.
`from brightway2 import *
ei33cutoff = SingleOutputEcospold2Importer(
r"C:\Users\HS254581\Documents\test\ecoinvent33cutoff\datasets",
"ecoinvent 3.3 cutoff"
)
ei33cutoff.apply_strategies()
ei33cutoff.statistics()`
All I get is
"Extracting XML data from 13831 datasets"
WITHOUT any error message. I have kept it running for hours sometimes, ultimately I have to quit.
I can however import Forwast database, do some LCA calculations here and some CSV files from Simapro.
I read most of the questions answered here and on github, but cannot find a solution to this problem.
Send help!
P.S. I am using Python 3.7 with Spyder. Here I need to activate bw2 environment in Anaconda prompt and set Python interpreter path to default. Have installed kernels too. This is the only way it seems to work. Also updated conda and brightway packages today just in case.

Can not make DDE connection using python. win32ui appears not working

I am new to Python (version 2.7). I have been using for a long time some excel workbooks that uses DDE function to capture "real-time market data" - that is served by one Market Data Provider ("Matriks", if matters).
In a bit to simplify my overall process flow, I decided to bypass this excel workbook thing for data capture, and instead, I decided to use python codes to perform the same task.
The problem is that,
1) I get "error: The server could not be created" when I run the code (provided below) using Pythonwin.
2) and I get "ImportError: This must be an MFC application - try 'import win32ui' first" when I run this very simple code using IDLE.
(and of course, I downloaded and run win32ui.)
Any help is appreciated.
# DDE code
# excel equivalent of this code is: =MTX|DATA!EURUSD.SON
import win32ui
import dde
s=dde.CreateServer()
s.Create("MTX")
c=dde.CreateConversation(s)
c.ConnectTo("DATA","SON")
c.Connected()
c.Request("EURUSD")
# returns
# Traceback (most recent call last):
# File "C:\IQTrader\_script\_obj\DDEClient.py", line 12, in <module>
# import dde
# ImportError: This must be an MFC application - try 'import win32ui' first
The issue is resolved. The very basic reason for the error is the pywin module.
For those facing with the same problem:
1.Delete pythonwin and pywin32_system32 folderes entirely (presumably under C:\Python27\Lib\site-packages)
2.Check your pywin32 version; it should be 214 (not 218) for those using v2.7
3.Download pywin32-214.win32-py2.7 from appropriate resources (one is this: http://sourceforge.net/projects/pywin32/files/pywin32/Build%20214/ )
4.Everything should be fine.
I had a similar problem. I had to:
uninstall Python 2.7 64-bit
install Python 2.7 32-bit
use the pywin32 version 214
Run cmd.exe as administrator so that the c.ConnectTo() connects successfully
However, I am still having the weird problem that I must have Excel open at the same time with a cell holding the value =MTX|DATA!EURUSD.SON for the python script to return a valid quote. Otherwise, I just get "N/A" as the return value.
#Aykut did you run into this issue as well?

Proper way to cleanup dynamic engines and can they be loaded twice?

I am having problems loading Engine PKCS #11 as a dynamic engine using python and M2Crypto. I am trying to access an Aladdin USB eToken.
Here are the important steps from my python code:
dynamic = Engine.load_dynamic_engine("pkcs11", "/usr/local/ssl/lib/engines/engine_pkcs11.so")
pkcs11 = Engine.Engine("pkcs11")
pkcs11.ctrl_cmd_string("MODULE_PATH", "/usr/lib/libeTPkcs11.so")
pkcs11.engine_init_custom() # initialize engine with custom M2Crypto patch
# next few steps which I deleted pass password and grab key & cert off token
Engine.cleanup()
This works fine the first time this method gets run. The second time, it fails when loading the dynamic engine (see error below).
Traceback (most recent call last):
File "", line 1, in ?
File "/usr/local/lib/python2.4/site-packages/M2Crypto/Engine.py", line 98, in load_dynamic_engine
e.ctrl_cmd_string("LOAD", None)
File "/usr/local/lib/python2.4/site-packages/M2Crypto/Engine.py", line 38, in ctrl_cmd_string
raise EngineError(Err.get_error())
M2Crypto.Engine.EngineError: 4002:error:260B606D:engine routines:DYNAMIC_LOAD:init failed:eng_dyn.c:521:
Is it impossible to load engines twice in a python session? Am I missing some kind of engine cleanup/deletion? The OpenSSL docs talk about engine_finish() but I don't think M2Crypto offers that. Is there a method to tell if the engine is already loaded?
Thanks!
M2Crypto does have ENGINE_finish and ENGINE_free available in the svn trunk version. The Engine class has init, and finish methods, and when an instance gets deleted it will be free'd. Can you give that a try? If you see any issues there is still time to fix them for next release.
My python code displayed nicer than it is in the comment section. The pkcs11.finish() method causes a segmentation fault in M2Crypto revision 723.
dynamic = Engine.load_dynamic_engine("pkcs11", "/usr/local/ssl/lib/engines/engine_pkcs11.so")
pkcs11 = Engine.Engine("pkcs11")
pkcs11.ctrl_cmd_string("MODULE_PATH", "/usr/lib/libeTPkcs11.so")
pkcs11.init()
# next few steps which I deleted pass password and grab key & cert off token
pkcs11.finish()
Engine.cleanup()
Anyone have advice on whether I'm doing something wrong or if there is a problem with the M2Crypto code?

Resources