'location' filter on database.search filtering everything - brightway

I have imported the ecoinvent database as ei
The search function works quite well:
In[0] eidb.search("glass",
filter = {'name':'green',
'product':'packaging'
}
)
Excluding 296 filtered results
Out[0]:
['packaging glass production, green' (kilogram, RER w/o CH+DE, None),
'packaging glass production, green' (kilogram, DE, None),
'packaging glass production, green' (kilogram, RoW, None),
'packaging glass production, green' (kilogram, CH, None),
'packaging glass production, green, without cullet' (kilogram, GLO, None),
'market for packaging glass, green' (kilogram, GLO, None)]
This is exactly as one would hope.
However, filtering on 'location' does not work so well:
In[1] eidb.search("glass",
filter = {'location':'DE',
}
)
Excluding 304 filtered results
Out[1]: []
According to the above result, I should have at least two results.
'location' is definitely an accepted filter, and DE is definitely one of the locations (e.g. eidb.get('d2db85e14baf9e47bdbb824797420f08').get('location') returns DE).
I observe this anytime location is used as filter, e.g. eidb.search('*', filter = {'location':'CA-QC'}) returns an empty list.
Why?

I have no idea why this is occurring, but you can get the behaviour you are looking for by putting the location code in lowercase:
In [1]: db.search("glass", filter={"location": "de"})
Excluding 103 filtered results
Out[1]:
['glass tube plant' (unit, DE, ['glass', 'construction']),
'glass tube, borosilicate, at plant' (kilogram, DE, ['glass', 'construction']),
'packaging glass, white, at plant' (kilogram, DE, ['glass', 'packaging']),
'packaging glass, brown, at plant' (kilogram, DE, ['glass', 'packaging']),
'packaging glass, green, at plant' (kilogram, DE, ['glass', 'packaging']),
'solar collector glass tube, with silver mirror, at plant' (kilogram, DE, ['glass', 'construction']),
'photovoltaic laminate, CdTe, at plant' (square meter, DE, ['photovoltaic', 'production of components'])]
Please file this as a bug for bw2data.

Probably you have already noticed, but for the case of Quebec, using just the final part would work (e.g. eidb.search('*', filter = {'location':'qc'})). I've checked and in ecoinvent there are no regions with the location code QC, so there is no risk of including activities from other regions.

The problem does not seems to be only with uppercase and punctuation characters
ei.search('photovoltaic laminate, CdTe',
filter={"location": "US"}
)
Excluding 7 filtered results
[]
P.S: weird but in this case filter={"location": "DE"} find the correct dataset also with uppercase

Related

AWARE method in ecoinvent through brightway2

i'm trying to extract all water inputs to several processes using brightway or ab, but i'm having troubles getting those values from the inventory_matrix. I need to get quantity of water consumed and origin of the water (surface, well, unspecified, tap water) along with its geography. mI'm using ecoinvent 3.7.1
One way i thought i could do that is create an impact assessment method with CFs for those compartments (each of those will be 1s), then apply it to the processes and analyse the elementary flows contributions.
I'm not sure though i can get the geography like this.
EDITED MOST OF THE QUESTION FOR CLARITY
Seems like i was trying to reinvent the wheel! My goal is to implement AWARE method and apply it to quite a few processes. the best result would be to use AWARE through activity-browser so i can use all of its functionalities which are very time-saving for me.
i just saw that there is brightway2-regional and bw2_aware that implement the aforementioned method.
So i'm now trying to install the packages in my brightway2 conda environment.
Managed to get bw2regional through conda but i cant manage to install bw2_aware if not through pip.
managed to install bw2_aware by using the wheel file and pip install --no-deps and then tweaking a line in the source code for fiona import, now i'm getting errors when running
bwr.bw2regionalsetup()
bw2_aware.import_aware()
SSLError: HTTPSConnectionPool(host='pandarus.brightwaylca.org', port=443): Max retries exceeded with url: / (Caused by SSLError(CertificateError("hostname 'pandarus.brightwaylca.org' doesn't match either of '*.parkingcrew.net', 'parkingcrew.net'",),))
Now i'm trying to understand if i can apply this to ecoinvent, and how. i'm really unsure if i can add the different geographies to the elementary flows in the database, so that i can correctly calculte aware in a lca.
I already saw that importing AWARE allows to choose it as a impact cathegory in activity-browser, though i cannot see the geographies in the CFS shown in the Charaterization Factors tab.
So i then tried to calculate an LCA with AB using AWARE method and 2 sample processes:
diesel, burned in agricultural machinery | diesel, burned in agricultural machinery | GLO | megajoule | ECOINVENT3.7.1
electricity, high voltage | market for electricity, high voltage | IT | kilowatt hour | ECOINVENT3.7.1
and i get this result (first is Agricultural, the other is non Agricultural):
diesel, burned in agricultural machinery | GLO
0.014757994762941706 0.00654978730728395
market for electricity, high voltage | IT 0.285207979534988 0.12657895834095712
I wonder if this is correct.
Your intuition is correct - constructing an LCIA method is an easy way to do this. This is because there are unspoken assumptions behind these flows - they will be mostly positive numbers, but some represent consumption, while others represent release.
Here is an example using Brightway 2.5, it would need to be adapted for version 2:
import bw2data as bd
import bw2calc as bc
import numpy as np
bd.projects.set_current("ecoinvent 3.7.1")
ecoinvent = bd.Database("ecoinvent 3.7.1")
beet = ecoinvent.search("beet")[1]
water_method = bd.Method(("Water", "raw"))
water_method.register()
water_method.write([(x, -1 if x['categories'][0] == 'natural resource' else 1)
for x in bd.Database("biosphere3")
if x['name'].startswith('Water')])
demand, data_objs, _ = bd.prepare_lca_inputs(demand={beet: 1}, method=("Water", "raw"))
lca = bc.LCA(demand=demand, data_objs=data_objs)
lca.lci()
lca.lcia()
coo = lca.characterized_inventory.tocoo()
results = sorted(zip(np.abs(coo.data), coo.data, coo.row, coo.col), reverse=True)
for a, b, c, d in results[:10]:
print("{:.6f}\t{}\n\t\t{}".format(
float(b),
bd.get_activity(lca.dicts.biosphere.reversed[c]),
bd.get_activity(lca.dicts.activity.reversed[d])
))
With the result:
0.009945 'Water' (cubic meter, None, ('water',))
'electricity production, hydro, run-of-river' (kilowatt hour, CH, None)
-0.009945 'Water, turbine use, unspecified natural origin' (cubic meter, None, ('natural resource', 'in water'))
'electricity production, hydro, run-of-river' (kilowatt hour, CH, None)
0.009514 'Water' (cubic meter, None, ('water',))
'electricity production, hydro, run-of-river' (kilowatt hour, RoW, None)
-0.009514 'Water, turbine use, unspecified natural origin' (cubic meter, None, ('natural resource', 'in water'))
'electricity production, hydro, run-of-river' (kilowatt hour, RoW, None)
0.007264 'Water' (cubic meter, None, ('water',))
'electricity production, hydro, run-of-river' (kilowatt hour, FR, None)
-0.007264 'Water, turbine use, unspecified natural origin' (cubic meter, None, ('natural resource', 'in water'))
'electricity production, hydro, run-of-river' (kilowatt hour, FR, None)
-0.003371 'Water, river' (cubic meter, None, ('natural resource', 'in water'))
'irrigation, sprinkler' (cubic meter, CH, None)
0.003069 'Water' (cubic meter, None, ('water', 'ground-'))
'sugar beet production' (kilogram, CH, None)
0.001935 'Water' (cubic meter, None, ('air',))
'sugar beet production' (kilogram, CH, None)
0.001440 'Water' (cubic meter, None, ('water',))
'electricity production, hydro, run-of-river' (kilowatt hour, CN-SC, None)
There isn't a way to use regionalized calculations in the activity browser, and the AWARE method (incorrectly) relies on an external web service which is not strictly necessary. So the short answer is that this isn't possible right now, but as you are the second person to ask about this in a week, we need to get it working ASAP.

How to structure (take out) JSON from a text string. (Python)

I have a text string/script which I took out from a webpage. I would like to clean/structure that text string/Script so that I can only get JSON out of it. But its very long that I lost finding beginning and ending of JSON from that text. Does anyone help me out or advice a online website which can help to find the beginning and ending of JSON from that text. Many Thanks
window.__NUXT__=function(e,l,a,t,r,s,i,o,n,d){return{layout:s,data:[{product:{active_gtin:"5711555000616",active_supplier:"0000009002",active_supplier_product_id:"000000000091052931-EA",brand:"Prosonic",description:"Prosonic 32\" TV med Android og Full-HD opløsning. Android styresystemet giver dig let adgang til Netflix, Viaplay og TV2 Play samt mange andre apps og med indbygget Chromecast kan du let caste indhold til TV'et.",display_list_price:l,display_sales_price:l,energy_class:"A+",energy_class_color_code:"lev_3",energy_label:i,erp_product_id:o,gallery_images:[i,"https://sg-dam.imgix.net/services/assets.img/id/13a13e85-efe7-48eb-bb6c-953abc94fb08/size/original","https://sg-dam.imgix.net/services/assets.img/id/e0c39be1-eb82-4652-88f4-992226390a3f/size/original","https://sg-dam.imgix.net/services/assets.img/id/9bc81449-64ba-44c0-b691-31b22bf5dc91/size/original"],hybris_code:n,id:n,image_primary:"https://sg-dam.imgix.net/services/assets.img/id/f8d59494-3da7-4cb7-9dd8-e8d16577e7c4/size/original",in_stock_stores_count:15,is_approved_for_sale:t,is_exposed:t,is_reservable:t,name:'Prosonic 32" 32and6021 LED tv',online_from:16000344e5,online_to:2534022108e5,primary_category_path:"/elektronik/tv",product_url:"/produkter/prosonic-32-32and6021-led-tv/100553115/",sales_price:e,show_discount_message:a,sku:o,specifications:'[{"features":[{"code":"text-TvMemory","label":"Tekst TV hukommelse","value":"1000"}],"label":"Tekst TV hukommelse"},{"features":[{"code":"tvFeatures","label":"TV funktioner","value":"Netflix"},{"code":"tvFeatures","label":"TV funktioner","value":"SmartTV"},{"code":"tvFeatures","label":"TV funktioner","value":"Wi-Fi indbygget"}],"label":"TV funktioner"},{"features":[{"code":"TV.tvApps","label":"TV Apps","value":"Amazon"},{"code":"TV.tvApps","label":"TV Apps","value":"Apple TV"},{"code":"TV.tvApps","label":"TV Apps","value":"Blockbuster"},{"code":"TV.tvApps","label":"TV Apps","value":"Boxer"},{"code":"TV.tvApps","label":"TV Apps","value":"Dplay"},{"code":"TV.tvApps","label":"TV Apps","value":"DR TV"},{"code":"TV.tvApps","label":"TV Apps","value":"Google Play Store"},{"code":"TV.tvApps","label":"TV Apps","value":"HBO Nordic"},{"code":"TV.tvApps","label":"TV Apps","value":"Min Bio"},{"code":"TV.tvApps","label":"TV Apps","value":"Netflix"},{"code":"TV.tvApps","label":"TV Apps","value":"Rakuten TV"},{"code":"TV.tvApps","label":"TV Apps","value":"SF Anytime"},{"code":"TV.tvApps","label":"TV Apps","value":"Skype"},{"code":"TV.tvApps","label":"TV Apps","value":"Spotify"},{"code":"TV.tvApps","label":"TV Apps","value":"TV2 play"},{"code":"TV.tvApps","label":"TV Apps","value":"Viaplay"},{"code":"TV.tvApps","label":"TV Apps","value":"YouSee"},{"code":"TV.tvApps","label":"TV Apps","value":"Youtube"}],"label":"TV Apps"},{"features":[{"code":"connectivity.videoConnectivity","label":"Video tilslutning","value":"composite"}],"label":"Video tilslutning"},{"features":[{"code":"screen.monitorLanguageList","label":"Skærmsprog","value":"Dansk"}],"label":"Skærmsprog"},{"features":[{"code":"builtInSpeakers.soundFunction","label":"Lydfunktioner","value":"Bluetooth"}],"label":"Lydfunktioner"},{"features":[{"code":"productionYear","label":"Produktionsår","value":"2.020"}],"label":"Produktionsår"},{"features":[{"code":"electronics.manufacturerNum","label":"Producentens Varenummer","value":"32AND6021"}],"label":"Producentens Varenummer"},{"features":[{"code":"TV.hdrLOV","label":"HDR","value":"HDR 10"}],"label":"HDR"},{"features":[{"code":"TV.isSleepTimerPresent","label":"Sleep timer","value":"Ja"}],"label":"Sleep timer"},{"features":[{"code":"isPVRFunctionPresent","label":"PVR funktion","value":"Ja"}],"label":"PVR funktion"},{"features":[{"code":"accessoriesIncluded","label":"Tilbehør inkluderet","value":"stand og remote"}],"label":"Tilbehør inkluderet"},{"features":[{"code":"screenTechnologyDesc","label":"Skærmteknologi","value":"LED"}],"label":"Skærmteknologi"},{"features":[{"code":"tvTunerList","label":"TV-tuners","value":"CI+"},{"code":"tvTunerList","label":"TV-tuners","value":"DVB-C"},{"code":"tvTunerList","label":"TV-tuners","value":"DVB-S"},{"code":"tvTunerList","label":"TV-tuners","value":"DVB-T2"},{"code":"tvTunerList","label":"TV-tuners","value":"MPEG4 tuner"}],"label":"TV-tuners"},{"features":[{"code":"TV.vesaStandardList","label":"Vægbeslag Vesa standard","value":"75x75"}],"label":"Vægbeslag Vesa standard"},{"features":[{"code":"connectivity.hdmiCount","label":"Antal HDMI","value":"3"}],"label":"Antal HDMI"},{"features":[{"code":"builtInSpeakers.speakerEffect","label":"Højtalereffekt","value":"12"}],"label":"Højtalereffekt"},{"features":[{"code":"usbCount","label":"Antal USB stik","value":"1"}],"label":"Antal USB stik"},{"features":[{"code":"TVResolution","label":"TV opløsning","value":"Full HD"}],"label":"TV opløsning"},{"features":[{"code":"picturePlayers.supportedImageFormats","label":"Understøttede Billed Formater","value":"JPG,BMP,PNG,GIF"}],"label":"Understøttede Billed Formater"},{"features":[{"code":"scartCount","label":"Antal scartstik","value":"0"}],"label":"Antal scartstik"},{"features":[{"code":"connectivity.usbcount2","label":"Antal USB 2.0 porte","value":"1"}],"label":"Antal USB 2.0 porte"},{"features":[{"code":"Color","label":"Produktfarve","value":"sort"}],"label":"Produktfarve"},{"features":[{"code":"TV.isWatchAndTimerFunctionOnOffPresent","label":"Ur og timerfunktion til\\/fra","value":"Ja"}],"label":"Ur og timerfunktion til\\/fra"},{"features":[{"code":"TV.isAutomaticChannelSearchAvailable","label":"Automatisk kanalsøgning","value":"Ja"}],"label":"Automatisk kanalsøgning"},{"features":[{"code":"screen.screenResolution","label":"Skærmopløsning","value":"Full-HD 1920 x 1080"}],"label":"Skærmopløsning"},{"features":[{"code":"TV.software","label":"TV software","value":"Android"}],"label":"TV software"},{"features":[{"code":"connectivity.connectivityDesc","label":"Andre tilslutningsmuligheder","value":"Composite, Audio in, VGA, optisk lyd ud,"}],"label":"Andre tilslutningsmuligheder"},{"features":[{"code":"TV.twinTuner","label":"Twin Tuner","value":"Nej"}],"label":"Twin Tuner"},{"features":[{"code":"picturePlayers.supportedVideoFileFormats","label":"Understøttede videofil formater","value":".MPG .MPEG.DAT.VOB.MKV.MP4 \\/ .M4A \\/ .M4V.MOV.FLV.3GP \\/ 3GPP.TS \\/ .M2TS.RMVB .RM.AVI.ASF .WMV.WEBM"}],"label":"Understøttede videofil formater"},{"features":[{"code":"isInternetBrowserPresent","label":"Internet browser","value":"Ja"}],"label":"Internet browser"},{"features":[{"code":"wirelessConnectivityOptionList","label":"Trådløse tilslutningsmuligheder","value":"Bluetooth"},{"code":"wirelessConnectivityOptionList","label":"Trådløse tilslutningsmuligheder","value":"Wi-Fi indbygget"}],"label":"Trådløse tilslutningsmuligheder"}]',step_product_id:"GR14425172",stock_count_online:2874,stock_count_status_online:"in_stock",stock_type:"NORMAL",summary:"Med Android og indbygget Chromecast",msg_sales_price_per_unit:l,package_display_sales_price:l,promotion_text:e,f_campaign_name:[]},loadingProduct:a}],error:e,state:{User:{UID:l,isLoggedIn:a,nickname:l,address:{firstName:l,lastName:l,address:l,postalCode:l,city:l,mobile:l,email:l,country:l},isDeliveryMethodSet:a,lastSeenProducts:[],wishlistProducts:[]},Tracking:{trackedOrders:[],activeRoute:e,oldRoute:e,cookieConsentGiven:a,initialRouteTracked:a},Search:{showDrawer:a,hideGlobalSearch:a,query:l,queryString:l,queries:[],brands:[],categories:[]},Products:{products:[]},ProductDialog:{showType:a,productId:e,quantity:e,error:e},plugins:{Cart:{checkoutErrorPlugin:{},productDialogPlugin:{}},TechnicalError:{technicalErrorPlugin:{}},Tracking:{gtmPlugin:{},gtmHandlers:{appInitializedHandler:{},bannerClickedHandler:{},bannerViewedHandler:{},checkoutStepChangedHandler:{},clickCollectCompletedHandler:{},cookieConsentGivenHandler:{},externalLinkClickedHandler:{},helpers:{},notFoundPageViewedHandler:{},orderCompletedHandler:{},plpProductsViewedHandler:{},productAddedHandler:{},productClickedHandler:{},productDetailViewedHandler:{},productQuantityChangeHandler:{},productRemovedHandler:{},recommendationsClickedHandler:{},recommendationsViewedHandler:{},routeChangedHandler:{},siteSearchHandler:{}}},User:{userPlugin:{}}},Payment:{paymentMethod:e,termsAccepted:a},OAuth:{accessToken:e,expiry:0,timestamp:e,trackingId:e},Navigation:{hierarchy:e,path:[],loading:a,lastFetchedTopNode:l},Layout:{eyebrow:{default:e},footer:{default:e},layout:s},InfoBar:{infoBars:[],infoBarMappers:{}},Delivery:{isFetchingPickups:a,deliveries:{},pickups:{},selectedDeliveries:{}},ClickCollect:{loading:a,showDrawer:a,baseMapLocation:e,stores:[],selectedStore:e,product:e,quantity:1,form:{name:l,email:l,countryDialCode:"45",phone:l,terms:a},reservation:e,error:a,filters:{inStockOnly:t}},Checkout:{panelState:{userInfo:{},delivery:{},payment:{mustVisit:t},store:{}},desiredPanel:"auto",panelValidators:{}},Cart:{data:{id:l,lineItems:[],totalLineItemsQuantity:0,totalSalesPrice:r,totalShippingSalesPrice:r,employeeNumber:e,loyaltyNumber:e,deliveries:[],totalLineItemSalesPrice:r,totalLineItemListPrice:r,totalLineItemDiscount:r,totalShippingListPrice:r,totalShippingPriceDiscount:r,orderNumber:e,totalSalesPriceNumber:0,isActive:t,isAllLineItemsValid:t,shippingAddress:d,billingAddress:d,hash:l,discountCodes:[],source:"USER_DEVICE"},loading:{},error:e,assistedSalesMode:a,assistedSalesStoreNumber:e},Breadcrumb:{categoryTree:{},productCategory:l,lookupBreadcrumbTasks:{},currentCategoryPage:[],helpers:{}}},serverRendered:t}}(null,"",!1,!0,"0,00","default","https://sg-dam.imgix.net/services/assets.img/id/87a045c1-0923-4575-81ce-fd9b7c3bfbf6/size/original","91052931-EA","100553115",void 0)
You can use a RegEx to get the Jsons from your string.
I have used this pattern: {(?:[^{}]*{[^{]*})*[^{}]*}
The above regex checks only the Json in one level deep.
Code:
import re
import json
input_data = """window.__NUXT__=funct ... A","100553115",void 0)"""
def json_validate(input_str):
founds = re.findall(r"{(?:[^{}]*{[^{]*})*[^{}]*}", input_str)
valid_jsons = []
for x in founds:
try:
valid_jsons.append(json.loads(x))
except json.JSONDecodeError:
continue
return valid_jsons
getting_jsons = json_validate(input_data)
for one_json in getting_jsons:
print(one_json)
print(len(getting_jsons))
It can find several (32) valid Jsons in your string:
>>> python3 test.py
{'features': [{'code': 'text-TvMemory', 'label': 'Tekst TV hukommelse', 'value': '1000'}], 'label': 'Tekst TV hukommelse'}
{'features': [{'code': 'tvFeatures', 'label': 'TV funktioner', 'value': 'Netflix'}, {'code': 'tvFeatures', 'label': 'TV funktioner', 'value': 'SmartTV'}, {'code': 'tvFeatures', 'label': 'TV funktioner', 'value': 'Wi-Fi indbygget'}], 'label': 'TV funktioner'}
{'features': [{'code': 'TV.tvApps', 'label': 'TV Apps', 'value': 'Amazon'}, {'code ...
I have found another solution which approaches the issue from totally different way: https://stackoverflow.com/a/54235803/11502612
I have tested the code from the above answer and I got the same output. It means the result is correct (probably).
Would it not be easier to do something like
import json
data = json.dumps(your_string)
Then iterate over it to find the values. Alternatively you can look for the value locations with
find("{")
Don't know if this is what your looking for but thought it may spark an idea / alternative view

How to resolve pandas length error for rows/columns

I have raised the SO Question here and blessed to have an answer from #Scott Boston.
However i am raising another question about an error ValueError: Columns must be same length as key as i am reading a text file and all the rows/columns are not of same length, i tried googling but did not get an answer as i don't want them to be skipped.
Error
b'Skipping line 2: expected 13 fields, saw 14\nSkipping line 5: expected 13 fields, saw 14\nSkipping line 6: expected 13 fields, saw 16\nSkipping line 7: expected 13 fields, saw 14\nSkipping line 8: expected 13 fields, saw 15\nSkipping line 9: expected 13 fields, saw 14\nSkipping line 20: expected 13 fields, saw 19\nSkipping line 21: expected 13 fields, saw 16\nSkipping line 23: expected 13 fields, saw 14\nSkipping line 24: expected 13 fields, saw 16\nSkipping line 27: expected 13 fields, saw 14\n'
My pandas dataframe generator
#!/usr/bin/python3
import pandas as pd
#
cvc_file = pd.read_csv('kids_cvc',header=None,error_bad_lines=False)
cvc_file[['cols', 0]] = cvc_file[0].str.split(':', expand=True) #Split first column on ':'
df = cvc_file.set_index('cols').transpose() #set_index and transpose
print(df)
Result
$ ./read_cvc.py
b'Skipping line 2: expected 13 fields, saw 14\nSkipping line 5: expected 13 fields, saw 14\nSkipping line 6: expected 13 fields, saw 16\nSkipping line 7: expected 13 fields, saw 14\nSkipping line 8: expected 13 fields, saw 15\nSkipping line 9: expected 13 fields, saw 14\nSkipping line 20: expected 13 fields, saw 19\nSkipping line 21: expected 13 fields, saw 16\nSkipping line 23: expected 13 fields, saw 14\nSkipping line 24: expected 13 fields, saw 16\nSkipping line 27: expected 13 fields, saw 14\n'
cols ab ad an ed eg et en eck ell it id ig im ish ob og ock ut ub ug um un ud uck ush
0 cab bad ban bed beg bet den beck bell bit bid big dim fish cob bog dock but cub bug bum bun bud buck gush
1 dab dad can fed keg get hen deck cell fit did dig him dish gob cog lock cut hub dug gum fun cud duck hush
2 gab had fan led leg jet men neck dell hit hid fig rim wish job dog rock gut nub hug hum gun dud luck lush
3 jab lad man red peg let pen peck jell kit kid gig brim swish lob fog sock hut rub jug mum nun mud muck mush
4 lab mad pan wed NaN met ten check sell lit lid jig grim NaN mob hog tock jut sub lug sum pun spud puck rush
5 nab pad ran bled NaN net then fleck tell pit rid pig skim NaN rob jog block nut tub mug chum run stud suck blush
File contents
$ cat kids_cvc
ab: cab, dab, gab, jab, lab, nab, tab, blab, crab, grab, scab, stab, slab
at: bat, cat, fat, hat, mat, pat, rat, sat, vat, brat, chat, flat, gnat, spat
ad: bad, dad, had, lad, mad, pad, sad, tad, glad
an: ban, can, fan, man, pan, ran, tan, van, clan, plan, scan, than
ag: bag, gag, hag, lag, nag, rag, sag, tag, wag, brag, drag, flag, snag, stag
ap: cap, gap, lap, map, nap, rap, sap, tap, yap, zap, chap, clap, flap, slap, snap, trap
am: bam, dam, ham, jam, ram, yam, clam, cram, scam, slam, spam, swam, tram, wham
ack: back, hack, jack, lack, pack, rack, sack, tack, black, crack, shack, snack, stack, quack, track
ash: bash, cash, dash, gash, hash, lash, mash, rash, sash, clash, crash, flash, slash, smash
ed: bed, fed, led, red, wed, bled, bred, fled, pled, sled, shed
eg: beg, keg, leg, peg
et: bet, get, jet, let, met, net, pet, set, vet, wet, yet, fret
en: den, hen, men, pen, ten, then, when
eck: beck, deck, neck, peck, check, fleck, speck, wreck
ell: bell, cell, dell, jell, sell, tell, well, yell, dwell, shell, smell, spell, swell
it: bit, fit, hit, kit, lit, pit, sit, wit, knit, quit, slit, spit
id: bid, did, hid, kid, lid, rid, skid, slid
ig: big, dig, fig, gig, jig, pig, rig, wig, zig, twig
im: dim, him, rim, brim, grim, skim, slim, swim, trim, whim
ip: dip, hip, lip, nip, rip, sip, tip, zip, chip, clip, drip, flip, grip, ship, skip, slip, snip, trip, whip
ick: kick, lick, nick, pick, sick, tick, wick, brick, chick, click, flick, quick, slick, stick, thick, trick
ish: fish, dish, wish, swish
in: bin, din, fin, pin, sin, tin, win, chin, grin, shin, skin, spin, thin, twin
ot: cot, dot, got, hot, jot, lot, not, pot, rot, tot, blot, knot, plot, shot, slot, spot
ob: cob, gob, job, lob, mob, rob, sob, blob, glob, knob, slob, snob
og: bog, cog, dog, fog, hog, jog, log, blog, clog, frog
op: cop, hop, mop, pop, top, chop, crop, drop, flop, glop, plop, shop, slop, stop
ock: dock, lock, rock, sock, tock, block, clock, flock, rock, shock, smock, stock
ut: but, cut, gut, hut, jut, nut, rut, shut
ub: cub, hub, nub, rub, sub, tub, grub, snub, stub
ug: bug, dug, hug, jug, lug, mug, pug, rug, tug, drug, plug, slug, snug
um: bum, gum, hum, mum, sum, chum, drum, glum, plum, scum, slum
un: bun, fun, gun, nun, pun, run, sun, spun, stun
ud: bud, cud, dud, mud, spud, stud, thud
uck: buck, duck, luck, muck, puck, suck, tuck, yuck, chuck, cluck, pluck, stuck, truck
ush: gush, hush, lush, mush, rush, blush, brush, crush, flush, slush
Note:
It's making the first row/column as a master one which has 13 values and skipping all the columns which are more than 13 columns.
I couldn't figure out a pandas way to extend the columns, but converting the rows to a dictionary made things easier.
ss = '''
ab: cab, dab, gab, jab, lab, nab, tab, blab, crab, grab, scab, stab, slab
at: bat, cat, fat, hat, mat, pat, rat, sat, vat, brat, chat, flat, gnat, spat
ad: bad, dad, had, lad, mad, pad, sad, tad, glad
.......
un: bun, fun, gun, nun, pun, run, sun, spun, stun
ud: bud, cud, dud, mud, spud, stud, thud
uck: buck, duck, luck, muck, puck, suck, tuck, yuck, chuck, cluck, pluck, stuck, truck
ush: gush, hush, lush, mush, rush, blush, brush, crush, flush, slush
'''.strip()
with open ('kids.cvc','w') as f: f.write(ss) # write data file
######################################
import pandas as pd
dd = {}
maxcnt=0
with open('kids.cvc') as f:
lines = f.readlines()
for line in lines:
line = line.strip() # remove \n
len1 = len(line) # words have leading space
line = line.replace(' ','')
cnt = len1 - len(line) # get word (space) count
if cnt > maxcnt: maxcnt = cnt # max word count
rec = line.split(':') # header : words
dd[rec[0]] = rec[1].split(',') # split words
for k in dd:
dd[k] = dd[k] + ['']*(maxcnt-len(dd[k])) # add extra values to match max column
df = pd.DataFrame(dd) # convert dictionary to dataframe
print(df.to_string(index=False))
Output
ab at ad an ag ap am ack ash ed eg et en eck ell it id ig im ip ick ish in ot ob og op ock ut ub ug um un ud uck ush
cab bat bad ban bag cap bam back bash bed beg bet den beck bell bit bid big dim dip kick fish bin cot cob bog cop dock but cub bug bum bun bud buck gush
dab cat dad can gag gap dam hack cash fed keg get hen deck cell fit did dig him hip lick dish din dot gob cog hop lock cut hub dug gum fun cud duck hush
gab fat had fan hag lap ham jack dash led leg jet men neck dell hit hid fig rim lip nick wish fin got job dog mop rock gut nub hug hum gun dud luck lush
jab hat lad man lag map jam lack gash red peg let pen peck jell kit kid gig brim nip pick swish pin hot lob fog pop sock hut rub jug mum nun mud muck mush
lab mat mad pan nag nap ram pack hash wed met ten check sell lit lid jig grim rip sick sin jot mob hog top tock jut sub lug sum pun spud puck rush
nab pat pad ran rag rap yam rack lash bled net then fleck tell pit rid pig skim sip tick tin lot rob jog chop block nut tub mug chum run stud suck blush
tab rat sad tan sag sap clam sack mash bred pet when speck well sit skid rig slim tip wick win not sob log crop clock rut grub pug drum sun thud tuck brush
blab sat tad van tag tap cram tack rash fled set wreck yell wit slid wig swim zip brick chin pot blob blog drop flock shut snub rug glum spun yuck crush
crab vat glad clan wag yap scam black sash pled vet dwell knit zig trim chip chick grin rot glob clog flop rock stub tug plum stun chuck flush
grab brat plan brag zap slam crack clash sled wet shell quit twig whim clip click shin tot knob frog glop shock drug scum cluck slush
scab chat scan drag chap spam shack crash shed yet smell slit drip flick skin blot slob plop smock plug slum pluck
stab flat than flag clap swam snack flash fret spell spit flip quick spin knot snob shop stock slug stuck
slab gnat snag flap tram stack slash swell grip slick thin plot slop snug truck
spat stag slap wham quack smash ship stick twin shot stop
snap track skip thick slot
trap slip trick spot

how to use the transverse_tagged_databases function in brightway2

I would like to know how to use the transverse_tagged_database method in brightway2. From the documentation is not entirely clear to me. Can we use for, example, to aggregate impacts by the isic code of the activities in the product system model?
The short answer is yes, aggregating impacts by ISIC code in your foreground product system model is exactly the kind of thing you can do using traverse_tagged_databases.
The traverse_tagged_databases function takes advantage of the fact that you can add arbitrary key:value pairs to activities in brightway to let you classify your the activities in your foreground model however you like.
For example, say your activities look like this:
('example_database', 'code_for_bread'):{
'name': 'Bread',
'code': 'code_for_bread',
'categories':[],
'exchanges':[...],
'location':'GLO'
'unit':'kg',
'database':'example_database',
'isic_code':'1071'
'isic_classifier':'Manufacture of bakery products'
},
You can tell traverse_tagged_databases to go through your database looking for a given key (tag), for example 'isic_code', or 'isic_classifier' and aggregate the impact based on these tags.
Say you were modelling a cheese sandwich, you could have the following ISIC codes in your model:
Sandwich: 1079 (Manufacture of other food products n.e.c.)
Bread: 1071 (Manufacture of bakery products)
Cheese: 1050 (Manufacture of dairy products)
Butter: 1050 (Manufacture of dairy products)
You can use traverse_tagged_databases to see the total impact of dairy (cheese and butter) vs bakery (bread).
You use it in a similar way to the LCA function, by specifying a functional unit as a dict and the method as a tuple, with an additional tag argument. Like this:
fu = {('example_database', 'code_for_sandwich'):1}
m = ('IPCC 2013', 'climate change', 'GWP 100a')
result, tree = traverse_tagged_databases(fu, m, 'isic_classifier')
The function returns two objects (designated result and tree in the line above)
For this analysis, your result will look something like this:
defaultdict(int,
{'Manufacture of other food products n.e.c.': 0,
'Manufacture of bakery products': 0.1875,
'Manufacture of dairy products': 0.55})
This is saying that dairy products in the foreground model have an aggregated impact of 0.55 kg CO2-eq, and bakery products have an aggregated impact of 0.1875 kg CO2-eq.
With a bit of post-processing you can turn this data into pie charts, stacked bar charts etc.
You also get a tree, which looks like this:
[{'activity': 'Sandwich' (kg, GLO, []),
'amount': 1,
'tag': 'Manufacture of other food products n.e.c.',
'impact': 0,
'biosphere': [],
'technosphere': [{'activity': 'Bread' (kg, GLO, []),
'amount': 0.75,
'tag': 'Manufacture of bakery products',
'impact': 0,
'biosphere': [{'amount': 0.1875,
'impact': 0.1875,
'tag': 'Manufacture of bakery products'}],
'technosphere': []},
{'activity': 'Butter' (kg, GLO, []),
'amount': 0.05,
'tag': 'Manufacture of dairy products',
'impact': 0,
'biosphere': [{'amount': 0.05,
'impact': 0.05,
'tag': 'Manufacture of dairy products'}],
'technosphere': []},
{'activity': 'Cheese' (kg, GLO, []),
'amount': 0.25,
'tag': 'Manufacture of dairy products',
'impact': 0,
'biosphere': [{'amount': 0.5,
'impact': 0.5,
'tag': 'Manufacture of dairy products'}],
'technosphere': []}]}]
This can look a bit difficult to parse at first, but is essentially a set of nested dictionaries, starting with the root activity (the functional unit = Sandwich), showing techosphere exchanges to other activities, and biosphere exchanges to emissions.
The tree here looks like this (with the amounts of each product in brackets)
Bread +----(0.75 kg)----------+
|
|
|
|
Cheese +----(0.20 kg)----------+------(1.00 kg)--------> Sandwich
|
|
|
|
Butter +----(0.05 kg)----------+
Again, with a bit of post-processing, you can turn this data into stuff like sankey diagrams, or the kind of impact tree diagram you get in SimaPro.

unable to get rid of all emojis

I need help removing emojis. I looked at some other stackoverflow questions and this is what I am de but for some reason my code doesn't get rid of all the emojis
d= {'alexveachfashion': 'Fashion Style * Haute Couture * Wearable Tech * VR\n👓👜⌚👠\nSoundCloud is Live #alexveach\n👇New YouTube Episodes ▶️👇', 'andrewvng': 'Family | Fitness | Friends | Gym | Food', 'runvi.official': 'Accurate measurement via SMART insoles & real-time AI coaching. Improve your technique & BOOST your performance with every run.\nSoon on Kickstarter!', 'triing': 'Augmented Jewellery™️ • Montreal. Canada.', 'gedeanekenshima': 'Prof na Etec Albert Einstein, Mestranda em Automação e Controle de Processos, Engenheira de Controle e Automação, Técnica em Automação Industrial.', 'jetyourdaddy': '', 'lavonne_sun': '☄️🔭 ✨\n°●°。Visual Narrative\nA creative heart with a poetic soul.\n————————————\nPARSONS —— Design & Technology', 'taysearch': 'All the World’s Information At Your Fingertips. (Literally) Est. 1991🇺🇸 🎀#PrincessofSearch 🔎Sample 👇🏽 the Search Engine Here 🗽', 'hijewellery': 'Fine 3D printed jewellery for tech lovers #3dprintedjewelry #wearabletech #jewellery', 'yhanchristian': 'Estudante de Engenharia, Maker e viciado em café.', 'femka': 'Fashion Futurist + Fashion Tech Lab Founder #technoirlab + Fashion Designer / Parsons & CSM Grad / Obsessed with #fashiontech #future #cryptocurrency', 'sinhbisen': 'Creator, TRiiNG, augmented jewellery label ⭕️ Transhumanist ⭕️ Corporeal cartographer ⭕️', 'stellawearables': '#StellaWearables ✉️Info#StellaWearables.com Premium Wearable Technology That Monitors Personal Health & Environments ☀️🏝🏜🏔', 'ivoomi_india': 'We are the manufacturers of the most innovative technologies and user-friendly gadgets with a global presence.', 'bgutenschwager': "When it comes to life, it's all about the experience.\nGoogle Mapper 🗺\n360 Photographer 📷\nBrand Rep #QuickTutor", 'storiesofdesign': 'Putting stories at the heart of brands and businesses | Cornwall and London | #storiesofdesign', 'trume.jp': '草創期から国産ウオッチの製造に取り組み、挑戦を続けてきたエプソンが世界に放つ新ブランド「TRUME」(トゥルーム)。目指すのは、最先端技術でアナログウオッチを極めるブランド。', 'themarinesss': "I didn't choose the blog life, the blog life chose me | Aspiring Children's Book Author | www.slayathomemum.com", 'ayowearable': 'The world’s first light-based wearable that helps you sleep better, beat jet lag and have more energy! #goAYO Get yours at:', 'wearyourowntechs': 'Bringing you the latest trends, Current Products and Reviews of Wearable Technology. Discover how they can enhance your Life and Lifestyle', 'roxfordwatches': 'The Roxford | The most stylish and customizable fitness smartwatch. Tracks your steps/calories/dist/sleep. Comes with FOUR bands, and a travel case!', 'playertek': "Track your entire performance - every training session, every match. \nBecause the best players don't hide.", '_kate_hartman_': '', 'hmsmc10': 'Health & Wellness 🍎\nBoston, MA 🏙\nSuffolk MPA ‘17 🎓 \n.\nJust Strong Ambassador 🏋🏻\u200d♀️', 'gadgetxtreme': 'Dedicated to reviewing gadgets, technologies, internet products and breaking tech news. Follow us to see daily vblogs on all the disruptive tech..', 'freedom.journey.leader': '📍MN\n🍃Wife • Homeschooling Mom to 5 🐵 • D Y I lover 🔨 • Small town living in MN. 🌿 \n📧Ashleybp5#gmail.com \n#homeschool #bossmom #builder #momlife', 'arts_food_life': 'Life through my phone.', 'medgizmo': 'Wearable #tech: #health #healthcare #wellness #gadgets #apps. Images/links provided as information resource only; doesn’t mean we endorse referenced', 'sawearables': 'The home of wearable tech in South Africa!\n--> #WearableTech #WearableTechnology #FitnessTech Find your wearable #', 'shop.mercury': 'Changing the way you charge.⚡️\nGet exclusive product discounts, and help us reach our goal below!🔋', 'invisawear': 'PRE-ORDERS NOW AVAILABLE! Get yours 25% OFF here: #girlboss #wearabletech'}
for key in d:
print("---with emojis----")
print(d[key])
print("---emojis removed----")
x=''.join(c for c in d[key] if c <= '\uFFFF')
print(x)
output example
---with emojis----
📍MN
🍃Wife • Homeschooling Mom to 5 🐵 • D Y I lover 🔨 • Small town living in MN. 🌿
📧Ashleybp5#gmail.com
#homeschool #bossmom #builder #momlife
---emojis removed----
MN
Wife • Homeschooling Mom to 5 • D Y I lover • Small town living in MN.
Ashleybp5#gmail.com
#homeschool #bossmom #builder #momlife
---with emojis----
Changing the way you charge.⚡️
Get exclusive product discounts, and help us reach our goal below!🔋
---emojis removed----
Changing the way you charge.⚡️
Get exclusive product discounts, and help us reach our goal below!
There is no technical definition of what an "emoji" is. Various glyphs may be used to render printable characters, symbols, control characters and the like. What seems like an "emoji" to you may be part of normal script to others.
What you probably want to do is to look at the Unicode category of each character and filter out various categories. While this does not solve the "emoji"-definition-problem per se, you get much better control over what you are actually doing without removing, for example, literally all characters of languages spoken by 2/3 of the planet.
Instead of filtering out certain categories, you may filter everything except the lower- and uppercase letters (and numbers). However, be aware that ꙭ is not "the googly eyes emoji" but the CYRILLIC SMALL LETTER DOUBLE MONOCULAR O, which is a normal lowercase letter to millions of people.
For example:
import unicodedata
s = "🍃Wife • Homeschooling Mom to 5 🐵 • D Y I lover 🔨 • Small town living in MN. 🌿"
# Just filter category "symbol"
t = ''.join(c for c in s if unicodedata.category(c) not in ('So', ))
print(t)
...results in
Wife • Homeschooling Mom to 5 • D Y I lover • Small town living in MN.
This may not be emoji-free enough, yet the • is technically a form of punctuation. So filter this as well
# Filter symbols and punctuations. You may want 'Cc' as well,
# to get rid of control characters. Beware that newlines are a
# form of control-character.
t = ''.join(c for c in s if unicodedata.category(c) not in ('So', 'Po'))
print(t)
And you get
Wife Homeschooling Mom to 5 D Y I lover Small town living in MN

Resources