Are the pre-trained layers of the Huggingface BERT models frozen? - nlp
I use the following classification model from Huggingface:
model = AutoModelForSequenceClassification.from_pretrained("dbmdz/bert-base-german-cased", num_labels=2).to(device)
As I understand, this adds a dense layer at the end of the pre-trained model which has 2 output nodes. But are all the pre-trained layers before that frozen? Or are they also updated when fine-tuning? I can't find information about that in the docs...
So do I still have to do something like this?:
for param in model.bert.parameters():
param.requires_grad = False
They are not frozen. All parameters are trainable by default. You can also check that with:
for name, param in model.named_parameters():
print(name, param.requires_grad)
Output:
bert.embeddings.word_embeddings.weight True
bert.embeddings.position_embeddings.weight True
bert.embeddings.token_type_embeddings.weight True
bert.embeddings.LayerNorm.weight True
bert.embeddings.LayerNorm.bias True
bert.encoder.layer.0.attention.self.query.weight True
bert.encoder.layer.0.attention.self.query.bias True
bert.encoder.layer.0.attention.self.key.weight True
bert.encoder.layer.0.attention.self.key.bias True
bert.encoder.layer.0.attention.self.value.weight True
bert.encoder.layer.0.attention.self.value.bias True
bert.encoder.layer.0.attention.output.dense.weight True
bert.encoder.layer.0.attention.output.dense.bias True
bert.encoder.layer.0.attention.output.LayerNorm.weight True
bert.encoder.layer.0.attention.output.LayerNorm.bias True
bert.encoder.layer.0.intermediate.dense.weight True
bert.encoder.layer.0.intermediate.dense.bias True
bert.encoder.layer.0.output.dense.weight True
bert.encoder.layer.0.output.dense.bias True
bert.encoder.layer.0.output.LayerNorm.weight True
bert.encoder.layer.0.output.LayerNorm.bias True
bert.encoder.layer.1.attention.self.query.weight True
bert.encoder.layer.1.attention.self.query.bias True
bert.encoder.layer.1.attention.self.key.weight True
bert.encoder.layer.1.attention.self.key.bias True
bert.encoder.layer.1.attention.self.value.weight True
bert.encoder.layer.1.attention.self.value.bias True
bert.encoder.layer.1.attention.output.dense.weight True
bert.encoder.layer.1.attention.output.dense.bias True
bert.encoder.layer.1.attention.output.LayerNorm.weight True
bert.encoder.layer.1.attention.output.LayerNorm.bias True
bert.encoder.layer.1.intermediate.dense.weight True
bert.encoder.layer.1.intermediate.dense.bias True
bert.encoder.layer.1.output.dense.weight True
bert.encoder.layer.1.output.dense.bias True
bert.encoder.layer.1.output.LayerNorm.weight True
bert.encoder.layer.1.output.LayerNorm.bias True
bert.encoder.layer.2.attention.self.query.weight True
bert.encoder.layer.2.attention.self.query.bias True
bert.encoder.layer.2.attention.self.key.weight True
bert.encoder.layer.2.attention.self.key.bias True
bert.encoder.layer.2.attention.self.value.weight True
bert.encoder.layer.2.attention.self.value.bias True
bert.encoder.layer.2.attention.output.dense.weight True
bert.encoder.layer.2.attention.output.dense.bias True
bert.encoder.layer.2.attention.output.LayerNorm.weight True
bert.encoder.layer.2.attention.output.LayerNorm.bias True
bert.encoder.layer.2.intermediate.dense.weight True
bert.encoder.layer.2.intermediate.dense.bias True
bert.encoder.layer.2.output.dense.weight True
bert.encoder.layer.2.output.dense.bias True
bert.encoder.layer.2.output.LayerNorm.weight True
bert.encoder.layer.2.output.LayerNorm.bias True
bert.encoder.layer.3.attention.self.query.weight True
bert.encoder.layer.3.attention.self.query.bias True
bert.encoder.layer.3.attention.self.key.weight True
bert.encoder.layer.3.attention.self.key.bias True
bert.encoder.layer.3.attention.self.value.weight True
bert.encoder.layer.3.attention.self.value.bias True
bert.encoder.layer.3.attention.output.dense.weight True
bert.encoder.layer.3.attention.output.dense.bias True
bert.encoder.layer.3.attention.output.LayerNorm.weight True
bert.encoder.layer.3.attention.output.LayerNorm.bias True
bert.encoder.layer.3.intermediate.dense.weight True
bert.encoder.layer.3.intermediate.dense.bias True
bert.encoder.layer.3.output.dense.weight True
bert.encoder.layer.3.output.dense.bias True
bert.encoder.layer.3.output.LayerNorm.weight True
bert.encoder.layer.3.output.LayerNorm.bias True
bert.encoder.layer.4.attention.self.query.weight True
bert.encoder.layer.4.attention.self.query.bias True
bert.encoder.layer.4.attention.self.key.weight True
bert.encoder.layer.4.attention.self.key.bias True
bert.encoder.layer.4.attention.self.value.weight True
bert.encoder.layer.4.attention.self.value.bias True
bert.encoder.layer.4.attention.output.dense.weight True
bert.encoder.layer.4.attention.output.dense.bias True
bert.encoder.layer.4.attention.output.LayerNorm.weight True
bert.encoder.layer.4.attention.output.LayerNorm.bias True
bert.encoder.layer.4.intermediate.dense.weight True
bert.encoder.layer.4.intermediate.dense.bias True
bert.encoder.layer.4.output.dense.weight True
bert.encoder.layer.4.output.dense.bias True
bert.encoder.layer.4.output.LayerNorm.weight True
bert.encoder.layer.4.output.LayerNorm.bias True
bert.encoder.layer.5.attention.self.query.weight True
bert.encoder.layer.5.attention.self.query.bias True
bert.encoder.layer.5.attention.self.key.weight True
bert.encoder.layer.5.attention.self.key.bias True
bert.encoder.layer.5.attention.self.value.weight True
bert.encoder.layer.5.attention.self.value.bias True
bert.encoder.layer.5.attention.output.dense.weight True
bert.encoder.layer.5.attention.output.dense.bias True
bert.encoder.layer.5.attention.output.LayerNorm.weight True
bert.encoder.layer.5.attention.output.LayerNorm.bias True
bert.encoder.layer.5.intermediate.dense.weight True
bert.encoder.layer.5.intermediate.dense.bias True
bert.encoder.layer.5.output.dense.weight True
bert.encoder.layer.5.output.dense.bias True
bert.encoder.layer.5.output.LayerNorm.weight True
bert.encoder.layer.5.output.LayerNorm.bias True
bert.encoder.layer.6.attention.self.query.weight True
bert.encoder.layer.6.attention.self.query.bias True
bert.encoder.layer.6.attention.self.key.weight True
bert.encoder.layer.6.attention.self.key.bias True
bert.encoder.layer.6.attention.self.value.weight True
bert.encoder.layer.6.attention.self.value.bias True
bert.encoder.layer.6.attention.output.dense.weight True
bert.encoder.layer.6.attention.output.dense.bias True
bert.encoder.layer.6.attention.output.LayerNorm.weight True
bert.encoder.layer.6.attention.output.LayerNorm.bias True
bert.encoder.layer.6.intermediate.dense.weight True
bert.encoder.layer.6.intermediate.dense.bias True
bert.encoder.layer.6.output.dense.weight True
bert.encoder.layer.6.output.dense.bias True
bert.encoder.layer.6.output.LayerNorm.weight True
bert.encoder.layer.6.output.LayerNorm.bias True
bert.encoder.layer.7.attention.self.query.weight True
bert.encoder.layer.7.attention.self.query.bias True
bert.encoder.layer.7.attention.self.key.weight True
bert.encoder.layer.7.attention.self.key.bias True
bert.encoder.layer.7.attention.self.value.weight True
bert.encoder.layer.7.attention.self.value.bias True
bert.encoder.layer.7.attention.output.dense.weight True
bert.encoder.layer.7.attention.output.dense.bias True
bert.encoder.layer.7.attention.output.LayerNorm.weight True
bert.encoder.layer.7.attention.output.LayerNorm.bias True
bert.encoder.layer.7.intermediate.dense.weight True
bert.encoder.layer.7.intermediate.dense.bias True
bert.encoder.layer.7.output.dense.weight True
bert.encoder.layer.7.output.dense.bias True
bert.encoder.layer.7.output.LayerNorm.weight True
bert.encoder.layer.7.output.LayerNorm.bias True
bert.encoder.layer.8.attention.self.query.weight True
bert.encoder.layer.8.attention.self.query.bias True
bert.encoder.layer.8.attention.self.key.weight True
bert.encoder.layer.8.attention.self.key.bias True
bert.encoder.layer.8.attention.self.value.weight True
bert.encoder.layer.8.attention.self.value.bias True
bert.encoder.layer.8.attention.output.dense.weight True
bert.encoder.layer.8.attention.output.dense.bias True
bert.encoder.layer.8.attention.output.LayerNorm.weight True
bert.encoder.layer.8.attention.output.LayerNorm.bias True
bert.encoder.layer.8.intermediate.dense.weight True
bert.encoder.layer.8.intermediate.dense.bias True
bert.encoder.layer.8.output.dense.weight True
bert.encoder.layer.8.output.dense.bias True
bert.encoder.layer.8.output.LayerNorm.weight True
bert.encoder.layer.8.output.LayerNorm.bias True
bert.encoder.layer.9.attention.self.query.weight True
bert.encoder.layer.9.attention.self.query.bias True
bert.encoder.layer.9.attention.self.key.weight True
bert.encoder.layer.9.attention.self.key.bias True
bert.encoder.layer.9.attention.self.value.weight True
bert.encoder.layer.9.attention.self.value.bias True
bert.encoder.layer.9.attention.output.dense.weight True
bert.encoder.layer.9.attention.output.dense.bias True
bert.encoder.layer.9.attention.output.LayerNorm.weight True
bert.encoder.layer.9.attention.output.LayerNorm.bias True
bert.encoder.layer.9.intermediate.dense.weight True
bert.encoder.layer.9.intermediate.dense.bias True
bert.encoder.layer.9.output.dense.weight True
bert.encoder.layer.9.output.dense.bias True
bert.encoder.layer.9.output.LayerNorm.weight True
bert.encoder.layer.9.output.LayerNorm.bias True
bert.encoder.layer.10.attention.self.query.weight True
bert.encoder.layer.10.attention.self.query.bias True
bert.encoder.layer.10.attention.self.key.weight True
bert.encoder.layer.10.attention.self.key.bias True
bert.encoder.layer.10.attention.self.value.weight True
bert.encoder.layer.10.attention.self.value.bias True
bert.encoder.layer.10.attention.output.dense.weight True
bert.encoder.layer.10.attention.output.dense.bias True
bert.encoder.layer.10.attention.output.LayerNorm.weight True
bert.encoder.layer.10.attention.output.LayerNorm.bias True
bert.encoder.layer.10.intermediate.dense.weight True
bert.encoder.layer.10.intermediate.dense.bias True
bert.encoder.layer.10.output.dense.weight True
bert.encoder.layer.10.output.dense.bias True
bert.encoder.layer.10.output.LayerNorm.weight True
bert.encoder.layer.10.output.LayerNorm.bias True
bert.encoder.layer.11.attention.self.query.weight True
bert.encoder.layer.11.attention.self.query.bias True
bert.encoder.layer.11.attention.self.key.weight True
bert.encoder.layer.11.attention.self.key.bias True
bert.encoder.layer.11.attention.self.value.weight True
bert.encoder.layer.11.attention.self.value.bias True
bert.encoder.layer.11.attention.output.dense.weight True
bert.encoder.layer.11.attention.output.dense.bias True
bert.encoder.layer.11.attention.output.LayerNorm.weight True
bert.encoder.layer.11.attention.output.LayerNorm.bias True
bert.encoder.layer.11.intermediate.dense.weight True
bert.encoder.layer.11.intermediate.dense.bias True
bert.encoder.layer.11.output.dense.weight True
bert.encoder.layer.11.output.dense.bias True
bert.encoder.layer.11.output.LayerNorm.weight True
bert.encoder.layer.11.output.LayerNorm.bias True
bert.pooler.dense.weight True
bert.pooler.dense.bias True
classifier.weight True
classifier.bias True
model = AutoModelForSequenceClassification.from_pretrained("dbmdz/bert-base-german-cased", num_labels=2).to(device)
all(p.requires_grad for p in model.parameters()
# True
So you'll need to freeze them yourself if you want them frozen.
Related
pandas dataframe makes a boolean column based on the date in the list
Assuming there is a data frame with 2 columns ds, y (the input of fbprophet https://facebook.github.io/prophet/docs/quick_start.html) ds y 2021-03-17 3135.73 2021-03-18 3027.99 2021-03-19 3074.96 2021-03-22 3110.87 2021-03-23 3110.87 2021-03-24 3110.87 2021-03-25 3110.87 Here is the code to generate this dataframe data = \ { 'ds':['2021-03-17','2021-03-18','2021-03-19','2021-03-22','2021-03-23','2021-03-24','2021-03-25'], 'y':['3135.73','3027.99','3074.96','3110.87','3110.87','3110.87','3110.87'] } df = pd.DataFrame(data) How could I generate 2 new boolean columns on_season, off_season based on the "date" in a loop? So for example, if the ds column is in between date and date-2, then on_season=TRUE, off_season = OFF desired output since the date in the loop here are '2021-03-19','2021-03-25'. So on_season is between '2021-03-19' and '2021-03-17' and '2021-03-25' and '2021-03-23'. ds y on_season off_season 2021-03-17 3135.73 TRUE FALSE 2021-03-18 3027.99 TRUE FALSE 2021-03-19 3074.96 TRUE FALSE 2021-03-22 3110.87 FALSE TRUE 2021-03-23 3110.87 TRUE FALSE 2021-03-24 3110.87 TRUE FALSE 2021-03-25 3110.87 TRUE FALSE Here is the working progress code # the date in a loop for date in ['2021-03-19','2021-03-25']: date_str='' date_str=date_str.join(date) # convert date from list into string on_season=pd.to_datetime(date_str) on_season = on_season.strftime("%Y-%m-%d") #try to make the day range off_season=pd.to_datetime(date_str) - timedelta(2) off_season=off_season.strftime("%Y-%m-%d") # convert datetime object to string # I am stuck in the dataframe part
It seems you just need to calculate the ds diff between adjacent rows: df.ds = pd.to_datetime(df.ds) # check if difference between two rows are smaller than 2 df['on_season'] = df.ds.diff().dt.days.fillna(0) < 2 df['off_season'] = ~df['on_season'] df ds y on_season off_season 0 2021-03-17 3135.73 True False 1 2021-03-18 3027.99 True False 2 2021-03-19 3074.96 True False 3 2021-03-22 3110.87 False True 4 2021-03-23 3110.87 True False 5 2021-03-24 3110.87 True False 6 2021-03-25 3110.87 True False
How to add 1 more breakdown color in plotly funnel?
I have 1 state on "og"- false, and 2 states on "no og" true and false is_retargeting media_source step Values False og step1 31992 True no og step1 33644 False no og step1 32199 False no og step2 20699 False og step2 47796 True no og step2 23668 True no og step3 10744 False no og step3 27622 False og step3 20979 False no og step4 16459 False og step4 28444 True no og step4 49749 i come up with this code, but it shows only 2 colors but breaks "no og" with no explanation. here is the screenshot what i get How can add one more breakdown column is_retargeting for this funnel so it will show true or false for "no og"? import pandas as pd pd.set_option('max_columns', None) df = pd.read_csv("name.csv") df.loc[df["media_source"] != "organic", "media_source"] = 'non organic' df = df.groupby(['is_retargeting', 'media_source']).sum().reset_index() df1 = df.melt(id_vars=["is_retargeting", "media_source"], var_name="step", value_name="Values") fig = px.funnel(df1, x='Values', y='step', color='media_source') fig.show()
Number of occurrences between datetime
I have the following code: import pandas as pd from pandas import datetime from pandas import DataFrame as df import matplotlib from pandas_datareader import data as web import datetime import numpy as np stock = '^GSPC' start = datetime.date(2000,1,1) end = datetime.date.today() data = web.DataReader(stock, 'yahoo',start, end) data.index = pd.to_datetime(data.index, format ='%Y-%m-%d') data['pct_day']= data['Adj Close'].pct_change() How can I find the number of occurrences of value over 0.05 in column pct_day and the date when it occurred? --Edit 1: In the comment i was suggested : df[data['pct_day'].gt(0.005)] which produces the following series: Date 2001-01-03 True 2002-07-24 True 2002-07-29 True 2008-09-30 True 2008-10-13 True 2008-10-28 True 2008-11-13 True 2008-11-21 True 2008-11-24 True 2008-12-16 True 2009-03-10 True 2009-03-23 True Name: pct_day, dtype: bool But when i use: df[data['pct_day'].gt(0.03)] it produces the following series: Date 2000-03-16 False 2000-04-17 False 2000-04-25 False 2000-05-30 False 2000-10-13 False ... 2011-10-27 False 2011-11-30 False 2015-08-26 False 2018-12-26 False 2019-01-04 False Name: pct_day, Length: 66, dtype: bool Why is it including the ones with False in the last example but not the first one? Does the length of the second which is 66, also include False? I ask this because data has over 5000 entries but the length in the last example is 66.
how to check if a list value exist in a column value which is also a list
I have the following Dataframe: Name 0 [Scrap] 1 [Machinery, Scrap] 2 [RMG] 3 [Leather] 4 [Machinery] 5 [RMG, Optical Frame, Machinery] I want to check if "Scrap" or "Machinery" exist in Name column. like below: Name Type 0 [Scrap] True 1 [Machinery, Scrap] True 2 [RMG] False 3 [Leather] False 4 [Machinery] True 5 [RMG, Optical Frame, Machinery] True I have tried the following code. df['Type'] = df.apply(lambda x: ["Scrap", "Iron","Machinery"] in x["Name"], axis=1) However, i get the following result: Name Type 0 [Scrap] False 1 [Machinery, Scrap] False 2 [RMG] False 3 [Leather] False 4 [Machinery] False 5 [RMG, Optical Frame, Machinery] False
This is one approach using .apply. Ex: to_check = ("Scrap", "Iron","Machinery") df = pd.DataFrame({"Name": [["Scrap"], ["Machinery", "Scrap"], ["RMG"], ["Leather"], ["Machinery"], ["RMG", "Optical Frame", "Machinery"]]}) df['Type'] = df["Name"].apply(lambda x: any(i in x for i in to_check)) print(df) Output: Name Type 0 [Scrap] True 1 [Machinery, Scrap] True 2 [RMG] False 3 [Leather] False 4 [Machinery] True 5 [RMG, Optical Frame, Machinery] True
You are checking if the whole list can be found, which is why you are getting no True. Try df['Type'] = df.apply(lambda elem: ("Scrap" in elem["Name"]) or ("Machinery" in elem["Name"]), axis=1)
You can also use set intersection to see whether there exists a common element. df['Type'] = df.apply(lambda x: bool( set(["Scrap", "Iron","Machinery"]) & set(x["Name"]) ), axis=1)
Test for overlapping times in Python
I am following this word for word, but am getting different results compared to the original post and I have no idea why. I am wanting to compare pairs of times to one another to check where they overlap. I am using Python 3.6.6 Find if dates are overlapping in a list of N pairs This is what I coded: import datetime from collections import namedtuple from itertools import combinations timesok = [('09:30', '10:00'), ('10:00', '10:30'), ('10:30', '11:00')] wrongtimes1 = [('9:30', '10:00'), ('9:00', '10:30'), ('10:30', '11:00')] wrongtimes2=[('9:30', '10:00'), ('10:00', '10:30'), ('9:15', '9:45')] def test_overlap(dt1_st, dt1_end, dt2_st, dt2_end): Range = namedtuple('Range', ['start', 'end']) r1 = Range(start=dt1_st, end=dt1_end) r2 = Range(start=dt2_st, end=dt2_end) latest_start = max(r1.start, r2.start) earliest_end = min(r1.end, r2.end) overlap = (earliest_end - latest_start) return overlap.seconds def find_overlaps(times): pairs = list(combinations(times, 2)) print(pairs) for pair in pairs: start1 = datetime.datetime.strptime(pair[0][0], '%H:%M') end1 = datetime.datetime.strptime(pair[0][1], '%H:%M') start2 = datetime.datetime.strptime(pair[1][0], '%H:%M') end2 = datetime.datetime.strptime(pair[1][1], '%H:%M') yield test_overlap(start1, end1, start2, end2) > 0 list(find_overlaps(timesok)) list(find_overlaps(wrongtimes1)) list(find_overlaps(wrongtimes2)) # timesok result: list(find_overlaps(timesok)) [(('09:30', '10:00'), ('10:00', '10:30')), (('09:30', '10:00'), ('10:30', '11:00')), (('10:00', '10:30'), ('10:30', '11:00'))] Out[7]: [False, True, False] # wrongtimes1 result: list(find_overlaps(wrongtimes1)) [(('9:30', '10:00'), ('9:00', '10:30')), (('9:30', '10:00'), ('10:30', '11:00')), (('9:00', '10:30'), ('10:30', '11:00'))] Out[8]: [True, True, False] # wrongtimes2 result: list(find_overlaps(wrongtimes2)) [(('9:30', '10:00'), ('10:00', '10:30')), (('9:30', '10:00'), ('9:15', '9:45')), (('10:00', '10:30'), ('9:15', '9:45'))] Out[9]: [False, True, True] I think the results should be as follows (which corresponds with the original example in the link above): Out[7]: [False, False, False] Out[8]: [True, False, False] Out[9]: [False, True, False] Have I missed something very obvious here? I am a total Python newbie so be gentle with me (!)
overlap.seconds does not return what you think it does. overlap is the result of a datetime subtraction, which is datetime.timedelta. timedelta.seconds does not return the total number of seconds. It returns the second part of the delta, and if the timedelta is -1 day, 23:45:00, timedelta.seconds is 85500. What you meant to write was overlap.total_seconds(): def test_overlap(dt1_st, dt1_end, dt2_st, dt2_end): Range = namedtuple('Range', ['start', 'end']) r1 = Range(start=dt1_st, end=dt1_end) r2 = Range(start=dt2_st, end=dt2_end) latest_start = max(r1.start, r2.start) earliest_end = min(r1.end, r2.end) overlap = (earliest_end - latest_start) return overlap.total_seconds() You still have the problem that your times all have to be at the same day and cannot span wrap around midnight. I don't think using datetime is the right approach here, I think it is easier to program something custom. Here you go: from itertools import combinations def test_overlap(t1_st, t1_end, t2_st, t2_end): def convert_to_minutes(t_str): hours, minutes = t_str.split(':') return 60*int(hours)+int(minutes) t1_st = convert_to_minutes(t1_st) t1_end = convert_to_minutes(t1_end) t2_st = convert_to_minutes(t2_st) t2_end = convert_to_minutes(t2_end) # Check for wrapping time differences if t1_end < t1_st: if t2_end < t2_st: # Both wrap, therefore they overlap at midnight return True # t2 doesn't wrap. Therefore t1 has to start after t2 and end before return t1_st < t2_end or t2_st < t1_end if t2_end < t2_st: # only t2 wraps. Same as before, just reversed return t2_st < t1_end or t1_st < t2_end # They don't wrap and the start of one comes after the end of the other, # therefore they don't overlap if t1_st >= t2_end or t2_st >= t1_end: return False # In all other cases, they have to overlap return True times = [('09:30', '00:00'), ('07:00', '00:30'), ('10:30', '00:15'), ('12:15', '13:30'), ('10:00', '11:00'), ('00:15', '01:15')] pairs = list(combinations(times, 2)) for pair in pairs: start1 = pair[0][0] end1 = pair[0][1] start2 = pair[1][0] end2 = pair[1][1] print(str(test_overlap(start1, end1, start2, end2)) + "\t" + str(pair)) True (('09:30', '00:00'), ('07:00', '00:30')) True (('09:30', '00:00'), ('10:30', '00:15')) True (('09:30', '00:00'), ('12:15', '13:30')) True (('09:30', '00:00'), ('10:00', '11:00')) False (('09:30', '00:00'), ('00:15', '01:15')) True (('07:00', '00:30'), ('10:30', '00:15')) True (('07:00', '00:30'), ('12:15', '13:30')) True (('07:00', '00:30'), ('10:00', '11:00')) True (('07:00', '00:30'), ('00:15', '01:15')) True (('10:30', '00:15'), ('12:15', '13:30')) True (('10:30', '00:15'), ('10:00', '11:00')) False (('10:30', '00:15'), ('00:15', '01:15')) False (('12:15', '13:30'), ('10:00', '11:00')) False (('12:15', '13:30'), ('00:15', '01:15')) False (('10:00', '11:00'), ('00:15', '01:15'))