How to use strategy.scope() in RoBERTa? - python-3.x

Question

I made a model using BERT, for a NLI problem, the algorithm ran without problems, however, when I wanted to adapt it to RoBERTa, and I use strategy.scope (), it generates an error that I don't know how to solve, I appreciate any indication.
´´´
max_len1 = 515 # 128*4 de premisa mas 128*4 de hipotesis
def build_model1():
input_word_ids = tf.keras.Input(shape=(max_len1,), dtype=tf.int32,name="input_word_ids")
input_mask = tf.keras.Input(shape = (max_len1,),dtype=tf.int32,name = "input_mask")
input_type_ids = tf.keras.Input(shape = (max_len1,),dtype=tf.int32,name="input_type_ids")
embedding = model([input_word_ids,input_mask,input_type_ids])[0]
output = tf.keras.layers.Dense(3,activation='softmax')(embedding[:,0,:])
model3 = tf.keras.Model(inputs=[input_word_ids, input_mask, input_type_ids], outputs=output)
model3.compile(tf.keras.optimizers.Adam(lr=1e-5),
loss = 'sparse_categorical_crossentropy', metrics= ['accuracy'])
return model3
with strategy.scope():
model3 = build_model1()
model3.summary()
WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
WARNING:tensorflow:AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7f2425631d00>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with #tf.autograph.experimental.do_not_convert
WARNING:tensorflow:AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7f2425631d00>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with #tf.autograph.experimental.do_not_convert
WARNING: AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7f2425631d00>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with #tf.autograph.experimental.do_not_convert
WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
WARNING:tensorflow:AutoGraph could not transform <function wrap at 0x7f243c214d40> and will run it as-is.
Cause: while/else statement not yet supported
To silence this warning, decorate the function with #tf.autograph.experimental.do_not_convert
WARNING:tensorflow:AutoGraph could not transform <function wrap at 0x7f243c214d40> and will run it as-is.
Cause: while/else statement not yet supported
To silence this warning, decorate the function with #tf.autograph.experimental.do_not_convert
WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
WARNING: AutoGraph could not transform <function wrap at 0x7f243c214d40> and will run it as-is.
Cause: while/else statement not yet supported
To silence this warning, decorate the function with #tf.autograph.experimental.do_not_convert
WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-24-e91a2e7e4b41> in <module>()
1 with strategy.scope():
----> 2 model3 = build_model1()
3 model3.summary()
2 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py in _validate_compile(self, optimizer, metrics, **kwargs)
2533 'with strategy.scope():\n'
2534 ' model=_create_model()\n'
-> 2535 ' model.compile(...)' % (v, strategy))
2536
2537 # Model metrics must be created in the same distribution strategy scope
ValueError: Variable (<tf.Variable 'tfxlm_roberta_model/roberta/encoder/layer_._0/attention/self/query/kernel:0' shape=(1024, 1024) dtype=float32, numpy=
array([[-0.00294119, -0.00129846, 0.00517603, ..., 0.03835522,
0.0218797 , 0.02100084],
[-0.00933813, -0.05062149, 0.01634834, ..., -0.02387142,
0.0113477 , -0.02262339],
[-0.02023344, -0.04181184, -0.00581416, ..., -0.00609464,
0.00801133, 0.00512151],
...,
[-0.02129102, -0.03157991, -0.04071935, ..., 0.04682101,
0.01948426, 0.00312433],
[-0.04902648, -0.01055507, 0.01377375, ..., 0.00845209,
0.01616496, -0.01041171],
[ 0.00759454, -0.00162496, -0.00215843, ..., -0.03199947,
-0.03871808, 0.04949447]], dtype=float32)>) was not created in the distribution strategy scope
of (<tensorflow.python.distribute.tpu_strategy.TPUStrategy object at 0x7f21fcbbb210>). It is most
likely due to not all layers or the model or optimizer being created outside the distribution
strategy scope. Try to make sure your code looks similar to the following.
with strategy.scope():
model=_create_model()
model.compile(...)
´´´
The same code, as I said above, works perfectly for BERT, obviously, for RoBERTa I made the changes in the tokenizer and the loading of the model

Answer 1

I managed to solve it, investigating, I reached that the implementation of roberta went beyond just calling the model