I'm currently using DNN regressor with 11 input neurons multiple hidden layers, and 11 output neurons.
I've have finished training the model. However, when I try to interface with the model with this code:
a = np.array(0,0,0,0,0,0,0,0,0,0,0)
y_pred = regressor.predict(a, as_iterable=False)
print(y_pred)
However this generates an error:
InvalidArgumentError (see above for traceback): Input to reshape is a tensor with 11 values, but the requested shape has 121
[[node dnn/input_from_feature_columns/input_layer/X/Reshape (defined at regressor_full.py:177) ]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "regressor_full.py", line 206, in <module>
a = np.array(0,0,0,0,0,0,0,0,0,0,0)
ValueError: only 2 non-keyword arguments accepted.
I'm just trying to generate a prediction for 11 input variables and predict 11 output variables(as I trained the model), however, it generates an error.
I've tried changing the as_iterable to True, however this doesn't change anything. What is currently causing the error, and how do I fix it? Thank you for your time.
As requested, I included the code that defines the model.
import tensorflow.contrib.learn as skflow
regressor = skflow.DNNRegressor(feature_columns=feature_columns,
label_dimension=11,
hidden_units=hidden_layers,
model_dir=MODEL_PATH,
dropout=dropout,
config=test_config,
activation_fn = tf.nn.relu,
optimizer = tf.compat.v1.train.AdamOptimizer(learning_rate = learning_rate)
)
Try changing the input array to a = np.array([0,0,0,0,0,0,0,0,0,0,0],dtype=" ") and also define the dtype which might solve the issue.
Related
I am trying to fine tune a hugging face model onto a Shell Code dataset (https://huggingface.co/datasets/SoLID/shellcode_i_a32)
The training code is a basic hugging face trainer method but we keep running into nan/inf issues
from transformers import PreTrainedTokenizerFast
tokenizer = PreTrainedTokenizerFast(tokenizer_file="tkn1.json", padding_side="right")
special_tokens={'pad_token': "[PAD]"}
tokenizer.add_special_tokens(special_tokens)
# token_wrap = PreTrainedTokenizer()
data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=model)
training_args = Seq2SeqTrainingArguments(
output_dir="./results",
evaluation_strategy="epoch",
lr_scheduler_type = "cosine",
weight_decay=0.01,
save_total_limit=3,
per_device_train_batch_size=128,
num_train_epochs=5,
warmup_ratio=0.06,
learning_rate=1.0e-04,
# fp16=True,
debug=["underflow_overflow"]
)
trainer = Seq2SeqTrainer(
model=model,
args=training_args,
train_dataset=tokenized_datasets["test"],
eval_dataset=tokenized_datasets["test"],
tokenizer=tokenizer,
data_collator=data_collator,
)
# trainer.train()
# print(tokenizer.)
trainer.train()
# eval_loss = trainer.evaluate()
# print(f">>> Perplexity: {math.exp(eval_loss['eval_loss']):.2f}")
The outputs look like -
You're using a PreTrainedTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
Detected inf/nan during batch_number=0
Last 1 forward frames:
abs min abs max metadata
shared Embedding
5.42e-06 2.04e+04 weight
0.00e+00 1.46e+03 input[0]
1.56e-03 2.04e+04 output
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-120-ff4a54906908> in <module>
33 # trainer.train()
34 # print(tokenizer.)
---> 35 trainer.train()
36 # eval_loss = trainer.evaluate()
37 # print(f">>> Perplexity: {math.exp(eval_loss['eval_loss']):.2f}")
9 frames
/usr/local/lib/python3.8/dist-packages/transformers/debug_utils.py in forward_hook(self, module, input, output)
278
279 # now we can abort, as it's pointless to continue running
--> 280 raise ValueError(
281 "DebugUnderflowOverflow: inf/nan detected, aborting as there is no point running further. "
282 "Please scroll up above this traceback to see the activation values prior to this event."
ValueError: DebugUnderflowOverflow: inf/nan detected, aborting as there is no point running further. Please scroll up above this traceback to see the activation values prior to this event.
The very first layer seems to start throwing inf/nans when we start training and doesn't go much beyond that
We have tried tweaking our training arguments but have hit a brick wall here. Any help appreciated!
I am trying to save a fine-tuned binary classification model based on pretrained Bert module 'uncased_L-12_H-768_A-12'. I'm using tf2.
The code set up the model structure:
bert_classifier, bert_encoder =bert.bert_models.classifier_model(bert_config, num_labels=2)
then:
# import pre-trained model structure from the check point file
checkpoint = tf.train.Checkpoint(model=bert_encoder)
checkpoint.restore(
os.path.join(gs_folder_bert, 'bert_model.ckpt')).assert_consumed()
then: I compiled and fit the model
bert_classifier.compile(
optimizer=optimizer,
loss=loss,
metrics=metrics)
bert_classifier.fit(
Text_train, Label_train,
validation_data=(Text_val, Label_val),
batch_size=32,
epochs=1)
at last: I saved the model in the model folder which then automatically generates a file named saved_model.pb within
bert_classifier.save('/content/drive/My Drive/model')
also tried this:
tf.saved_model.save(bert_classifier, export_dir='/content/drive/My Drive/omg')
now I try to load the model and apply it on test data:
from tensorflow import keras
ttt = keras.models.load_model('/content/drive/My Drive/model')
I got:
KeyError Traceback (most recent call last)
<ipython-input-77-93f80aa585da> in <module>()
----> 1 tf.keras.models.load_model(filepath='/content/drive/My Drive/omg', custom_objects={'Transformera':bert_classifier})
9 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/saving/saved_model/load.py in _revive_graph_network(self, metadata, node_id)
392 else:
393 model = models_lib.Functional(
--> 394 inputs=[], outputs=[], name=config['name'])
395
396 # Record this model and its layers. This will later be used to reconstruct
KeyError: 'name'
This error message doesn't help me with what to do...please kindly advice.
I also tried to save the model in h5 format, but when i load it
ttt = keras.models.load_model('/content/drive/My Drive/model.h5')
I got this error:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-36-12f76139ec24> in <module>()
----> 1 ttt = keras.models.load_model('/content/drive/My Drive/model.h5')
5 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/utils/generic_utils.py in class_and_config_for_serialized_keras_object(config, module_objects, custom_objects, printable_module_name)
294 cls = get_registered_object(class_name, custom_objects, module_objects)
295 if cls is None:
--> 296 raise ValueError('Unknown ' + printable_module_name + ': ' + class_name)
297
298 cls_config = config['config']
ValueError: Unknown layer: BertClassifier
Seems as if you have the answer right in the question: '/content/drive/My Drive/model' will fail due to the whitespace character.
You could try it with escaping the backspace: '/content/drive/My\ Drive/model'.
Other option, after I had exactly the same problem with saving and loading. What helped was to just save the weights of the pre-trained model and not saving the whole model:
Just take a look right here: https://keras.io/api/models/model_saving_apis/, especially at the methods save_weights() and load_weights().
I have built a Sequential Keras model with three layers: A Gaussian Noise layer, a hidden layer, and the output layer with the same dimension as the input layer. For this, I'm using the Keras package that comes with Tensorflow 2.0.0-beta1. Thus, I'd like to get the output of the hidden layer, such that I circumvent the Gaussian Noise layer since it's only necessary in the training phase.
To achieve my goal, I followed the instructions in https://keras.io/getting-started/faq/#how-can-i-obtain-the-output-of-an-intermediate-layer, which are pretty much described in Keras, How to get the output of each layer? too.
I have tried the following example from the official Keras documentation:
from tensorflow import keras
from tensorflow.keras import backend as K
dae = keras.Sequential([
keras.layers.GaussianNoise( 0.001, input_shape=(10,) ),
keras.layers.Dense( 80, name="hidden", activation="relu" ),
keras.layers.Dense( 10 )
])
optimizer = keras.optimizers.Adam()
dae.compile( loss="mse", optimizer=optimizer, metrics=["mae"] )
# Here the fitting process...
# dae.fit( ยท )
# Attempting to retrieve a decoder functor.
encoder = K.function([dae.input, K.learning_phase()],
[dae.get_layer("hidden").output])
However, when K.learning_phase() is used to create the Keras backend functor, I get the error:
Traceback (most recent call last):
File "/anaconda3/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/keras/backend.py", line 534, in _scratch_graph
yield graph
File "/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/keras/backend.py", line 3670, in __init__
base_graph=source_graph)
File "/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/eager/lift_to_graph.py", line 249, in lift_to_graph
visited_ops = set([x.op for x in sources])
File "/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/eager/lift_to_graph.py", line 249, in <listcomp>
visited_ops = set([x.op for x in sources])
AttributeError: 'int' object has no attribute 'op'
The code works great if I don't include K.learning_phase(), but I need to make sure that the output from my hidden layer is evaluated over an input that is not polluted with noise (i.e. in "test" mode -- not "training" mode).
I know my other option is to create a model from the original denoising autoencoder, but can anyone point me into why my approach from the officially documented functor creation fails?
Firstly, ensure your packages are up-to-date, as your script works fine for me. Second, encoder won't get the outputs - continuing from your snippet after # Here is the fitting process...,
x = np.random.randn(32, 10) # toy data
y = np.random.randn(32, 10) # toy labels
dae.fit(x, y) # run one iteration
encoder = K.function([dae.input, K.learning_phase()], [dae.get_layer("hidden").output])
outputs = [encoder([x, int(False)])][0][0] # [0][0] to index into nested list of len 1
print(outputs.shape)
# (32, 80)
However, as of Tensorflow 2.0.0-rc2, this will not work with eager execution enabled - disable via:
tf.compat.v1.disable_eager_execution()
How to save the weights of VGG-16 model after training it? How to load the saved weights in to the model?
I tried this:
fname = "weights-Test-CNN.hdf5"
custom_vgg_model.save_weights(fname,overwrite=True)
custom_vgg_model.load_weights(weights-Test-CNN.hdf5, by_name=False)
I got following error:
NameError Traceback (most recent call
last) in ()
----> 1 custom_vgg_model.load_weights(weights-Test-CNN.hdf5, by_name=False)
NameError: name 'weights' is not defined
Surround the second weights-Test-CNN.hdf5 with quotes or you use fname as you have defined...
custom_vgg_model.load_weights("weights-Test-CNN.hdf5", by_name=False)
My model is correctly formed:
model = Sequential()
model.add(Lambda(lambda x:x/255.0 - 0.5, input_shape=(160,320,3)))
model.compile(loss='mse', optimizer='adam')
model.fit(train_generator, samples_per_epoch= len(train_samples), validation_data=validation_generator, nb_val_samples=len(validation_samples), nb_epoch=3)
Note, parenthesis are in place. However when I fit I get the following error:
Traceback (most recent call last): File "modell.py", line 70, in <module>
model.fit(train_generator, samples_per_epoch= len(train_samples), validation_data=validation_generator, nb_val_samples=len(validation_samples), nb_epoch=3)
TypeError: fit() missing 1 required positional argument: 'y'
train_generator is a 2D array
train_generator = generator(train_samples, batch_size=32)
I must be blind because I can't spot the problem. Does anyone know why fit is looking for an extra argument?
When using a generator to train, you must use the method model.fit_generator.
The method fit will always demand for inputs (X) and outputs/targets (Y)