print("sequences",len(sequences))
print("seq_length",(seq_length))
print("vocab size",(vocab_size))
X = tf.zeros((len(sequences), seq_length, vocab_size), dtype=tf.bool)
y = tf.zeros((len(sequences), vocab_size), dtype=tf.bool)
Output
sequences 30373553
seq_length 30
vocab size 1290174
ResourceExhaustedError Traceback (most recent call last)
<ipython-input-35-1bd9b1544ba0> in <module>()
2 print("seq_length",(seq_length))
3 print("vocab size",(vocab_size))
----> 4 X = tf.zeros((len(sequences), seq_length, vocab_size), dtype=tf.bool)
5 y = tf.zeros((len(sequences), vocab_size), dtype=tf.bool)
6
3 frames
/usr/local/lib/python3.6/dist-packages/six.py in raise_from(value, from_value)
ResourceExhaustedError: OOM when allocating tensor with shape[30373553,30,1290174] and type bool on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu [Op:Fill] name: zeros/
Working on tensorflow 2.0
I want to make a matrix of zeroes of shape [30373553,30,1290174]
when runing the same code on TensorFlow 1.5 there was no such error but giving this error when working on Tensorflow 2.0
Assuming each bool element uses 1 byte of memory, your tensor of shape [30373553, 30, 1290174] will take about 1200 TB of memory to materialize. That's a lot of memory...
I'm guessing that this didn't error out in TensorFlow 1.5 because of the old deferred-execution paradigm, where you can call tf.zeros([30373553, 30, 1290174]) without any issue because the symbolic tensor returned by the call won't be actually allocated in memory until you call tf.Session.run() on a tf.Graph that contains the tensor. In TensorFlow 2.0, however, eager execution will perform the memory allocation as soon as the call is made.
Related
I've been running a resnet50 pytorch script on colab for nine months. I haven't run the script for about three weeks and I get the following error now: RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)
My script barfs on this colab cell:
# Train the model
if IS_TRAINING:
# Create the model
mymodel = torchvision.models.resnet50(weights='ResNet50_Weights.DEFAULT')
n_features = mymodel.fc.in_features
# Replace the last layer by our own Linear layer
mymodel.fc = DisMaxLossFirstPart(n_features, len(class_names))
mymodel = mymodel.to(device)
criterion = DisMaxLossSecondPart(mymodel.fc)
optimizer_conv = torch.optim.Adam(mymodel.parameters(), lr=1e-4)
mymodel, train_acc_1, train_loss_1, val_acc_1, val_loss_1 = train_model(
mymodel, criterion=criterion, optimizer=optimizer_conv, scheduler=None, num_epochs=TRAIN_EPOCHS
)`
With this error:
Downloading: "https://download.pytorch.org/models/resnet50-11ad3fa6.pth" to /root/.cache/torch/hub/checkpoints/resnet50-11ad3fa6.pth
100%
97.8M/97.8M [00:00<00:00, 308MB/s]
Epoch 0/3
----------
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-22-01c27b985121> in <module>
12 optimizer_conv = torch.optim.Adam(mymodel.parameters(), lr=1e-4)
13
---> 14 mymodel, train_acc_1, train_loss_1, val_acc_1, val_loss_1 = train_model(
15 mymodel, criterion=criterion, optimizer=optimizer_conv, scheduler=None, num_epochs=TRAIN_EPOCHS
16 )
2 frames
<ipython-input-19-c213dabd46bb> in forward(self, logits, targets, debug, precompute_thresholds)
87 num_classes = logits.size(1)
88 half_batch_size = batch_size//2
---> 89 targets_one_hot = torch.eye(num_classes)[targets].long().cuda()
90
91 if self.model_classifier.training:
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)
As described above, I am running a resnet50 saved model using pytorch on colab. The script used to run without a problem. All I have done today is change the input set of images. I've also verified this by running the last version of my script that worked for me and it barfs in the same way.
I am specifying the device to be used as follows:
device = torch.device("cuda:0" if torch.cuda.is_available() else "CPU")
Shouldn't this be forcing everything on colab to be on the GPU?
I saw that someone else had a similar problem with YOLOv7 but I don't have any code that is "from_which_layer.append((torch.ones(size=(len(b),)) * i)"
As jhso said, I should enter the device in this line:
targets_one_hot = torch.eye(num_classes)[targets].long().cuda()
so I changed it to this:
targets_one_hot = torch.eye(num_classes, device="cuda")[targets].long().cuda()
And that works.
I'm been trying to make a balanced clusters for my masther thesis using Google Colab (because I don't have a GPU) and balanced k-means Library (https://pypi.org/project/balanced-kmeans/), this is the process that I made:
Install the balanced k-means Library in Google Colab using this command:
!pip install balanced_kmeans
Import a CSV file to my Google Colab instance:
from google.colab import files
subido=files.upload()
Create a Pandas dataframe
import pandas as pd
import io
df = pd.read_csv(io.BytesIO(subido['prueba.csv']))
Then, changed the string data type to integer in the columns than I going to use:
df["bytes"] = pd.to_numeric(df["bytes"],errors='coerce')
df["bytes"] = df["bytes"].fillna(0)
df["paquete"]= pd.to_numeric(df["paquete"],errors='coerce')
df["paquete"]= df["paquete"].fillna(0)
Once I changed the data type, proceed to create a tensors (in this case 2 colums from the pandas dataframe) then stacked those tensors:
import torch
device = 'cuda'
A= torch.cuda.IntTensor(df['bytes'].values,device=device)
B=torch.cuda.IntTensor(df['paquete'].values,device=device)
prueba=[A,B]
X = (torch.stack(prueba,dim=1))
Finally I tried to create the clusters:
from balanced_kmeans import kmeans_equal
N=X.shape[0]
num_clusters=100
device='cuda'
cluster_size=N//num_clusters
choices,centers=kmeans_equal(X,num_clusters=num_clusters,cluster_size=cluster_size)
But unfourtanelly I got this error:
RuntimeError Traceback (most recent call last)
in ()
16 #print (W)
17
---> 18 choices, centers = kmeans_equal(X, num_clusters=num_clusters,cluster_size=cluster_size)
1 frames
/usr/local/lib/python3.6/dist-packages/balanced_kmeans/init.py in initialize(X, num_clusters)
16 indices = torch.empty(X.shape[:-1], device=X.device, dtype=torch.long)
17 for i in range(bs):
---> 18 indices = torch.randperm(num_samples, device=X.device)
19 initial_state = torch.gather(X, 1, indices.unsqueeze(-1).repeat(1, 1,X.shape[-1])).reshape(bs,num_clusters, -1, X.shape[-1]).mean(dim=-2)
20 return initial_state
RuntimeError: expand(torch.cuda.LongTensor{[20839]}, size=[]): the number of sizes provided (0) must be greater or equal to the number of dimensions in the tensor (1)
I don't know if this error is caused by the tensor X having a 1 dimention, and in the library the initialization function create a empty tensor with zero dimension (I took this code from initialize method in the init.py):
indices=torch.empty(X.shape[:-1], device=X.device, dtype=torch.long)
for i in range(bs):
indices[i] = torch.randperm(num_samples, device=X.device)
So Am I doing right? (I'm new using tensors) or there is a bug in the initialization function?
I already checked the solution from PyTorch: The number of sizes provided (0) must be greater or equal to the number of dimensions in the tensor (1) but doesn't work for me becauseI don't need a scalar value.
I want to train my BERT NER model on colab. But following error occurs
Code:
tr_logits = tr_logits.detach().cpu().numpy()
tr_label_ids = torch.masked_select(b_labels, (preds_mask == 1))
tr_batch_preds = np.argmax(tr_logits[preds_mask.squeeze()], axis=1)
tr_batch_labels = tr_label_ids.to(device).numpy()
tr_preds.extend(tr_batch_preds)
tr_labels.extend(tr_batch_labels)
Error:
Using TensorFlow backend.
Saved standardized data to ./data/en/combined/train_combined.txt.
Saved standardized data to ./data/en/combined/dev_combined.txt.
Saved standardized data to ./data/en/combined/test_combined.txt.
Constructed SentenceGetter with 25650 examples.
Constructed SentenceGetter with 8934 examples.
Loaded training and validation data into DataLoaders.
Initialized model and moved it to cuda.
Initialized optimizer and set hyperparameters.
Epoch: 0% 0/5 [00:00<?, ?it/s]Starting training loop.
Epoch: 0% 0/5 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/content/FYP_Presentation/python/main.py", line 102, in <module>
valid_dataloader,
File "/content/FYP_Presentation/python/utils/main_utils.py", line 431, in train_and_save_model
tr_batch_preds = torch.max(tr_logits[preds_mask.squeeze()], axis=1)
File "/usr/local/lib/python3.6/dist-packages/torch/tensor.py", line 412, in __array__
return self.numpy()
TypeError: can't convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
How would I solve this issue?
In the first line of your code, tr_logits = tr_logits.detach().cpu().numpy() already turns tr_logits into a numpy array. In the line that raises the error:
tr_batch_preds = torch.max(tr_logits[preds_mask.squeeze()], axis=1)
the first thing for the program to do is to evaluate tr_logits[preds_mask.squeeze()]. Now that tr_logits is numpy array, its index preds_mask must also be numpy array. So the programs calls preds_mask.numpy() to change it to a numpy array. However, it is on GPU and hence the error.
I'd suggest using either numpy arrays or pytorch tensors all the way in one program, not alternatively .
Not able to reshape the image in mnist dataset using sklean
This is the starting portion of my code just load the data
some_digit = X[880]
some_digit_image = some_digit.reshape(28, 28)
ERROR PART
ValueError Traceback (most recent call last)
<ipython-input-15-4d618bdb57bc> in <module>
1 some_digit = X[880]
----> 2 some_digit_image = some_digit.reshape(28,28)
ValueError: cannot reshape array of size 64 into shape (28,28)
You can only reshape it into a 8, 8 array. 8x8=64
try:
some_digit = X[880]
some_digit_image = some_digit.reshape(8, 8)
I am classifying images into 30 different classes using CNN and on running the CNN ,I am getting the following traceback-
Traceback (most recent call last):
File "thapar.py", line 165, in <module>
Y = np_utils.to_categorical(labels, num_classes)
File "/home/rishav/anaconda3/envs/tensorflow/lib/python3.6/site-packages/keras/utils/np_utils.py", line 31, in to_categorical
categorical[np.arange(n), y] = 1
IndexError: index 10 is out of bounds for axis 1 with size 10
Python 3.6
Keras 2.1.5 using tensorflow backend
Ubuntu 16.04
I have explored other similar questions but unable to find the similarity in issue.
How to resolve this issue?
Edit:
num_classes = 30
num_of_samples = img_data.shape[0]
labels = np.ones((num_of_samples,),dtype='int64')
labels[0:20]=0
labels[20:40]=1
labels[40:60]=2
labels[60:80]=3
labels[80:100]=4
labels[100:120]=5
labels[120:140]=6
labels[140:160]=7
labels[160:180]=8
labels[180:200]=9
labels[200:220]=10
labels[220:240]=11
labels[240:260]=12
labels[260:280]=13
labels[280:300]=14
labels[300:320]=15
labels[320:340]=16
labels[340:360]=17
labels[360:380]=18
labels[380:400]=19
labels[400:420]=20
labels[420:440]=21
labels[440:460]=22
labels[460:480]=23
labels[480:500]=24
labels[500:520]=25
labels[520:540]=26
labels[540:560]=27
labels[560:580]=28
labels[580:600]=29
names =
['bent','bowl','claw','fist','good','thumbs_up','A','B','C','D','E','F','G','H','I','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y']
Y = np_utils.to_categorical(labels, num_classes)
Recently, I have faced the same problem, then I got to know it was giving error with some images only, I remove those images and it worked perfectly for me then.