pytorch , changing learning rate during training - pytorch

Question

x=np.linspace(0,20,100)
g=1+0.2*np.exp(-0.1*(x-7)**2)
y=np.sin(g*x)
plt.plot(x,y)
plt.show()
x=torch.from_numpy(x)
y=torch.from_numpy(y)
x=x.reshape((100,1))
y=y.reshape((100,1))
MM=nn.Sequential()
MM.add_module('L1',nn.Linear(1,128))
MM.add_module('R1',nn.ReLU())
MM.add_module('L2',nn.Linear(128,128))
MM.add_module('R2',nn.ReLU())
MM.add_module('L3',nn.Linear(128,128))
MM.add_module('R3',nn.ReLU())
MM.add_module('L4',nn.Linear(128,128))
MM.add_module('R5',nn.ReLU())
MM.add_module('L5',nn.Linear(128,1))
MM.double()
L=nn.MSELoss()
lr=3e-05 ######
opt=torch.optim.Adam(MM.parameters(),lr) #########
Epo=[]
COST=[]
for epoch in range(8000):
opt.zero_grad()
err=L(torch.sin(MM(x)),y)
Epo.append(epoch)
COST.append(err)
err.backward()
if epoch%100==0:
print(err)
opt.step()
Epo=np.array(Epo)/1000.
COST=np.array(COST)
pred=torch.sin(MM(x)).detach().numpy()
Trans=MM(x).detach().numpy()
x=x.reshape((100))
pred=pred.reshape((100))
Trans=Trans.reshape((100))
fig = plt.figure(figsize=(10,10))
#ax = fig.gca(projection='3d')
ax = fig.add_subplot(2,2,1)
surf = ax.plot(x,y,'r')
#ax.plot_surface(x_dat,y_dat,z_pred)
#ax.plot_wireframe(x_dat,y_dat,z_pred,linewidth=0.1)
fig.tight_layout()
#plt.show()
ax = fig.add_subplot(2,2,2)
surf = ax.plot(x,pred,'g')
fig.tight_layout()
ax = fig.add_subplot(2,2,3)
surff=ax.plot(Epo,COST,'y+')
plt.ylim(0,1100)
ax = fig.add_subplot(2,2,4)
surf = ax.plot(x,Trans,'b')
fig.tight_layout()
plt.show()
This is the original code 1.
For changing learning rate during training, I tried to move the position of 'opt' as
Epo=[]
COST=[]
for epoch in range(8000):
lr=3e-05 ######
opt=torch.optim.Adam(MM.parameters(),lr) #########
opt.zero_grad()
err=L(torch.sin(MM(x)),y)
Epo.append(epoch)
COST.append(err)
err.backward()
if epoch%100==0:
print(err)
opt.step()
This is code 2.
The code 2 also operate, but the result is quite different with code 1.
What is the difference and for changing learning rate during training(like lr=(1-epoch/10000 *0.99), what should I do?

Answer 1

You shouldn't move the optimizer definition into the training loop, because the optimizer keeps many other information related to training history, e.g in case of Adam there are running averages of gradients that are stored and updated dynamically in the optimizer's internal mechanism,...
So instanciating a new optimizer each iteration makes you lose this history track.
To update the learning rate dynamically there are lot of schedulers classes proposed in pytorch (exponential decay, cyclical decay, cosine annealing , ...). you can check them from the documentation for the full list of schedulers or you can implement your own if needed: https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
Example from the documentation: to decay the learning rate by multiplying it by 0.5 each 10 epochs you can use the StepLR scheduler as follows:
opt = torch.optim.Adam(MM.parameters(), lr)
scheduler = torch.optim.lr_scheduler.StepLR(opt, step_size=10, gamma=0.5)
And in your original code 1 you can do :
for epoch in range(8000):
opt.zero_grad()
err=L(torch.sin(MM(x)),y)
Epo.append(epoch)
COST.append(err)
err.backward()
if epoch%100==0:
print(err)
opt.step()
scheduler.step()
As I say you have many other type of lr schedulers so you can choose from the documentation or implement your own