TensorFlow 2.0 : ValueError - No Gradients Provided (After Modifying DDPG Actor) - python-3.x

Question

Background
I'm currently trying to implement a DDPG framework to control a simple car agent. At first, the car agent would only need to learn how to reach the end of a straight path as quickly as possible by adjusting its acceleration. This task was simple enough, so I decided to introduce an additional steering action as well. I updated my observation and action spaces accordingly.
The lines below are the for loop that runs each episode:
for i in range(episodes):
observation = env.reset()
done = False
score = 0
while not done:
action = agent.choose_action(observation, evaluate)
observation_, reward, done, info = env.step(action)
score += reward
agent.remember(observation, action, reward, observation_, done)
if not load_checkpoint:
agent.learn()
observation = observation_
The lines below are my choose_action and learn functions:
def choose_action(self, observation, evaluate=False):
state = tf.convert_to_tensor([observation], dtype=tf.float32)
actions = self.actor(state)
if not evaluate:
actions += tf.random.normal(shape=[self.n_actions],
mean=0.0, stddev=self.noise)
actions = tf.clip_by_value(actions, self.min_action, self.max_action)
return actions[0]
def learn(self):
if self.memory.mem_cntr < self.batch_size:
return
state, action, reward, new_state, done = \
self.memory.sample_buffer(self.batch_size)
states = tf.convert_to_tensor(state, dtype=tf.float32)
states_ = tf.convert_to_tensor(new_state, dtype=tf.float32)
rewards = tf.convert_to_tensor(reward, dtype=tf.float32)
actions = tf.convert_to_tensor(action, dtype=tf.float32)
with tf.GradientTape() as tape:
target_actions = self.target_actor(states_)
critic_value_ = tf.squeeze(self.target_critic(
states_, target_actions), 1)
critic_value = tf.squeeze(self.critic(states, actions), 1)
target = reward + self.gamma*critic_value_*(1-done)
critic_loss = keras.losses.MSE(target, critic_value)
critic_network_gradient = tape.gradient(critic_loss,
self.critic.trainable_variables)
self.critic.optimizer.apply_gradients(zip(
critic_network_gradient, self.critic.trainable_variables))
with tf.GradientTape() as tape:
new_policy_actions = self.actor(states)
actor_loss = -self.critic(states, new_policy_actions)
actor_loss = tf.math.reduce_mean(actor_loss)
actor_network_gradient = tape.gradient(actor_loss,
self.actor.trainable_variables)
self.actor.optimizer.apply_gradients(zip(
actor_network_gradient, self.actor.trainable_variables))
self.update_network_parameters()
And finally, my ActorNetwork is as follows:
class ActorNetwork(keras.Model):
def __init__(self, fc1_dims=512, fc2_dims=512, n_actions=2, name='actor',
chkpt_dir='tmp/ddpg'):
super(ActorNetwork, self).__init__()
self.fc1_dims = fc1_dims
self.fc2_dims = fc2_dims
self.n_actions = n_actions
self.model_name = name
self.checkpoint_dir = chkpt_dir
self.checkpoint_file = os.path.join(self.checkpoint_dir,
self.model_name+'_ddpg.h5')
self.fc1 = Dense(self.fc1_dims, activation='relu')
self.fc2 = Dense(self.fc2_dims, activation='relu')
self.mu = Dense(self.n_actions, activation='tanh')
def call(self, state):
prob = self.fc1(state)
prob = self.fc2(prob)
mu = self.mu(prob) * 3.5
return mu
Note: The code I'm working with is just building off of the code from this tutorial
The Problem
Up until now, I hadn't faced any issues with the code but I did want to adjust the maximum/minimum values of my actions. When I was only considering the acceleration action, I simply multiplied mu by 3.5. However, I wanted the steering actions to exist within a range of -30 to 30 degrees, but I couldn't just multiply mu as I had before. To try to adjust the desired steering range, I made the following (not so elegant) changes to my ActorNetwork
def call(self, state):
prob = self.fc1(state)
prob = self.fc2(prob)
mu = self.mu(prob)# * 3.5
mu_ = []
mu_l = mu.numpy().tolist()
for i, elem1 in enumerate(mu_l):
temp_ = []
for j, elem2 in enumerate(elem1):
if j-1 == 0:
temp_.append(float(elem2 * 3.5))
else:
temp_.append(float(elem2 * math.radians(30)))
mu_.append(temp_)
mu = tf.convert_to_tensor(mu_, dtype=tf.float32)
return mu
The new lines that I added were meant to:
Convert the mu tensor into a list
Iterate through the elements in the mu list (mu_l) and if a value had an index of 0 (acceleration) then multiply by 3.5; otherwise, multiply the value at index=1 (steering) by the radians conversion of 30 degrees.
Append each adjusted value into a new list (mu_)
Set mu to be equal to a tensor conversion of mu_
It was at this point that I ran into the following error:
ValueError: No gradients provided for any variable: ['actor_network/dense/kernel:0', 'actor_network/dense/bias:0', 'actor_network/dense_1/kernel:0', 'actor_network/dense_1/bias:0', 'actor_network/dense_2/kernel:0', 'actor_network/dense_2/bias:0'].
I have tried to find solutions provided within StackOverflow and from outside sources (e.g. including watch, checking to make sure that I am using model() instead of model.predict() while in GradientTape(), making sure I'm not performing calculations outside of the Tape context) but I haven't had any luck resolving the issue. I suspect that my issue is similar to the one presented in this previous post but I'm not sure how to diagnose whether my problem stems from also overwritting mu with a tensor. Does anyone have any insight regarding this problem?

Answer 1

The issue has been resolved thanks to some simple but helpful advice I received on Reddit. I was disrupting the tracking of my variables by making changes using my custom for-loop. I should have used a TensorFlow function instead. The following changes fixed the problem for me:
def call(self, state):
prob = self.fc1(state)
prob = self.fc2(prob)
mu = self.mu(prob)
mult = tf.convert_to_tensor([3.5, math.radians(30)], dtype=tf.float32)
mu = tf.math.multiply(mu, mult)
return mu