Deep Reinforcement Learning: The Actor-Critic Method

that irritating hovering drone from ? The one that learned to descend toward the platform, pass through it, and then just… hang out below it forever? Yeah, me too. I spent an entire afternoon watching it hover there, accumulating negative rewards like a slow-motion crash, and I couldn’t even be mad because technically it was doing exactly what I told it to do.

Let’s dive in. Grab some coffee, you’re going to need it. All the code can be found in my repository on my github.

What’s Actor-Critic?

REINFORCE had one basic drawback: we needed to wait. Look ahead to the drone to crash. Look ahead to the episode to finish. Wait to compute the complete return. Then, and solely then, may we replace the coverage. One studying sign per episode. For a 150-step trajectory, that’s one replace after watching 150 actions play out.

I ran REINFORCE for 1200 iterations (6 hours on my machine) to hit 55% success fee. And the entire time I saved pondering: this feels wasteful. Why can’t I be taught throughout the episode?

Actor-Critic fixes this with a easy thought: prepare a second neural community (the “critic”) to estimate future returns for any state. Then use these estimates to replace the coverage after each single step. No extra ready for episodes to complete. Simply steady studying because the drone flies.

The end result? 68% success fee in 600 iterations (3 hours). Half the time. Higher efficiency. Identical {hardware}.

The way it works: Two networks collaborate in real-time.

The Actor (π(a|s)): Identical coverage community from REINFORCE. Takes the present state and outputs motion possibilities. That is the community that truly controls the drone.

The Critic (V(s)): New community. Takes the present state and estimates “how good is that this state?” It outputs a single worth representing anticipated future rewards. Not tied to any particular motion, simply evaluates states.

Right here’s the intelligent half: the critic gives instant suggestions. The actor takes an motion, the surroundings updates, and the critic instantly evaluates whether or not that moved us to a greater or worse state. The actor learns from this sign and adjusts. The critic concurrently learns to make higher predictions. Each networks enhance collectively as episodes unfold.

In code, they appear like this:

class DroneGamerBoi(nn.Module):
    """The Actor: outputs motion possibilities"""
    def __init__(self, state_dim=15):
        tremendous().__init__()
        self.community = nn.Sequential(
            nn.Linear(state_dim, 128), nn.LayerNorm(128), nn.ReLU(),
            nn.Linear(128, 128), nn.LayerNorm(128), nn.ReLU(),
            nn.Linear(128, 64), nn.LayerNorm(64), nn.ReLU(),
            nn.Linear(64, 3),  # Three unbiased thrusters
            nn.Sigmoid()
        )

    def ahead(self, state):
        return self.community(state)  # Output: possibilities for every thruster


class DroneTeacherBoi(nn.Module):
    """The Critic: outputs state worth estimate"""
    def __init__(self, state_dim=15):
        tremendous().__init__()
        self.community = nn.Sequential(
            nn.Linear(state_dim, 128), nn.LayerNorm(128), nn.ReLU(),
            nn.Linear(128, 128), nn.LayerNorm(128), nn.ReLU(),
            nn.Linear(128, 64), nn.LayerNorm(64), nn.ReLU(),
            nn.Linear(64, 1)  # Single worth: V(s)
        )

    def ahead(self, state):
        return self.community(state)  # Output: scalar worth estimate

Discover the critic community is sort of similar to the actor, besides the ultimate layer outputs a single worth (how good is that this state?) as an alternative of motion possibilities.

The Bootstrapping Trick

Okay, right here’s the place it will get intelligent. In REINFORCE, we would have liked the complete return to replace the coverage:

[ G_t = r_t + gamma r_{t+1} + gamma^2 r_{t+2} + cdots + gamma^{T-t} r_T ]

We needed to wait till the episode ended to know all of the rewards. However what if… we didn’t? What if we simply estimated the long run utilizing our critic community?

As a substitute of computing the precise return, we estimate it:

[ G_t approx r_t + gamma V(s_{t+1}) ]

That is referred to as bootstrapping. The critic “bootstraps” its personal worth estimate to approximate the complete return. We use its prediction of “how good will the subsequent state be?” to estimate the return proper now.

Why does this assist?

Decrease variance. We’re not ready for the precise random sequence of future rewards. We’re utilizing an estimate based mostly on what we’ve realized about states normally. That is noisier than the bottom reality (the critic could be flawed!), but it surely’s much less noisy than any single episode final result.

On-line studying. We are able to replace instantly at each step. No want to complete the episode first. As quickly because the drone takes one motion, we all know the instant reward, and we are able to estimate what comes subsequent, so we are able to be taught.

Higher pattern effectivity. In REINFORCE with 6 parallel video games, every drone learns as soon as per episode completion. In Actor-Critic with 6 parallel video games, every drone learns at each step (about 150 steps per episode). That’s 150x extra studying indicators per episode!

After all, there’s a trade-off: we introduce bias. If our critic is flawed (and will probably be, particularly early in coaching), our agent learns from incorrect estimates. However the critic doesn’t should be excellent. It simply must be much less noisy than a single episode final result. Because the critic regularly improves, the actor learns from higher suggestions. They bootstrap one another upward. In follow, the variance discount is so highly effective that it’s price accepting the small bias.

TD Error: The New Benefit

Now we have to reply: how significantly better or worse was this motion than anticipated?

In REINFORCE, we had the benefit: precise return minus baseline. The baseline was a worldwide common. However we are able to do significantly better. As a substitute of a worldwide baseline, we use the critic’s state-specific estimate.

The TD (Temporal Distinction) error is our new benefit:

[ delta_t = r_t + gamma V(s_{t+1}) – V(s_t) ]

In plain phrases:

• (r_t + gamma V(s_{t+1})) = TD goal. The instant reward plus our estimate of the subsequent state’s worth.
• (V(s_t)) = Our prediction for the present state.
• (delta_t) = The distinction. Did we do higher or worse than anticipated?

If (delta_t > 0), we did higher than anticipated → reinforce that motion.

If (delta_t < 0), we did worse than anticipated → lower that motion’s chance.

If (delta_t approx 0), we had been spot on → motion was about common.

That is far more informative than REINFORCE’s world baseline. The sign is now state-specific. The drone in a difficult spin would possibly get -10 reward and that’s truly fairly good (normally will get -50 there). But when it’s hovering peacefully over the platform, -10 is horrible. The critic is aware of the distinction. The TD error captures that.

Right here’s how this flows by the coaching loop (simplified):

# 1. Take one motion in every parallel sport
motion = actor(state)
next_state, reward = env.step(motion)

# 2. Get worth estimates
value_current = critic(state)
value_next = critic(next_state)

# 3. Compute TD error (our benefit)
td_error = reward + gamma * value_next - value_current

# 4. Replace the critic: it ought to have predicted higher
#    The critic desires to reduce prediction error, so we use squared error.
#    The gradient then pushes the critic's predictions nearer to precise returns.
critic_loss = td_error ** 2
critic_loss.backward()
critic_optimizer.step()

# 5. Replace the actor: reinforce or discourage based mostly on TD error
#    (identical coverage gradient as REINFORCE, however with TD error as an alternative of returns)
actor_loss = -log_prob(motion) * td_error
actor_loss.backward()
actor_optimizer.step()

Discover we’re updating each networks per step, not per episode. That’s the net studying magic.

Yet another comparability to make this crystal clear:

The second is extra exact, extra informative, and arrives a lot sooner.

That is why Actor-Critic converged in 600 iterations on my machine whereas REINFORCE wanted 1200. Identical reward operate, identical surroundings, identical drone. However getting suggestions after each step as an alternative of each 150 steps? That’s a 150x info benefit per iteration.

The Three Bugs: A Debugging Odyssey

Alright, I’m about to inform you about three bugs that just about broke me. Not “oops, off-by-one error” damaged. I imply the form of damaged the place you stare at coaching curves for six hours, severely query whether or not you perceive backpropagation, debug your code 5 instances, after which spend one other two hours studying tutorial papers to persuade your self you’re not insane.

These bugs are sufficiently subtle that even skilled RL practitioners must watch out. The excellent news: when you perceive them, they turn into apparent. The unhealthy information: it’s important to perceive them first, and I realized the exhausting means.

Bug #1: The Transferring Goal Drawback

The Setup

I applied Actor-Critic precisely because it appeared logical. I’ve two networks. One predicts actions, one predicts values. Easy, proper? I wrote out the TD error computation:

# Compute worth estimates
values = critic(batch_data['states'])
next_values = critic(batch_data['next_states'])

# Compute TD targets and errors
td_targets = rewards + gamma * next_values * (1 - dones)
td_errors = td_targets - values

# Critic loss
critic_loss = (td_errors ** 2).imply()

# Backward cross
critic_loss.backward()

This seemed utterly affordable to me. We compute what we anticipated (values), we compute what we should always have gotten (td_targets), we measure the error, and we replace. Customary supervised studying stuff.

The Symptom: Nothing Works

I skilled for 200 iterations and the critic loss was… sitting round 500-1000 and never transferring. Not lowering, not rising, simply oscillating wildly like a sine wave. I checked my reward operate. Regarded positive. I checked the critic community. Customary structure, nothing bizarre. I checked the TD error values themselves. They had been bouncing round between -50 and +50, which appeared affordable given the reward scale.

However the loss refused to converge.

I spent two days on this. I added dropout, pondering perhaps overfitting. (Unsuitable drawback, didn’t assist.) I lowered the educational fee from 1e-3 to 1e-4, pondering perhaps the optimizer was overshooting. (Nope, simply realized slower whereas oscillating.) I checked if my surroundings was returning NaNs. (It wasn’t.) I even puzzled if PyTorch’s autograd had a bug. (Spoiler: PyTorch is okay, I used to be the bug.)

The Breakthrough

I used to be studying the Actor-Critic chapter in Sutton & Barto (once more, for the fifth time) when one thing caught my eye. The pseudocode had a line about “computing the subsequent worth estimate.” And I believed: wait, after I compute next_values = critic(next_states), what occurs to these gradients throughout backprop?

After which my mind went click on. Oh no. The goal is transferring as we attempt to optimize towards it. That is referred to as the transferring goal drawback.

Why This Breaks The whole lot

After we compute next_values = critic(next_states) with out detaching, PyTorch’s autograd flows gradients by BOTH V(s) and V(s’). Meaning we’re updating the prediction AND the goal concurrently—the critic chases a goal that strikes each time it updates. The gradient turns into:

[ frac{partial L}{partial theta} = 2 cdot (r + gamma V(s’) – V(s)) cdot left( gamma frac{partial V(s’)}{partial theta} – frac{partial V(s)}{partial theta} right) ]

That γ · ∂V(s')/∂θ time period is the issue—we’re telling the critic to vary the goal, not simply the prediction. The loss oscillates without end.

The Repair (Lastly)

I wanted to deal with the TD goal as a hard and fast fixed. In PyTorch, which means detaching the gradients:

# ✅ CORRECT
values = critic(batch_data['states'])

with torch.no_grad():  # CRITICAL LINE
    next_values = critic(batch_data['next_states'])

td_targets = rewards + gamma * next_values * (1 - dones)
td_errors = td_targets - values

critic_loss = (td_errors ** 2).imply()
critic_loss.backward()

The torch.no_grad() context supervisor says: “Compute these subsequent values, however don’t bear in mind the way you computed them. For gradient functions, deal with this as a relentless.” Now in the course of the backward cross:

[ frac{partial L}{partial theta} = 2 cdot (r + gamma V(s’) – V(s)) cdot left( – frac{partial V(s)}{partial theta} right) ]

That problematic time period is gone! Now we’re solely updating V(s), the prediction, to match the fastened goal r + γV(s’). That is precisely what we would like.

The TD goal turns into what it ought to be: a fastened label, like the bottom reality in supervised studying. We’re now not attempting to hit a transferring goal. We’re simply attempting to foretell one thing steady.

I modified precisely one line. The critic loss went from oscillating chaotically round 500-1000 to lowering easily: 500 → 250 → 100 → 35 → 8 over 200 iterations. This bug is insidious as a result of the code seems to be utterly affordable—however all the time detach your TD targets.

Bug #2: Gamma Too Low (Invisible Rewards)

The Setup

Alright, Bug #1 was refined. This bug is embarrassingly apparent on reflection. However you realize what? Typically the obvious errors are the best to overlook since you don’t count on the issue to be that easy.

I fastened the transferring goal bug and all of a sudden the critic loss began converging. Implausible! I felt like an actual engineer for a second there. However then I ran the agent for a full coaching iteration and… nothing. Completely nothing improved. The drone would take just a few random strikes after which instantly crash into the bottom or fly off the display screen. No studying. No enchancment. No indicators of life.

Truly, wait. The critic was studying. The loss was happening. However the drone wasn’t getting higher. That appeared backwards. Why would the critic be taught to foretell values if the agent wasn’t studying something from these values?

The Discovery

I printed the TD targets and so they had been all unfavorable—starting from -5 to -30. No signal of the +500 touchdown reward. Then I did the mathematics: with 150-step episodes and gamma=0.90:

[ 500 times 0.90^{150} approx 0.00000006 ]

The touchdown reward had been discounted into oblivion. The agent realized to crash instantly as a result of attempting to land was actually invisible to the worth operate.

The low cost issue γ controls the efficient horizon (≈ 1/(1-γ)). With gamma = 0.90, that’s solely 10 steps—means too quick for 100-300 step episodes.

The repair: change gamma from 0.90 to 0.99.

The Influence

I modified gamma from 0.90 to 0.99. Identical community, identical rewards, identical every little thing else.

End result: Iteration 5, the drone moved towards the platform. Iteration 50, it slowed when approaching. Iteration 100, first touchdown. By iteration 600, 68% success fee.

One parameter change, utterly completely different agent conduct. The terminal reward went from invisible to crystal clear. All the time verify: efficient horizon (1/(1-γ)) ought to match your episode size.

Bug #3: Reward Exploits (The Arms Race)

At this level, I’d fastened each the transferring goal drawback and the gamma concern. My agent was truly studying! It approached the platform, slowed down sometimes, and even landed typically. I used to be genuinely excited. Then I began watching the failures extra fastidiously, and one thing bizarre occurred.

After fixing bugs #1 and #2, the agent realized two new exploits:

Zoom-past: Speed up towards the platform at most pace, overshoot, crash distant. Internet reward: -140 (strategy rewards +60, crash penalty -200). Higher than crashing instantly (-300), however not touchdown.

Hovering: Get near the platform and vibrate in place with tiny actions (pace 0.01-0.02) to farm strategy rewards indefinitely whereas avoiding crash penalties.

Why This Occurs: The Basic Drawback

Right here’s the factor that bothered me: My reward operate may solely see the present state, not the trajectory.

The reward operate is r(s', a): given the subsequent state and the motion I simply took, compute my reward. It has no reminiscence. It may possibly’t inform the distinction between:

1. A drone making real progress towards touchdown: approaching from above with managed, purposeful descent
2. A drone farming the reward construction: hovering with meaningless micro-movements

Each situations may need:

• distance_to_platform < 0.3 (shut to focus on)
• pace > 0 (technically transferring)
• velocity_alignment > 0 (pointed in the correct path)

The agent isn’t dumb. It’s doing precisely what I advised it to do—maximize the scalar rewards I’m feeding it. The issue is that the rewards don’t truly encode touchdown, they encode proximity and motion. And proximity with out touchdown is exploitable.

That is the core perception of reward hacking: the agent will discover loopholes in your reward specification, not as a result of it’s intelligent, however since you under-specified the duty.

The Repair: Reward State Transitions, Not Snapshots

The repair: reward based mostly on state transitions r(s, s'), not simply present state r(s'). As a substitute of asking “Is distance < 0.3?”, ask “Did we get nearer (distance_delta > 0) AND transfer quick sufficient to imply it (pace ≥ 0.15)?”

def calc_reward(state: DroneState, prev_state: DroneState = None):
    if prev_state will not be None:
        distance_delta = prev_state.distance_to_platform - state.distance_to_platform
        pace = state.pace
        velocity_toward_platform = calculate_alignment(state)  # cosine similarity

        MIN_MEANINGFUL_SPEED = 0.15

        if pace >= MIN_MEANINGFUL_SPEED and velocity_toward_platform > 0.1:
            speed_multiplier = 1.0 + pace * 2.0
            rewards['approach'] = distance_delta * 15.0 * speed_multiplier
        elif pace < 0.05:
            rewards['hovering_penalty'] = -1.0

Key modifications: (1) Reward distance_delta (progress), not proximity, (2) MIN_SPEED threshold blocks hovering, (3) Velocity multiplier encourages decisive motion.

To make use of this, observe prev_state in your coaching loop and cross it to calc_reward(next_state, prev_state).

90% of RL is reward engineering. The opposite 90% is debugging your reward engineering. Rewards are a specification of the target, and the agent will discover each loophole.

Primary Actor-Critic Outcomes

I’ve to confess, after I fastened the third bug (that velocity-magnitude-weighted reward operate) and launched a contemporary coaching run with all three fixes in place, I used to be skeptical. I’d spent a lot time chasing my tail with these algorithms that I half anticipated Actor-Critic to hit some new, inventive failure mode I hadn’t anticipated. However one thing stunning occurred: it simply… labored.

And I imply actually labored. Higher than REINFORCE, in actual fact—noticeably higher. After lots of of hours debugging REINFORCE’s reward hacking, I used to be anticipating Actor-Critic to at the least match its efficiency. As a substitute, it blew previous it.

Why This Beats REINFORCE (And Why That Issues):

Actor-Critic’s on-line updates create a suggestions loop that REINFORCE can’t match. Each single step, the critic whispers within the actor’s ear: “Hey, that state is nice” or “That state is unhealthy.” It’s not a worldwide baseline like REINFORCE makes use of. It’s state-specific analysis that will get higher and higher because the critic learns.

That is why the convergence is 2x sooner. That is why the ultimate efficiency is 13% higher. That is why the educational curves are so clear.

And all of it hinged on three issues: detaching the TD goal, utilizing the correct low cost issue, and monitoring state transitions within the reward operate. No new algorithm tips wanted. Simply right implementation.

What’s Subsequent: Pushing Past Actor-Critic

With Actor-Critic working alright, you will have seen that the coverage is persistently touchdown the drone on the left facet of the platform, and likewise the actions are barely jittery. To unravel this, I’m engaged on convering Proximal Coverage Optimization (PPO), which is meant to assist with this by “making the educational course of extra steady”. The great factor is, this methodology has utilized by the researchers at OpenAI to coach their flagship “GPT” fashions.

References

Foundational RL Papers

1. Sutton, R. S., & Barto, A. G. (2018). Reinforcement Studying: An Introduction (2nd ed.). MIT Press.

Actor-Critic Strategies

2. Konda, V. R., & Tsitsiklis, J. N. (2000). “Actor-Critic Algorithms.” SIAM Journal on Management and Optimization, 42(4), 1143-1166.
   • Theoretical foundations of Actor-Critic with convergence proofs
3. Mnih, V., Badia, A. P., Mirza, M., et al. (2016). “Asynchronous Strategies for Deep Reinforcement Studying.” Worldwide Convention on Machine Studying.

Temporal Distinction Studying

4. Sutton, R. S. (1988). “Studying to Predict by the Strategies of Temporal Variations.” Machine Studying, 3(1), 9-44.
   • Authentic TD studying paper

Earlier Posts in This Sequence

5. Jumle, V. (2025). “Deep Reinforcement Studying: 0 to 100 – Coverage Gradients (REINFORCE).”

Code Repository & Implementation

6. Jumle, V. (2025). “Reinforcement Studying 101: Supply Drone Touchdown.”

All photographs on this article are both AI-generated (utilizing Gemini or Sora), personally made by me, or screenshots & plots that I made, until specified in any other case.