Imitation learning

Imitation learning

• Data: Given trajectories collected by an expert demonstrations .
• Goal: Learn a policy that imitates the expert behavior.

Behavior cloning (BC)

The most primitive algorithm for imitation learning is behavior cloning. Idea is to do regress to expert’s actions:

1. Given expert demonstrations: .
2. For deterministic policy, regress to expert’s actions with simple MSE loss (actions are just vectors representing desired next joint angles):
3. Deploy policy on robot.

Limitation

However, a small error in the predicted action can lead to drifting away from the expert distribution.

States visited by the expert and the policy are not the same. This is because the i.i.d. assumption common in supervised learning (that inputs are independently and identically distributed and independent of predicted labels i.e., predictions for different samples are independent!) does not hold for robot action. TL;DR: BC is sensitive to even a slight train-test distribution shift.

For rare states that require error recovery, producing these states can be expensive and dangerous for the experts (e.g., replicating a drifting car crashing onto the side of a road). For this reason, BC can make use of virtual data augmentation to work well.

For example, with navigation problems, we can attach multiple cameras at different views and have experts only demonstrate once. Then we label the actions for the other cameras (other than the “straightahead” camera) as “steer towards straight ahead” depending on the camera view. This way, with just one demonstration, we collected trajectories ( being the number of cameras) with a lot of corrective actions.

Data aggregation (DAgger)

To augment the states covered by the expert and cover the distributional shift, we simply make the expert and policy distribution converge by doing online rollouts during training.

1. Roll out the policy and collect states
2. Query/label expert actions at visited states i.e., what an expert would have done in these situations created by the policy?
3. Aggregate corrections with existing data:
4. Update policy

• 👌 Algorithm will converge
• 👌 Achieves instead of upper bound BC
• 👎 Hindsight labeling and expert querying is difficult

Paradox: BC works better if data has more mistakes and recoveries.

Aggegating online interventions (Human-gated DAgger, HA-DAgger)

Standard DAgger is expensive. We can move to a more efficient version that only requires expert intervention intermittently at rollout time when the policy makes a mistake.

1. Roll out the policy and collect states
2. Expert intervenes at time when policy makes a mistake.
3. Expert provides new partial demo
4. Aggregate new demos:
5. Update policy

• 👌 Let the humans take control
• 👎 How to detect when to intervene? Can a robot detects its own uncertainty and ask for human help when it’s about to make a mistake?

Non-Markovian behavior

The real world is non-Markovian, i.e., the next state depends on the current state and the action. Human operators’ actions also depend on their current mood, experience, temperatement, etc. So when they encounter the same state, a human operator might take many different actions. This makes it difficult to learn a policy that imitates the expert’s actions. We call the operator’s mismatched POV from the robot when demo-ing privileged observation.

The first idea is to encode the history into the policy with some seq2seq architecture. However, this doesn’t always make things better. A common mode of failure with this is causal confusion - the learned policy often infers spurious correlations between the history and the action.

Multimodal behavior

Human operators might use different modes across trials -> expert inconsistency. Also many different ways, basically inifinite, to do the same task -> stochasticity.

The reason for this multimodal behavior can be explained by the MSE loss used in learning the deterministic BC policy. This produces a mean-seeking behavior (averaging of modes). For example, in a situation where it needs to avoid a tree by turning left or right, it learns the average behavior while minimizing the MSE loss, which results in the car running straight into the tree.

Thus we need to go further than regressing a single point to the expert’s action -> a policy that can represents multiple valid modes/peaks of actions.

Gaussian mixture model (GMM)

• More expressive than a single Gaussian
• Need to predefine number of Gaussians

Initialize these weights, means, and covariances randomly and learn them with gradient descent. The nice thing is that you can model any distribution with a Gaussian mixture model.

Autogressive discretization

With GMM, the number of Gaussians needs to be pre-defined. For a humanoid, this might be a 1000 or more, where things get a little bit tricky. To overcome this while also able to represent multimodal distributions, we turn to autogressive discretization. Instead of representing the action space as continuous, we turn it into a grid of bins and treat action prediction as a classification problem.

Discretization is great for representing multimodal distributions, such as drone or car navigation, but it is impractical to scale to higher dimensions because the number of bins increase exponentially with higher dimensions.

We can solve this scaling problem by doing per-dimension autogressive discretization, which is linear in the number of bins. And you can make use of sequence models like autogressive transformers to decompose this action distribution using the chain rule.

Diffusion

Models complex distributions over continuous variables, replacing images with robot actions.

Latent variable models

• Can represent any distribution given large enough network
• Output still Gaussian, but receives addditional input
• Popular: conditional VAEs

The prior acts as a seed for the network, which results in the network predicting different modes. We train the model to relate the prior to the desired mode.

Caution

Latent variable models are different from task conditioning in that in task conditioning, the input task id specifies which specific behavior the robot should execute i.e., the goal. While in latent variable models, the input prior represents which style/model within that goal you want it to represent.