Action Pipeline: From Policy to Motor Commands

Overview

This document describes the complete processing chain of an action coming from the policy until it ends up as a ctrl[] value in MuJoCo and drives the physics simulation there.

---

Pipeline Overview

manager_based_rl_env.py  (entry point)
    │
    ▼  action_manager.py
    │  ActionManager.process_action
    │  → abstract ActionTerm.process_actions()
    │    concrete class: JointPositionAction
    │
    ▼  actions.py
    │  BaseAction.process_actions
    │  a_scaled = a_raw * scale + offset
    │
    ▼  actions.py
    │  JointPositionAction.apply_actions
    │  q_target = a_scaled − encoder_bias
    │
    ▼  entity.py
    │  set_joint_position_target
    │  Actuator mapping: target IDs → ctrl IDs (reordering)
    │
    ▼  MuJoCo ctrl[]
    │  PD controller: τ = kp·(ctrl − q) − kv·q̇
    │
    ▼  qfrc_actuator  (generalized force in DOF space)
    │
    ▼  Dynamics
       τ_total = qfrc_actuator + qfrc_passive + qfrc_constraint − qfrc_bias
       q̈ = M⁻¹ · τ_total  →  Integration  →  q̇, q

---

Step 1 – Entry Point

File: manager_based_rl_env.py

The environment loop calls the ActionManager, which forwards the raw policy action.
ActionManager.process_action calls the abstract method ActionTerm.process_actions().
The concrete subclass is JointPositionAction (from actions.py).

---

Step 2 – Scaling & Offset

Files: action_manager.py → actions.py: BaseAction.process_actions

a_scaled = a_raw * scale + offset

• scale: joint-specific scaling factors — limit the effective range of the policy output
• offset: joint-specific rest position (e.g. 1.57 for a 90° initial pose)

The raw policy output typically lies in the range [−1, 1]. After scaling and offset, it lies in a mechanically meaningful joint-angle range.

In mjlab, the offset is by default the default position (qpos) of the respective joint.

---

Step 3 – Encoder Bias

File: actions.py: JointPositionAction.apply_actions

q_target = a_scaled − encoder_bias

The encoder bias compensates for mechanical zero-offsets of the real hardware. It is subtracted from the target position so that the controller references the actual joint position.

---

Step 4 – Actuator Mapping

File: entity.py: set_joint_position_target

ctrl[ctrl_ids] = q_target[target_ids]

The order of the joints in the policy output does not directly match the MuJoCo actuator order. The mapping assigns the values correctly:

• Target IDs: indices in the q_target tensor (policy order)
• Ctrl IDs: indices in the MuJoCo ctrl[] array (simulator order)

---

Step 5 – PD Controller in MuJoCo

τ = kp · (ctrl − q) − kv · q̇

XML parameters: kp, kv → internally in MuJoCo: actuator_gainprm, actuator_biasprm

MuJoCo rewrites this algebraically into affine form:

τ = kp·ctrl + b1·q + b2·q̇    with  b1 = −kp, b2 = −kv

Example (actuator 0, joint 11, DOF 16):

kp·ctrl  = 100.0  ×  0.03468  =   3.468
b1·q     = −100.0 ×  0.06821  =  −6.821
b2·q̇    = −6.366  × −0.05890  =   0.375
─────────────────────────────────────────
τ_actuator (computed)         =  −2.978
τ_actuator (MuJoCo)           =  −2.9779  ✓

---

Step 6 – Mapping into DOF Space

τ_dof = qfrc_actuator[dof]

qfrc_actuator is a vector in DOF coordinates — the generalized actuator force in joint space.

For joint actuators (<position joint="...">): J_act = 1
→ qfrc_actuator[dof] = τ_actuator (direct 1:1 mapping)

---

Step 7 – Overall Dynamics

τ_total = qfrc_actuator + qfrc_passive + qfrc_constraint − qfrc_bias
q̈      = M⁻¹ · τ_total
q̇_new  = q̇_old + q̈ · dt
q_new   = q_old + q̇_new · dt

Process:

1. Compute all generalized forces
2. Solve the dynamics: q̈ = M⁻¹ · τ_total
3. Integrate velocity
4. Integrate position

Term	Source
qfrc_actuator	PD controller (steps 4–5)
qfrc_passive	mj_passive(): springs, damping, friction
qfrc_constraint	KKT solver: contacts, limits, constraints
qfrc_bias	mj_rne() with q̈=0: gravity, Coriolis