Overview for PAL Kangaroo Velocity Task

---

1. System Architecture

┌─────────────────────────────────────────────┐
│              mjlab (Framework)              │
│  - Environment Management                   │
│  - Observation / Action / Reward Manager    │
│  - Domain Randomization                     │
│  - RSL-RL Integration                       │
└─────────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────────┐
│          MuJoCo Warp (GPU Backend)          │
│  - GPU-accelerated Physics                  │
│  - 8192 parallel environments on GPU        │
└─────────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────────┐
│              MuJoCo (Physics)               │
│  - Contact Dynamics                         │
│  - Actuator Models                          │
│  - Constraint Solver                        │
└─────────────────────────────────────────────┘

---

2. Observation Space

Policy Group (Shape: 86)

Term	Dim	Description	Noise
base_ang_vel	3	Root-link angular velocity	±0.2
joint_pos	26	Joint positions relative to default pose	±0.01
joint_vel	26	Joint velocities	±1.5
actions	22	Last actions (22 actuators)	–
command	3	[vx_cmd, vy_cmd, wz_cmd]	–
imu_projected_gravity	3	Gravity vector from IMU	±0.5
base_lin_acc	3	Linear acceleration from IMU	±0.5

base_lin_vel and projected_gravity are disabled in the policy.

Critic Group (Shape: 104)

Policy terms (without noise) + privileged information:

Term	Dim	Critic only
base_lin_vel	3	✓
projected_gravity	3	✓
foot_height	2	✓
foot_air_time	2	✓
foot_contact	2	✓
foot_contact_forces	6	✓

The critic receives no noise – it always sees the true values.

Observation Normalization

Both networks normalize their observations independently using running statistics:

ToDo: Running statistics could be problematic. We may want to get rid of it!

mean_t = β × mean_{t-1} + (1-β) × x_t
var_t  = β × var_{t-1}  + (1-β) × (x_t - mean_t)²

normalized_obs = (obs - mean) / sqrt(var + 1e-8)

• actor_normalizer: EmpiricalNormalization(86)
• critic_normalizer: EmpiricalNormalization(104)

---

3. Network Architecture

Network	Architecture	Parameters	Activation
Policy	86→512→256→128→22	~211K	ELU
Critic	104→512→256→128→1	~218K	ELU

---

4. Action Pipeline

Todo: check whether clip() is a tanh?

NN Output  ∈ (−∞, +∞)
    ↓  clip(−1, +1)
a_clipped  ∈ [−1, +1]
    ↓  × action_scale
motor_target
    ↓  PD-Regler
τ = kp × (motor_target − q) + kd × (0 − q̇)
    ↓  clip(−effort_limit, +effort_limit)
τ_actual → MuJoCo

action_scale derivation (arm/torso motors):

action_scale = 0.25 × effort_limit / stiffness

---

5. Actuators – Motor Types

Global constants

NATURAL_FREQ  = 62.83 rad/s  (10 Hz)
DAMPING_RATIO = 2.0
FACTOR        = 0.05

S_PLUS Motor (Arm Joint 1–2, Pelvis)

gear_ratio    = 121
motor_inertia = 1.728×10⁻⁵ kg·m²
armature      = FACTOR × motor_inertia × gear_ratio² = 0.01264
stiffness     = armature × NATURAL_FREQ²             ≈ 50.0
damping       = 2 × DAMPING_RATIO × armature × NATURAL_FREQ ≈ 1.593

S_MINUS Motor (Arm Joint 3–4)

gear_ratio    = 101
motor_inertia = 1.3×10⁻⁵ kg·m²
armature      = 0.05 × 1.3e-5 × 101² = 0.00656
stiffness     ≈ 25.9
damping       ≈ 0.825

LEG Motor (direct parameterization, no gearbox model)

armature = 0.01 (fest)
damping  = 2 × DAMPING_RATIO × stiffness / NATURAL_FREQ

---

6. All 22 Actuators

Leg joints (12 joints, 6 per leg)

Joint	kp	kd	Effort	Armature	Action Scale
leg_*/1_joint (hip abduction)	100.0	25.133	80 Nm	0.002540	0.200
leg_*/2_joint (hip pitch)	100.0	25.133	230 Nm	0.002540	0.575
leg_*/3_joint (hip roll)	100.0	25.133	139 Nm	0.002540	0.348
leg_*/4_joint (knee)	30.0	13.746	140 Nm	0.000762	1.167
leg_*/5_joint (ankle)	30.0	13.746	82 Nm	0.000762	0.683
leg_*/length_joint (telescopic)	1600.0	401.070	1100 Nm	0.040640	0.172

Arm joints (8 joints, 4 per arm)

Joint	Motor	kp	kd	Effort	Armature	Action Scale
arm_*/1_joint	S_PLUS	578.962	288.506	43 Nm	0.014706	0.019
arm_*/2_joint	S_PLUS	578.962	288.506	43 Nm	0.014706	0.019
arm_*/3_joint	S_MINUS	399.425	199.174	43 Nm	0.010143	0.027
arm_*/4_joint	S_MINUS	399.425	199.174	43 Nm	0.010143	0.027

Torso / pelvis (2 joints)

Joint	Motor	kp	kd	Effort	Armature	Action Scale
pelvis_1_joint	S_PLUS	578.962	288.506	100 Nm	0.014706	0.043
pelvis_2_joint	S_PLUS	578.962	288.506	100 Nm	0.014706	0.043

---

7. Joints – Overall

TOTAL: 26 joints
    ├─ 22 actuated joints  (controllable by the policy)
    └─  4 passive joints   (visible to the policy, but NOT controllable)

Passive joints:

• leg_left_femur_joint
• leg_left_knee_joint
• leg_right_femur_joint
• leg_right_knee_joint

The policy sees joint_pos and joint_vel for all 26 joints and therefore learns the kinematic coupling between active and passive joints.

8. Command Ranges & Curriculum

Stage	Iterations	lin_vel_x	ang_vel_z
0	Start	[−1.0, 1.0] m/s	[−0.5, 0.5] rad/s
1	5 000 × 24	[−1.5, 2.0] m/s	[−0.7, 0.7] rad/s
2	10 000 × 24	[−2.0, 3.0] m/s	–

---

9. Reward Structure

Positive Rewards

Term	Weight	Formula
track_linear_velocity	+2.0	exp(−‖v − v_cmd‖² / (2×0.5²))
track_angular_velocity	+2.0	exp(−‖ω_z − ω_cmd‖² / (2×0.707²))
upright	+1.0	exp(−(g_x² + g_y²) / 0.2)
default_pose	+1.0	exp(−‖q − q_default‖² / (2σ²))
air_time	+0.25	Rewards air time of 0.05–0.5 s (only for v_cmd > 0.5 m/s)

Negative Costs (Penalties)

Term	Weight	Description
joint_limits	−1.0	Approaching soft limits (90% of hard limits)
action_rate_l2	−0.1	‖a_t − a_{t-1}‖² – smooth actions
foot_clearance	−2.0	Feet scraping the ground (target: 0.1 m height)
foot_swing_height	−0.25	Swing motion too low
foot_slip	−0.1	Slipping during ground contact
soft_landing	−1e-5	Hard landings (high contact forces)
self_collisions	−1.0	Self-collisions in the base_link subtree
body_ang_vel	−0.05	ω_x² + ω_y² (yaw is not penalized)
angular_momentum	−0.02	‖L‖² (total angular momentum)
convex_hull_joint_limits_hip	−10.0	Hip (pitch+roll) outside convex hull
convex_hull_joint_limits_ankle	−10.0	Ankle (pitch+roll) outside convex hull

Adaptive Rewards (velocity-dependent)

cmd_vel	Active additional rewards
= 0.0	Only tracking + upright + tight pose (standing)
0.05–0.5 m/s	+ foot_clearance, foot_slip, foot_swing
0.5–1.5 m/s	+ air_time
> 1.5 m/s	Looser pose tolerances (running)

Pose Tolerances (velocity-dependent)

Mode	Condition	Tolerance σ
Standing	v_cmd < 0.05 m/s	all joints: 0.05 (tight)
Walking	0.05 ≤ v < 1.5 m/s	legs: 0.1–0.3, torso: 0.08–0.2
Running	v ≥ 1.5 m/s	legs: 0.15–0.5, torso: 0.08–0.3

---

10. Termination Conditions

Condition	Criterion
time_out	1000 control steps = 20 seconds (timestep: 0.002 s, decimation: 10)
fell_over	Tilt > 70°: \|arccos(−g_z)\| > 1.222 rad
illegal_contact	Femur or knee touches the ground → immediate termination
NaN/Inf	Invalid physics state → automatic

11. Domain Randomization

Startup (once per episode)

ToDo: Is this once per epsiode or once per env?

Event	Field	Range
foot_friction	geom_friction	[0.3, 1.2]
joint_friction	dof_frictionloss	±0.008 Nm
mass	body_mass	±0.2 kg (pelvis_2_link)
base_com	body_ipos	±0.025 m (XY), ±0.03 m (Z)
encoder_bias	all joints except leg_length	±0.015 rad (≈ ±0.86°)
leg_length_encoder_bias	leg_*/length_joint	±0.005 rad (≈ ±0.29°)

Reset (at every episode start)

Event	Range
reset_base	Position: ±0.5 m (XY), yaw: [−π, π]
reset_robot_joints	Default pose, velocity = 0

Interval (every 1–3 seconds during the episode)

Event	Linear	Angular
push_robot	±0.5 m/s (X, Y, Z)	±0.52 rad/s (roll/pitch), ±0.78 rad/s (yaw)

push_robot is disabled in play mode.

Batch Parallelization

model_field[environment_id, entity_id, axis]

8192 environments run in parallel on the GPU, each with independently sampled parameters.

---

12. PPO Training

Hyperparameters

Parameter	Value
learning_rate	1e-3
schedule	adaptive (KL-based)
clip_param (ε)	0.2
num_learning_epochs	5
num_mini_batches	4
gamma (discount)	0.99
lam (GAE λ)	0.95
value_loss_coef	1.0
entropy_coef	0.01
max_grad_norm	1.0
desired_kl	0.01
num_steps_per_env	24
num_envs	8192
max_iterations	30 000

Training Scale

Steps per Update = 24 × 8192        = 196 608
Mini-Batch Size  = 196608 / 4       =  49 152
Total Steps      = 30000 × 196608   ≈ 5.9 Mrd.

PPO Loss

L(θ) = L^CLIP − value_loss_coef × L^VF + entropy_coef × L^ENT
     = L^CLIP − 1.0 × L^VF + 0.01 × L^ENT

Policy Loss (Clipped Surrogate):

r_t(θ) = π_θ(a|s) / π_θ_old(a|s)
L^CLIP  = −E[min(r_t × A_t, clip(r_t, 1−ε, 1+ε) × A_t)]

Value Loss:

V_clipped = V_old + clip(V_θ − V_old, −ε, +ε)
L^VF      = E[max((V_θ − R_t)², (V_clipped − R_t)²)]

Entropy Bonus:

L^ENT = −E[H(π_θ)] = −E[Σ π_θ(a|s) × log(π_θ(a|s))]

Generalized Advantage Estimation (GAE)

δ_t = r_t + γ × V(s_{t+1}) × (1 − done) − V(s_t)
A_t = Σ_{l=0}^∞ (γλ)^l × δ_{t+l}
R_t = A_t + V(s_t)

λ	Property
0	Low variance, high bias (1-step TD)
1	High variance, low bias (Monte Carlo)
0.95	Good compromise

Adaptive Learning Rate

if   KL > 2 × desired_kl:   lr × 0.5   (decrease)
elif KL < desired_kl / 2:   lr × 2.0   (increase)

lr = clip(lr, 1e-5, 1e-3)