Abstract:
The motor skills performed by human beings every day are still unmatched by artificial systems, despite the rapid advances of control and motion generation methods driven by deep learning.
The fields of biomechanics and computational motor control investigate these capabilities with a variety of different methods, among which predictive simulation has been a successful paradigm, allowing researchers to gather insight into human motor control as well as develop new methods for rehabilitation.
While existing control methods achieve impressive results by using simplified models or by neglecting aspects such as feedback, environment uncertainty and biologically plausible sensory inputs, a scalable control algorithm capable of generating closed-loop policies for high-dimensional musculoskeletal systems under a variety of conditions is still missing.
One likely candidate to produce such a controller is Reinforcement Learning~(RL). RL has been able to create robust controllers in the robot learning domain, performing diverse behaviors in simulation and the real-world by leveraging advances in algorithms and simulation engines.
Nevertheless, the high dimensionality and complex dynamics of musculoskeletal systems have limited its adoption in biomechanics.
From the perspective of bio-inspired robotics, although biological muscles exhibit properties that likely enhance movement control, it remains unclear which features are essential and which are only evolutionary artifacts, detrimental to movement generation.
This dissertation addresses these challenges through novel algorithmic developments and computational frameworks that enable scalable RL for complex biomechanical systems, while also investigating muscular actuation properties in robotic systems across both simulation and real-world settings.
In two computational studies, we could first develop a RL control algorithm capable of generating behaviors for high-dimensional musculoskeletal systems, before extending it with bio-inspired reward terms, making the resulting controller act close-to-natural, while being incredibly robust.
We found that overactuation is a key problem in musculoskeletal systems, which exarcerbates exploration issues in RL.
By leveraging a technique from the domain of self-organizing systems, we could propose a controller capable of generating effective exploration for arbitrary muscle-driven systems, while performing well in a wide range of environments.
Our proposed reward terms for natural walking are adaptive and produce close-to-natural motion in 4 different models without the need for parameter tuning.
Additionally, all proposed methods have been shown to work across different simulators of varying computational speed and biomechanical fidelity, showcasing the versatility of our approach.
In another study, we investigated the contributions of muscles to control in a robotic setting.
By emulating a simplified muscle actuator in simulation across a wide range of tasks and models, we could investigate the influence of bio-inspired properties on control aspects such as robustness and data efficiency.
Our investigations reveal that while idealized torque actuators can achieve optimal performance under precise conditions, muscle-like properties offer superior robustness and learning efficiency across varied tasks and environments.
By not relying on prescribed trajectories, but only abstract reward terms, we gave the optimization enough freedom such that the embodiment could have a measurable effect on the learned behaviors.
In an extension to this result, we applied a similar muscle actuator to a simulated quadruped robot.
It was observed that the muscular actuation could, in addition to producing more robust behaviors, also bias the learned behaviors towards being more natural and smooth.
By building an experimental pipeline where a muscle actuator, identical to the simulated one, could be emulated on real robotic hardware, we were able to perform sim-to-real transfer with a learned RL policy and achieve walking with a real quadruped robot.
After performing a parameter optimization of the actuator where feasibility on the hardware was ensured through a newly derived damping rule, it could be seen that the bias induced through the muscle actuator led to behaviors transferring better from simulation to the real system.
Our results show that RL is a viable candidate to create closed-loop controllers that produce robust and close-to-natural behaviors in high-dimensional musculoskeletal systems under a variety of conditions.
We believe his to be a first step to enable new advances in biomechanics, motor control and rehabilitation. This thesis also highlights the potential benefits of muscle actuation for robotic systems and the use of emulated actuators to enable the simultaneous optimization of robot morphology and control architecture on real hardware.
