Constrained Optimization for Safe and Visibility-Aware Shared Control of Magnetically Actuated Microrobots

Safe and intuitive telemanipulation of multiple microrobots is limited by visual occlusions that compromise control and task success. This paper introduces a shared control framework that guarantees safety and visibility by formulating the control problem as a constrained optimization problem. Our framework combines Control Lyapunov Functions (CLFs) for operator-driven stability with High-Order Control Barrier Functions (HOCBFs) to enforce collision and occlusion avoidance, all resolved within a real-time Quadratic Program (QP). A key innovation is a rendering technique based on the QP's Lagrange multipliers, which are used to decompose the optimal control solution. This method isolates the effects of individual constraints to provide the human operator with distinct, interpretable visuo-haptic cues for navigation and avoidance. The architecture also includes a viewpoint optimization system with a virtual autonomous camera to maximize task visibility. The framework is validated via two user studies in simulation and a demonstration on a real-world electromagnetic microrobotics actuation system. Results confirm that our visuo-haptic shared control strategies significantly improve user performance and task completion rates, while the control design provably prevents the microrobots from colliding or entering occluded regions. Note to Practitioners - Remote manipulation of magnetic microrobots presents significant challenges. Operators frequently struggle with collisions or losing track of the robot within the camera field of view, compromising safety. To counter this, we propose a shared-control architecture that functions as an intelligent assistant. Our algorithm automatically restricts robot motion to prevent collisions and maintain visibility. Moreover, we couple this autonomy with haptic feedback. When the system intervenes to stop an unsafe move, the operator feels a resistive force on the control interface. This feedback provides immediate awareness of system constraints, maintaining operator agency while ensuring safety. Experimental validation demonstrates that this approach reduces task completion time and error rates compared to manual control. We validated the system in a laboratory setting with two microrobots. While not yet deployed in clinical scenarios, this framework lays the groundwork for, e.g., controlling medical tools via ultrasound imaging. Furthermore, these feedback principles extend beyond the micro-scale, offering safety solutions for industrial manipulators in cluttered environments.