VizFlyt: Perception-centric Pedagogical Framework For Autonomous Aerial Robots

WPI 2024

Autonomous aerial robots are becoming commonplace in our lives. Hands-on aerial robotics courses are pivotal in training the next-generation workforce to meet the growing market demands. Such an efficient and compelling course depends on a reliable testbed. In this paper, we present VizFlyt, an open-source perception-centric Hardware-In-The-Loop (HITL) photorealistic testing framework for aerial robotics courses. We utilize pose from an external localization system to hallucinate real-time and photorealistic visual sensors using 3D Gaussian Splatting. This enables stress-free testing of autonomy algorithms on aerial robots without the risk of crashing into obstacles. We achieve over 100Hz of system update rate. Lastly, we build upon our past experiences of offering hands-on aerial robotics courses and propose a new open-source and open-hardware curriculum based on VizFlyt for the future. We test our framework on various course projects in real-world HITL experiments and present the results showing the efficacy of such a system and its large potential use cases.

M. Velmurugan*, K. Srivastava*, R. Kulkarni*, and N. J. Sanket, "VizFlyt: Perception-centric Pedagogical Framework For Autonomous Aerial Robots," submitted to ICRA 2025 (under review, * means equal contribution).

EinsteinVision: A Deep Learning-Powered 3D Visualization Tool for Autonomous Vehicle Dashboard Displays

WPI 2024

EinsteinVision is a self-driving car visualization project aimed at providing visually appealing and informative displays for autonomous vehicle systems. The project was developed during a computer vision course at WPI by Manoj Velmurugan and Rishabh Singh. We used deep learning models to detect objects, human poses, lanes, and depth in images. These detections were converted into 3D coordinates and placed in Blender at the reprojected 3D locations, where they were rendered to create a dashboard visualizer for self-driving vehicles. Deep neural network models like Depth Anything, Mask R-CNN, and OpenPifPaf were used for accurate depth estimation, lane detection, and car pose estimation. Traffic lights, tail lights, and speed bumps were also detected for comprehensive scene understanding.

PX4 Hardware-in-the-Loop (HITL) Simulation with Fixed-Wing Plant

MathWorks 2022

I developed a Simulink and Stateflow model for flight mode management, guidance, and control of a fixed-wing UAV. The custom controller model integrates into the PX4 stack, enabling seamless flight operations. An L1 guidance scheme was used for path tracking, while a cascaded PID controller architecture, inspired by PX4, was employed for control.
I formulated a cascaded control structure to control the following states:

• Airspeed
• Altitude & climb rate
• Course
• Attitude & angular rate

Using Embedded Coder, the controller is automatically generated and deployed onto a PX4 Autopilot Pixhawk board. The deployed code is rigorously tested in a Hardware-in-the-Loop (HIL) setup, with the Simulink-based plant model running on a PC or a Speedgoat real-time target computer to ensure accurate and reliable performance.
Credits: The plant dynamics model mostly was modified/adopted from Mariano Lizarraga's PhD work.

PX4 Hardware-in-the-Loop (HITL) Simulation with Quadrotor Plant

MathWorks 2020-2022

Contributed significantly to the development of a key example in the UAV Toolbox support package for PX4 Autopilots. This example showcases the process of designing controllers for PX4-based autopilots for quadrotors and validating them using a Simulink-based plant model. My work focused on:

• Developing the plant dynamics model
• Designing the controller model, position controllers to angular rate controllers
• Implementing portions of the sensor simulation

The controller was deployed and tested in a Hardware-in-the-Loop (HIL) configuration, with the Simulink plant model executed on a PC or a Speedgoat real-time target computer to ensure robust validation of the UAV's control system.

Controllers and Estimators for Parrot Minidrone

MathWorks 2019-2020

Enhanced the Parrot Minidrone example and addressed issues with the IMU temperature-induced bias, which caused significant errors in attitude and position estimates. Key improvements included:

• Replacing the complementary filter with a linear Kalman Filter (KF) for roll and pitch estimation, incorporating gyro bias as a state.
• Adding accelerometer bias as a state in the existing velocity KF for x and y directions.
• Redesigning the control architecture from a position PID → attitude PID loop to a cascaded structure: position P → velocity PI → attitude P → angular velocity PID control loops, improving stability and performance.
• Revising the dynamics, control, and estimator model equations to fix errors in the original formulation.

These modifications resulted in a significant reduction in position drift, from 1 meter to less than 20 cm during a 1-minute flight, and enhanced the controller's responsiveness to disturbances.

In-House Autopilot for an Indoor Quadcopter

RAFT Lab, IIT Madras 2016-2019

Developed a custom Arm Mbed-based autopilot system for controlling an indoor quadrotor autonomously. Highlights include:

• Engineered multi-loop PID controls with derivative filtering and integral anti-windup, optimizing for a critically damped response and a 0.5-0.8 s settling time.
• Implemented a complementary filter for pitch, roll, and downward velocity, alongside a linear Kalman Filter for lateral velocities, factoring in accelerometer bias and optical flow data.
• Derived a realistic quadcopter dynamics model through empirical research on inertia, motor, and propeller characteristics.
• Authored extensive C++ codebase and designed a custom PCB for sensor interfacing on the ST-Nucleo board.
• Presented findings at the Indian Control Conference 2019, IIT Delhi, fostering academic and practical applications in aerial robotics.

This project led to robust flight performance and practical educational tools for peers, significantly advancing the capabilities of indoor UAVs.

First Solar-Powered Airplane of IIT Madras

RAFT Lab, IIT Madras 2017

As part of a team of five students at IIT Madras, we developed the institute's first solar-powered airplane that successfully taxied under its own power. I designed and fabricated a solar cell array and battery pack, managed the electrical subsystem, and contributed to the aerodynamic design, ensuring the aircraft's operational efficiency. The airplane, with a 3-meter wingspan and equipped with 52 SunPower C60 cells, demonstrated the effectiveness of our solar power design.

MuJoCo Simulink Blockset

MathWorks 2022-2023

Developed a Simulink Blockset for the MuJoCo simulator, enabling high-performance robot simulation and autonomous algorithm development within Simulink. The blockset uses C++ MEX S-Functions to interface directly with the MuJoCo API, providing real-time access to actuators and sensors, along with camera rendering within Simulink.

VEX Mobile Robot Simulation in Gazebo

MathWorks 2020-2021

Developed realistic models of VEX Mobile Robot in Gazebo, including mechanisms like skid steer drive, grippers, four-bar linkages, and gears. The work involved exact CAD modeling and setting up critical parameters such as inertia, friction, and stiffness coefficients. The Gazebo model was then interfaced with Simulink using the Robotics System Toolbox for seamless integration.

Vision-Centric Quadrotor Simulator in Blender

Worcester Polytechnic Institute 2023

Blender is excellent for simulating vision systems like cameras and depth sensors, but lacks built-in dynamics for accurate UAV simulation. To overcome this, I rewrote the quadrotor physics and controllers from scratch in Python and integrated them with Blender using its Python scripting APIs. The simulator was used in motion planning assignments for the Aerial Robotics course at WPI. I was responsible for implementing the dynamics, control systems, state machines, and API integration. Camera rendering was done in collaboration with Siyuan Huang (MS WPI).