On-campus Hybrid Experience System: Integrating Software and Hardware

Category	Duration	Tool / Material
Real-Time Communications; Design; Development	September 2022 - December 2022 @ New York University	JavaScript; C++

This is part of my research assistant work in 2022 fall semester. My supervisor is Aidan Nelson who owns a cool startup called Arium! Our goal is to break the limitations of traditional conferencing software and explore the possibilities of real-time video and audio applications on campus, so that remote access students and faculty can have a variety of scenarios and ways to interact with onsite students and faculty. Guided by Aidan, I analyzed the project objectives and task decomposition and explored the following respectively (and more than these).

① 360° camera to replace the traditional 2D video with a 3D video that can capture more environmental information.
② WebRTC to realize real-time web-based communication.
③ WebGL to complete the construction of 3D video and scenes.
④ Arduino to communicate with the web wirelessly so that online users can remotely control the onsite Arduino device to realize gesture control to help the students and faculty who are unable to get in-time onsite responses through video and audio have more ability to get attention.
⑤ Potential usage of such real-time experience leveraged by AI. This then becomes my thesis project called 3Description, which is published as 3Description: An Intuitive Human-AI Collaborative 3D Modeling Approach.

In this project, I selected Equipment Room as a field experiment location and 10 audiences with diverse backgrounds for 1-on-1 interviews and iterated the solution based on user test results. The content below is the work I did during 2022 fall and winter.