About This Project
AI-driven interspecies communication projects to decode the vocalizations of sperm whales are underway. Yet there is room to reimagine and extend existing methods of visualizing the complex, multidimensional recorded data and contextual metadata. Using advanced features of Unreal Engine, this project aims to create an engaging, multimodal, realistic simulation of sperm whales and their environment, including their intricate behaviors, perception, and profoundly 3D existence.
Ask the Scientists
Join The DiscussionWhat is the context of this research?
The complexity of sperm whale vocalizations represents an exciting challenge for researchers seeking to unravel interspecies communication. Significant strides have been made in the necessarily massive data capture and its AI-driven analysis, but there is a relatively unmet need to visualize this data in a multidimensional, multimodal, interactive way that authentically represents the whales' behavior and their environment. CetaSim aims to fill this gap by leveraging advanced features of Unreal Engine to create an engaging and extensive data-driven simulation of the sperm whales' motions, behaviors, interactions, and the truly three dimensional world they inhabit.
What is the significance of this project?
CetaSim has the potential to enrich our understanding and interpretation of AI-analyzed non-human communication. By presenting complex data in an immersive, realistic simulation, it will enable us not just to visualize data but also experience the world as the whales do; a valuable perspective for researchers decoding their communication as well as sharing the results. On the evaluation front, CetaSim could potentially provide an effective environment for generating spatial/behavioral inference in tandem with testing language models. Importantly, while initially focused on sperm whales, CetaSim's underlying concept and framework will be designed to be extended to other cetacean language initiatives, and potentially other interspecies communication research.
What are the goals of the project?
The core objectives of CetaSim are:
1. Creating a VR-capable, multimodal, realistic simulation of sperm whales and their environment using Unreal Engine. CetaSim will provide novel approaches to representing the fundamentally three dimensional nature of the whales' behaviors, vocalizations, perception, and their world.
2. Visualizing spatially complex, multidimensional data from research projects in a more engaging and informative way. This includes bioacoustic data as well as contextual metadata like location, motion, and environmental factors.
3. Developing a flexible and extensible framework that incorporates numerous next-generation game engine features, ranging from procedural animation and sound generation to machine learning agent-based behaviors.
Budget
As an active game developer, I am currently committed to continue working on an online VR game I helped create. This proposed budget will facilitate setting time aside from that work, enabling me to devote more of my attention to this project's completion. Additionally, while Unreal Engine itself is free and open source at this scale, there are costs associated with numerous 3rd party plugins and assets needed to properly bring this ambitious project to life within a reasonable timeframe.
Endorsed by
Project Timeline
By the end of the year, I plan to have configurable/dynamic whales swimming around a realistic ocean environment. Early next year, more advanced behaviors will be scripted and/or trained, such as hunting squid. By next Spring, the pipeline for importing (and ideally streaming) research data and metadata will be implemented. During Summer 2024, CetaSim should be ready for Beta testing, and then release.
Oct 25, 2023
Project Launched
Dec 01, 2023
Marine environment with topographical sea floor, Fluid Flux ocean simulation, production whale rigs, configurable physical attributes and materials
Jan 01, 2024
Procedural/physics-based animation system, Learning Agents model integration and training, dynamic/procedural Metasounds audio
Feb 01, 2024
3D audio visualization system developed with view modes, incorporated with various data views, overlays, graphs, UI set up for single screen and VR modes
Mar 01, 2024
Advanced behaviors scripted or trained, integration between 3D audio visualization and whale AI sense/perception
Meet the Team
Ethan Abramson
I'm a video game and VR developer/artist/designer with decades of experience in a wide range of projects and technologies. For the past couple of years my work has increasingly focused on AI-related features for an online VR platform. such as implementing GPT-4 based NPCs, Stable Diffusion for interactive in-game art created by players, and most recently, in-engine machine learning agents.
I've also had a lifelong fascination with cetaceans going back to some of my earliest memories at the Museum of Natural History, whale watching off Cape Cod, and countless hours of listening to Roger Payne's groundbreaking Songs of the Humpback Whale as a kid. This was in part due to my father being a record producer for Judy Collins and having used Payne's whale recordings in her hit "Farewell to Tarwathie". So my connection to whales and their calls is pretty deeply ingrained.
Presently, the growing confluence between AI/ML and interspecies communication is a source of exhilaration for me. I believe if these endeavors are successful, it will likely have humbling, far reaching effects for human civilization. Notably, we must inevitably recognize that there are many diverse possible forms of sentience, our own being but one anthropocentric example.
Lab Notes
Nothing posted yet.
Additional Information
My intention first and foremost is to make CetaSim as useful as possible to its primary user base, namely sperm whale communication researchers and anyone with an interest in visualizing those projects' data. To that end, I endeavor to incorporate requested features or changes based on feedback.
However, it's worth noting that CetaSim will not at all be limited to researchers in the field. Teachers, students, and the general public can also benefit from this engaging, immersive experience, ideally helping to raise awareness and foster a deeper connection with these enigmatic beings, discover the intricacies of their rich socio-familial cultures, and perhaps appreciate the gravity of their present existential vulnerability.
Project Backers
- 8Backers
- 101%Funded
- $4,050Total Donations
- $506.25Average Donation