Interactive data exploration with multimodal AI

Music City Center 

Natural world data collection has increased significantly in recent years, including images, video, and audio captured to understand ecosystems and biodiversity. These data provide a vast and largely uncurated source of scientific information, only a fraction of the data has been analyzed and that analysis is limited, usually focusing on a few target species or categories. However, these sources contain a wealth of "secondary data" including crucial insights into, e.g., interactions, animal social behavior, morphology, habitat, co-occurrence, that is too costly, time-consuming, or expert-dependent to extract at scale. Recent advances in AI methods enable language-based interaction with large-scale databases, potentially enabling the efficient and automated processing techniques needed to unlock the "hidden treasure" in such datasets– being able to directly search large data collections for any scientifically relevant concept would enable richer analyses that span beyond species identification. We propose interactive, open-ended data retrieval via language as a mechanism to support scientific discovery in these collections, and demonstrate a proof-of-concept in partnership with iNaturalist.