Molecular graphics have been instrumental in advancing chemistry, drug discovery, materials science and structural biology, enabling visualization of molecular systems from static images to dynamic web displays and immersive platforms. While effective visualization is largely a solved problem, in this perspective we argue that the next significant leap lies beyond passive viewing. Rather, the frontier is in enabling intuitive, immersive, direct 3D manipulation and interaction with molecular systems to address the inherent limitations of carrying out complex 3D tasks via 2D interfaces. The vision, which is starting to be realized, is of immersive molecular science environments in which researchers leverage multi-modal inputs that feel natural (with hands plus possibly haptic or pseudo-haptic feedback, voice user interfaces and AI-based assistance), collaborating seamlessly in concurrent sessions, naturally visualizing and connecting different types of data and models, and engaging with molecules assisted by real-time simulation engines. Many of the technologies required to develop such environments already exist, at least in basic forms. In this Perspective, we discuss current prototypes and software solutions that incorporate some of the elements needed, and that are available for use. We discuss applications and practical demonstrations, and outline the developments that are required to make the Molecular Holodeck a reality. We also discuss challenges that need to be addressed in order to achieve this vision. The coming shift toward hands-on, multiuser, immersive, natural and physics-informed manipulation will transform hypothesis generation, molecular design, fundamental understanding, collaborative working, discussions, and thus research and education in the chemical sciences, as others have envisioned for "molecular visualization in the holodeck."