We delve into the intersection of large language models (LLMs) and health care with David Lareau, a health care executive. Since the debut of ChatGPT, the health care sector has been abuzz with anticipation and intrigue regarding the potential applications of LLMs in medicine. However, amidst the excitement lies a critical discussion about the inherent limitations and risks associated with these powerful tools. Join us as we explore the challenges of LLMs hallucinating inaccurate information, the perpetuation of biases in medical data, and the importance of integrating clinical relevancy engines to refine LLM outputs.