Even though Artificial Intelligence (AI) has existed for a long time, its broad accessibility is a recent development, thanks to Generative AI models like ChatGPT for its human-like interactions. While such broad accessibility provides a great opportunity to democratize AI across general people, it comes with several key risks and challenges, including but not limited to a lack of Knowledge Grounding/Contextual Understanding in unseen/new domains, an abundance of Biased Contents/Narratives, and a lack of Utility-Centric Evaluation of Generative AI systems. This talk will focus on two specific challenges related to the democratization of AI, i.e., 1) Controlled Narrative Generation and 2) Knowledge Grounding in Conversational-AI systems, and discuss practical solutions and appropriate evaluation approaches for them. The talk will also introduce several utility-centric evaluation metrics for measuring the quality of Generative and Conversational AI systems that correlate with human judgments better than traditional metrics. Finally, the talk will highlight some interesting future directions in line with the democratization of AI and its associated challenges.