Distinguished Seminar Series on Data Science & Artificial Intelligence - "Building Generative AI Applications: Architectures, Evaluation, and Experiments" by Prof. John C. Mitchell

As machine learning models continue to improve, architectures for building applications that leverage foundation models are also developing rapidly. In addition to various prompting methods, retrieval-augmented generation (RAG) is well developed and agentic architectures are gaining traction. For any of these methods to be effective, systematic evaluation is also essential: it is often relatively easy to produce a working tool simply by writing a prompt. However, building a reliable, effective and trustworthy tool requires continued effort based on clear evaluation metrics. This talk will consider three approaches – prompting, RAG and agents – and  corresponding evaluation methods, illustrated through a set of experiments related to educational support tools and other domains.