The SciQA benchmark for scientific question answering presents a challenging task for next-generation question-answering systems, where standard large language models often fall short. In this seminar, we will analyze the performance of language models on this benchmark, exploring both prompting and fine-tuning techniques to enhance their adaptability to the SciQA task. Our findings demonstrate that intelligent few-shot selection in prompting and targeted fine-tuning can lead to excellent results on the SciQA benchmark. We will also discuss valuable insights gained, common error categories encountered, and their implications for optimizing large language models for question answering over knowledge graphs.