Исследование эффективности промпт-инжиниринга и квантованных LLM в создании структуры академических курсов

Polina Shnaider; Anastasiia Chernysheva; Anna Nikiforova; Anton Govorov; Maksim Khlopotov

doi:10.32603/2071-2340-2024-1-32-44

Polina Shnaider ITMO University, 49 Kronverksky, bldg. A, 197101, Saint Petersburg, Russia https://orcid.org/0000-0002-4147-6561
Anastasiia Chernysheva ITMO University, 49 Kronverksky, bldg. A, 197101, Saint Petersburg, Russia http://orcid.org/0000-0002-9956-6607
Anna Nikiforova ITMO University, 49 Kronverksky, bldg. A, 197101, Saint Petersburg, Russia
Anton Govorov ITMO University, 49 Kronverksky, bldg. A, 197101, Saint Petersburg, Russia http://orcid.org/0009-0005-6674-1666
Maksim Khlopotov ITMO University, 49 Kronverksky, bldg. A, 197101, Saint Petersburg, Russia http://orcid.org/0000-0002-9053-027X

DOI: https://doi.org/10.32603/2071-2340-2024-1-32-44

Keywords: Large Language Models, Prompt Engineering, Quantized Models, few-shot, zero-shot, chain-of-thought

Abstract

This article presents the outcomes of an experiment employing large language models (LLMs) in the development of university course structures. Various prompt engineering methods, including zero-shot, few-shot, chain-of-thought, and tree-of-thought, were employed to formulate queries to LLMs. Primarily, quantized models such as mistral-7b-instruct, mixtral-8x7b-instruct, openchat_3.5, saiga2_13b, starling-lm-7b-alpha, tinyllama, among others, were utilized for the experiment. The generated course structures were compared with data obtained from ChatGPT-4. Models openchat_3.5.q5_k_m and starling-lm-7b-alpha.q5_k_m demonstrated comparable quality in generating educational program structures to ChatGPT-4. The experiment underscores the potential applications of LLMs in the field of education and highlights promising directions for further research.

Author Biographies

Polina Shnaider, ITMO University, 49 Kronverksky, bldg. A, 197101, Saint Petersburg, Russia

Postgraduate, Assistant, Faculty of Infocommunication Technologies, ITMO University, beatrix.linkoln@gmail.com

Anastasiia Chernysheva, ITMO University, 49 Kronverksky, bldg. A, 197101, Saint Petersburg, Russia

Assistant, Faculty of Infocommunication Technologies, ITMO University, avchernysheva@itmo.ru

Anna Nikiforova, ITMO University, 49 Kronverksky, bldg. A, 197101, Saint Petersburg, Russia

3rd year Student of the bachelor’s degree program, Faculty of Infocommunication Technologies, ITMO University, 34743@niuitmo.ru

Anton Govorov, ITMO University, 49 Kronverksky, bldg. A, 197101, Saint Petersburg, Russia

Senior Lecturer, Faculty of Infocommunication Technologies, ITMO University, govorov@itmo.ru

Maksim Khlopotov, ITMO University, 49 Kronverksky, bldg. A, 197101, Saint Petersburg, Russia

Candidate of Sciences (Tech.), Associate Professor, Faculty of Infocommunication Technologies, ITMO University, khlopotov@itmo.ru

References

E. Jermakowicz, “The Coming Transformative Impact of Large Language Models and Artificial Intelligence on Global Business and Education,” Journal of Global Awareness, vol. 4, no. 2, pp. 1–22, 2023; doi:10.24073/jga/4/02/03

S. Laato, B. Morschheuser, J. Hamari, and J. Bj¨orne, “AI-Assisted Learning with ChatGPT and Large Language Models: Implications for Higher Education,” in 2023 IEEE International Conference on Advanced Learning Technologies (ICALT), Orem, Utah, United States, pp. 226–230, 2023; doi:10.1109/icalt58122.2023.00072

A. Giretti et al., “Integrating large language models in art and design education,” in Proc. of the Int. Conf. on Cognition and Exploratory Learning in the Digital Age, Madeira Island, Portugal, 21-23 October, 2023, pp. 1–7, 2023.

J. Jeon and S. Lee, “Large language models in education: A focus on the complementary relationship between human teachers and ChatGPT,” Education and Information Technologies, vol. 28, no. 12, pp. 15873–15892, 2023; doi:10.1007/s10639-023-11834-1

M. Abedi, I. Alshybani, M. Shahadat, and M. Murillo, “Beyond Traditional Teaching: The Potential of Large Language Models and Chatbots in Graduate Engineering Education,” in qeios.com, 2023 [Preprint]; doi:10.32388/md04b0.2

J. Prather et al., “The Robots Are Here: Navigating the Generative AI Revolution in Computing Education,” in Proc. of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education, 2023; doi:10.1145/3623762.3633499

M. Irfan and L. Murray, Micro-Credential: A Guide to Prompt writing and Engineering in Higher Education: A tool for Artificial Intelligence in LLM, Limerick, Ireland: University of Limerick, 2023; doi:10.13140/RG.2.2.15596.95367

T. Kojima, S. (Shane) Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large Language Models Are Zero-Shot Reasoners,” in Advances in Neural Information Processing Systems, vol. 35, pp. 22199–22213, 2022.

P. Dhariwal et al., “Language Models Are Few-Shot Learners,” in 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada, vol. 33. pp. 1–25, 2020.

J. Wei et al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,” in Advances in Neural Information Processing Systems, vol. 35, pp. 1–14, 2022.

S. Yao et al., “Tree of Thoughts: Deliberate Problem Solving with Large Language Models,” in arxiv.org, 2023. [Online]. Available: https://arxiv.org/abs/2305.10601

S. Li et al., “LLM-MQ: Mixed-precision Quantization for Efficient LLM Deployment,” in NeurIPS 2023 Efficient Natural Language and Speech Processing Workshop, pp. 1–5, 2023.

W. Li, A. Hu, N. Xu, and G. He, “Quantization and Hardware Architecture Co-Design for Matrix-Vector Multiplications of Large Language Models,” IEEE Transactions on Circuits and Systems I: Regular Papers, pp. 1–14, 2024; doi:10.1109/tcsi.2024.3350661

A. Q. Jiang et al., “Mixtral of Experts,” in arxiv.org, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2401.04088

G. Wang et al., “OpenChat: Advancing Open-source Language Models with Mixed-Quality Data,” in arxiv.org, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2309.11235

B. Zhu et. al, “Starling-7B: Improving LLM Helpfulness & Harmlessness with RLAIF,” in starling.cs.berkeley.edu, 2023. [Online]. Available: https://starling.cs.berkeley.edu/

C. Irugalbandara, “A Trade-off Analysis of Replacing Proprietary LLMs with Open Source SLMs in Production,” in arxiv.org, 2024. [Online]. Available: arXiv:2312.14972https://doi.org/10.48550/arXiv. 2312.14972

M. Tikhomirov and D. Chernyshev, “Impact of Tokenization on LLaMa Russian Adaptation,” in arxiv.org, 2023. [Online]. Available: https://arxiv.org/html/2312.02598v119. J. Chen, H. Lin, X. Han, and L. Sun, “Benchmarking Large Language Models in Retrieval-Augmented Generation,” in arxiv.org, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2309.01431

Exploring the Effectiveness of Prompt Engineering and Quantized Large Language Models in the Development of Academic Courses

Abstract

Author Biographies

References