Исследование эффективности промпт-инжиниринга и квантованных LLM в создании структуры академических курсов
Аннотация
В данной статье представлены итоги эксперимента по применению больших языковых моделей (LLM) для создания структуры университетских курсов. Для формирования запросов к LLM использовались такие методы промпт-инжиниринга, как zero-shot, few-shot, chain-of-thought и tree-of-thought. Для эксперимента преимущественно использовались квантованные модели, такие как mistral-7b-instruct, mixtral-8x7b-instruct, openchat_3.5, saiga2_13b, starling-lm-7b-alpha, tinyllama и другие. Сгенерированные ими структуры курсов сравнивались с данными, полученными с помощью ChatGPT-4. Модели openchat_3.5.q5_k_m и starling-lm-7b-alpha.q5_k_m показали сопоставимое с ChatGPT-4 качество генерации рабочих программ дисциплин. Эксперимент подчеркивает возможности применения LLM в сфере образования и указывает на перспективные направления для дальнейших исследований.
Литература
E. Jermakowicz, “The Coming Transformative Impact of Large Language Models and Artificial Intelligence on Global Business and Education,” Journal of Global Awareness, vol. 4, no. 2, pp. 1–22, 2023; doi:10.24073/jga/4/02/03
S. Laato, B. Morschheuser, J. Hamari, and J. Bj¨orne, “AI-Assisted Learning with ChatGPT and Large Language Models: Implications for Higher Education,” in 2023 IEEE International Conference on Advanced Learning Technologies (ICALT), Orem, Utah, United States, pp. 226–230, 2023; doi:10.1109/icalt58122.2023.00072
A. Giretti et al., “Integrating large language models in art and design education,” in Proc. of the Int. Conf. on Cognition and Exploratory Learning in the Digital Age, Madeira Island, Portugal, 21-23 October, 2023, pp. 1–7, 2023.
J. Jeon and S. Lee, “Large language models in education: A focus on the complementary relationship between human teachers and ChatGPT,” Education and Information Technologies, vol. 28, no. 12, pp. 15873–15892, 2023; doi:10.1007/s10639-023-11834-1
M. Abedi, I. Alshybani, M. Shahadat, and M. Murillo, “Beyond Traditional Teaching: The Potential of Large Language Models and Chatbots in Graduate Engineering Education,” in qeios.com, 2023 [Preprint]; doi:10.32388/md04b0.2
J. Prather et al., “The Robots Are Here: Navigating the Generative AI Revolution in Computing Education,” in Proc. of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education, 2023; doi:10.1145/3623762.3633499
M. Irfan and L. Murray, Micro-Credential: A Guide to Prompt writing and Engineering in Higher Education: A tool for Artificial Intelligence in LLM, Limerick, Ireland: University of Limerick, 2023; doi:10.13140/RG.2.2.15596.95367
T. Kojima, S. (Shane) Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large Language Models Are Zero-Shot Reasoners,” in Advances in Neural Information Processing Systems, vol. 35, pp. 22199–22213, 2022.
P. Dhariwal et al., “Language Models Are Few-Shot Learners,” in 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada, vol. 33. pp. 1–25, 2020.
J. Wei et al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,” in Advances in Neural Information Processing Systems, vol. 35, pp. 1–14, 2022.
S. Yao et al., “Tree of Thoughts: Deliberate Problem Solving with Large Language Models,” in arxiv.org, 2023. [Online]. Available: https://arxiv.org/abs/2305.10601
S. Li et al., “LLM-MQ: Mixed-precision Quantization for Efficient LLM Deployment,” in NeurIPS 2023 Efficient Natural Language and Speech Processing Workshop, pp. 1–5, 2023.
W. Li, A. Hu, N. Xu, and G. He, “Quantization and Hardware Architecture Co-Design for Matrix-Vector Multiplications of Large Language Models,” IEEE Transactions on Circuits and Systems I: Regular Papers, pp. 1–14, 2024; doi:10.1109/tcsi.2024.3350661
A. Q. Jiang et al., “Mixtral of Experts,” in arxiv.org, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2401.04088
G. Wang et al., “OpenChat: Advancing Open-source Language Models with Mixed-Quality Data,” in arxiv.org, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2309.11235
B. Zhu et. al, “Starling-7B: Improving LLM Helpfulness & Harmlessness with RLAIF,” in starling.cs.berkeley.edu, 2023. [Online]. Available: https://starling.cs.berkeley.edu/
C. Irugalbandara, “A Trade-off Analysis of Replacing Proprietary LLMs with Open Source SLMs in Production,” in arxiv.org, 2024. [Online]. Available: arXiv:2312.14972https://doi.org/10.48550/arXiv. 2312.14972
M. Tikhomirov and D. Chernyshev, “Impact of Tokenization on LLaMa Russian Adaptation,” in arxiv.org, 2023. [Online]. Available: https://arxiv.org/html/2312.02598v119. J. Chen, H. Lin, X. Han, and L. Sun, “Benchmarking Large Language Models in Retrieval-Augmented Generation,” in arxiv.org, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2309.01431
Материал публикуется под лицензией:
