Systems and Means of Informatics
2025, Volume 35, Issue 4, pp 60-72
MOVEMENT, VELOCITY, AND TRAJECTORIES OF KEYWORD REPRESENTATIONS IN THE VECTOR SPACE OF THE LANGUAGE MODEL
- M. M. Charnine
- N. V. Somin
Abstract
A method for calculating the positions, velocities, and evolutionary trajectories of keywords in the vector space of a static language model is proposed.
The semantic distance between word vectors at times ti and t2 is defined as the cosine distance between these vectors. The rate of semantic change is calculated as the semantic distance divided by time (t2 - t1). The rate of semantic change expresses how quickly the meaning/semantics of a word, its context, position in the vector space, and semantically close words change. The method allows one to calculate the velocities and evolutionary trajectories of topics representing a set of several related keywords. To calculate the velocities and trajectories, special evolutionary labels are inserted into the analyzed source text next to the words from the topic of interest. The case of the velocities and trajectories of keywords in the field of "machine learning" obtained from the PubMed library is considered. Keyword vectors and their changes over time are calculated using the Word2Vec neural network. A semantic map is presented that allows one to visually assess evolutionary trajectories and velocities. It is based on the PCA (Principal Component Analysis) algorithm which allows one to obtain a projection of trajectories onto a two-dimensional plane.
[+] References (21)
- Burlak, S.A., and S.A. Starostin. 2005. Sravnitel'no-istoricheskoe yazykoznanie [Comparative historical linguistics]. Ser. Vysshee professional'noe obrazovanie. Yazykoznanie [Higher professional education ser. Linguistics]. Moscow: Academia. 432 p. EDN: QRMHLV.
- Blank, A. 1997. Prinzipien des lexikalischen Bedeutungswandels am Beispiel der romanischen Sprachen. Berlin, Boston: Max Niemeyer Verlag. 533 p. doi: 10.1515/9783110931600.
- Gulordava, K., and M. Baroni. 2011. A distributional similarity approach to the detection of semantic change in the Google Books Ngram Corpus. Workshop on Geometrical Models of Natural Language Semantics Proceedings. Edinburgh, U.K.: Association for Computational Linguistics. 67-71.
- Mikolov, T. 2013. Efficient estimation of word representations in vector space. Cornell University. 12 p. Available at: https://arxiv.org/pdf/1301.3781 (accessed November 10, 2025).
- Devlin, J., M.-W. Chang, K. Lee, and K. Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Proceedings. Minneapolis, MN: Association for Computational Linguistics. 1:4171M186.
- Liu, X., Y. Zheng, Z. Du, et al. 2024. GPT understands, too. AI Open 5:208{215. doi: 10.1016/j.aiopen.2023.08.012.
- Mina, A., R. Ramlogan, G. Tampubolon, and S. Metcalfe. 2007. Mapping evolutionary trajectories: Applications to the growth and transformation of medical knowledge. Res. Policy 36(5):789{806. doi: 10.1016/j.respol.2006.12.007.
- Bamler, R., and S. Mandt. 2017. Dynamic word embeddings. 34th Conference (International) on Machine Learning Proceedings. Sydney, Australia: PMLR. 70:380{ 389.
- Yao, Z., Y. Sun, W. Ding, et al. 2018. Dynamic word embeddings for evolving semantic discovery. 11th ACM Conference (International) on Web Search and Data Mining Proceedings. New York, NY: Association for Computing Machinery. 673{681. doi: 10.1145/3159652.3159703.
- Maaten, L., and G. Hinton. 2008. Visualizing data using t-SNE. J. Mach. Learn. Res. 9(11):2579{2605.
- Hamilton, W.L., J. Leskovec, and D. Jurafsky. 2016. Diachronic word embeddings reveal statistical laws of semantic change. Cornell University. 13 p. Available at: https://arxiv.org/pdf/1605.09096 (accessed November 10, 2025).
- Englhard, A., J. Willkomm, M. Schaler, et al. 2020. Improving semantic change analysis by combining word embeddings and word frequencies. Int. J. Digital Libraries 21(3):247{264. doi: 10.1007/s00799-019-00271-6.
- Li, Y., and C. S. Siew. 2022. Diachronic semantic change in language is constrained by how people use and learn language. Mem. Cognition 50(6):1284M298. doi: 10.3758/s13421 -022-01331 -0.
- Wu, S., M. Koo, L. Blum, A. Black, L. Kao, F. Scalzo, and I. Kurtz. 2023. A comparative study of open-source large language models, GPT-4 and Claude 2: Multiple-choice test taking in nephrology. Cornell University. 7 p. Available at: https://arxiv.org/pdf/2308.04709 (accessed November 10, 2025).
- Rahman, A., S. H. Mahir, MdT. AnTashrif, et al. 2025. Comparative analysis based on DeepSeek, ChatGPT, and Google Gemini: Features, techniques, performance, future prospects. Cornell University. 20 p. Available at: https://arxiv.org/pdf/2503.04783 (accessed November 10, 2025).
- Moshninov, A., and M. Komarov. 2025. Artificial intelligence in Russia: Evaluating Yandex GPT-5 and GigaChat 2.0 MAX in global context. Conference (International) on Artificial Intelligence Management and Trends Proceedings. Abu Dhabi, UAE: ADSM. 38M4. doi: 10.63962/URBB4192.
- Hofmann, V., J.B. Pierrehumbert, and H. Schutze. 2020. Dynamic contextualized word embeddings. Cornell University. 15 p. Available at: https://arxiv.org/pdf/ 2010.12684 (accessed November 10, 2025).
- Firth, J. 1957. A synopsis of linguistic theory 1930-1955. Studies in linguistic analysis. Oxford: Blackwell. 1-32.
- PubMed biomedical literature database. Available at: https://pubmed.ncbi.nlm.nih. gov (accessed November 10, 2025).
- Charnine, M., A. Klokov, L. Kochiev, and A. Tishchenko. 2021. Research trending topic prediction as cognitive enhancement. Conference (International) on Cyberworlds Proceedings. IEEE. 217-220. doi: 10.1109/CW52790.2021.00044.
- Charnine, M. M., and N. V. Somin. 2024. Metod avtomatizirovannoy otsenki dostovernosti al'ternativnykh utverzhdeniy v kollektsii nauchnykh statey na primere temy "okna Overtona" [A method for automated assessment of the reliability of alternative statements in a collection of scientific articles using the example of the topic "Overton windows"]. Iskusstvennyy intellekt i prinyatie resheniy [Artificial Intelligence and Decision Making] 1:118-128. doi: 10.14357/20718594240110. EDN: YJICCC.
[+] About this article
Title
MOVEMENT, VELOCITY, AND TRAJECTORIES OF KEYWORD REPRESENTATIONS IN THE VECTOR SPACE OF THE LANGUAGE MODEL
Journal
Systems and Means of Informatics
Volume 35, Issue 4, pp 60-72
Cover Date
2025-12-25
DOI
10.14357/08696527250405
Print ISSN
0869-6527
Publisher
Institute of Informatics Problems, Russian Academy of Sciences
Additional Links
Key words
rate of semantic change; evolutionary trajectories; vector space; static language model
Authors
M. M. Charnine  and N. V. Somin
Author Affiliations
Federal Research Center "Computer Science and Control", Russian Academy of Sciences, 44-2 Vavilov Str., Moscow 119333, Russian Federation
|