Measuring Semantic Similarity in Bilinguals and Language Models

Many studies have provided evidence that bilinguals systematically differ from monolinguals across a wide range of cognitive phenomena. The specific patterns are complex, with these bilingual adaptations sometimes playing out as advantages and sometimes as disadvantages. The exact origins of these effects remains unclear. In a recent meta-analysis, we have advanced the understanding of these effects by showing that the similarity between the bilinguals’ two languages can act as a modulator of these bilingual adaptations. However, current metrics of language similarity only take into account the similarity of the lexicon or morpho-syntactic similarities. A crucial component, cross-language semantic similarity, remains unaddressed. The present project (MESSI) aims to establish such a measure. Due to their scalability across many different languages, it would be desirable to automatize this measurement with Large Language Models (LLMs). However, the reliability and validity of semantic similarity measures derived from LLMs across different languages needs to be established first. To this end, we will systematically compare LLMs’ semantic similarities, within and across languages, to analogous similarity data derived from monolingual and bilingual human speakers of eight different languages. In the next step, we then examine how the obtained cross-language semantic similarity measures modulate bilingual adaptations.