Effects of Tool Description Language on Language Model's Tool Calling in Korean Dialogue

Abstract

Tool calling has become a key component that allows language models (LMs) to interact with the real-world. In particular, clearly written tool descriptions guide LMs to select the appropriate tool and extract correct arguments from user queries. However, the effect of the language used in tool descriptions on tool calling in LM, especially in non-English conversation contexts, has been underexplored. This study investigates how the language of tool descriptions (e.g., English vs. Korean) affects tool calling in LMs in Korean dialogues. Using FunctionChat-Bench, we evaluated various LMs, including multilingual, fine-tuned in Korean, and non-supported Korean. The results showed that large multilingual LMs performed well regardless of the language in tool description, while small low-performing LMs tended to perform better with tool descriptions written in Korean. These findings underscore the importance of linguistic consistency between user queries and tool descriptions, particularly for small LMs with lower multilingual proficiency.