Cookie settings

Choose what to allow. The site always stores what it needs; everything else stays off until you turn it on.

Choice

AI chatbots in Armenian: what they understand and where they fail

  • 5 November 2026
  • 5 min read
  • Harutyun Elibekyan

An AI chatbot can read and write Armenian, but the quality depends on the model, and general leaderboards won’t tell you how good it is. In a 2026 benchmark for Armenian, Gemini 3 Flash scored higher on Armenian tasks than Gemini 3 Pro. The authors titled that finding “Better Globally ≠ Better for Armenian”: a model that’s stronger overall isn’t necessarily stronger in Armenian. So test a bot on your own questions, in different writing styles, before you put it in front of customers.

What we know about model quality in Armenian

There isn’t much data, so here is everything I found. The first source is ArmBench-LLM 1.0 from Metric AI Lab, published on the Hugging Face blog on April 2, 2026. The tasks include text classification, multiple-choice questions, grammar correction, summarization, translation, reading comprehension, translated MMLU-Pro questions, and tasks on Armenian history and literature. Gemini 3 Flash had the highest average score, 0.6350; the lowest score in the post was 0.5767. That isn’t a rating of chatbot conversations but an average over test tasks. I couldn’t find independent verification of this benchmark, and the authors note that a single simple prompt may have lowered the scores of smaller models.

The second source is Anthropic’s documentation on multilingual support. Armenian isn’t in its table, which lists 15 of the 42 languages in Global-MMLU. Anthropic says the model handles more languages than the table shows and recommends testing the ones you need. For a business, the conclusion is simple: you probably won’t find a public figure for how good a model is in Armenian, so you have to test it yourself.

What kind of Armenian will your customers write?

More than one. The bot has to cope with at least four writing styles, and it makes different mistakes in each.

Writing styleWhat we knowWhat to check
Eastern Armenian, Armenian alphabetSpoken mostly in Armenia. Reformed orthography was introduced in Soviet Armenia in 1922–1924Whether the bot replies in the same spelling, writes the dram sign ֏, and doesn’t garble numbers
Western Armenian, classical orthographyOriginally spoken mainly in what is now Turkey; after the genocide, spread across the diaspora. Western Armenians and Armenians in Iran kept the classical orthography. Wikipedia notes the two varieties become mutually intelligible after significant practiceWhether the bot mixes up forms: a diaspora customer might write differently
Armenian in Latin lettersWikipedia: unofficial Latin-only input methods “do not conform to any established national or international standard.” People write the same letter in different ways, like Թ as t’Whether the bot understands the meaning and what language it replies in
Mixed with Russian or EnglishIt’s something to test directlyWhether the bot loses track of the question when half the words are in another language

How the bot chooses a response language

By itself, from the customer’s message, unless you set a rule. Anthropic’s docs say the model detects the language during the conversation, but in a production app you should set the response language explicitly in the system prompt. They also advise sending text in its native script rather than transliteration. Don’t expect Telegram to help: in the Bot API the user object has an optional language code, but the message has no language field.

In practice, write the rule yourself: reply in the language of the latest message, and if the customer writes in Latin letters and the meaning isn’t clear, ask which language they prefer.

How to test a bot before launch

Take 30 real questions from your Telegram, WhatsApp, or Viber chats. Write each one in Armenian, in Russian, and in Latin script. Have someone who reads Armenian review the replies.

What you’re checkingWhat failure looks like
Response languageYou asked in Armenian, the bot answered in Russian
Business factsPrices, hours, or addresses don’t match your data or are made up
Handling unknownsInstead of “I don’t know, I’ll get a human”, the bot invents an answer
Latin scriptThe bot didn’t understand the question or answered in the wrong language
Writing mechanicsBroken spelling, missing or malformed ֏ sign, or mangled numbers

If the bot fails more than two of the thirty questions on any row of the table, fix the instructions and data first, then change the model if needed, and launch only after that. The two-question threshold is my own rule of thumb, not an industry standard.

What I do and how much it costs

Every chatbot and AI automation project of mine starts with an assessment, which takes 1–2 weeks, followed by a test run on your real data. We decide the response languages in advance (Armenian, Russian, or English) and test them on your questions before launch. A basic chatbot or automation starts at 130,000 ֏; complex projects are priced after the assessment. I don’t promise savings in advance. Before launch, we agree what success looks like, for example the share of questions the bot resolves without a human.

This is how I make a living. If you get only a few customer questions and they’re repetitive, a messenger button on your site is enough; my article on WhatsApp, Viber, and Telegram covers how to choose a channel. You need a bot when the same questions land in your inbox every day.

Sources

Related services

  • AI automation

    Routine work goes to software: reports and data move on their own, and an AI assistant answers common questions.

Talk to us

Ready to talk about your project?

Phone us, write to us or send a short brief. You get a clear next step and, after the first conversation, a written estimate.

Or reach us directly

  • Quick reply to your request or call
  • Languages Armenian, Russian, English
  • Written estimate after the first conversation

Animation is off because your system asks for reduced motion.