What Makes a Language Look Like Itself?

communicate a phrase of a language, you’ll be able to typically guess what language you’re simply by recognizing sure character patterns. I don’t communicate Icelandic, for instance, however I immediately acknowledge Icelandic textual content after I see letters like “ð” or “þ”, as these characters are extraordinarily uncommon elsewhere. Likewise, I’ve seen that after I see numerous “ijk” in a textual content, it’s most likely Dutch.

This text explores how we will use easy statistics to be taught these visible fingerprints, the character sequences that almost all strongly sign which language you’re , throughout 20 totally different European languages.

Be taught Linguistic Fingerprints With Statistics

To be taught the visible “fingerprints” of a language, we first want a technique to measure how distinctive a given character sample is. A pure place to begin is perhaps to take a look at the most typical character patterns inside every language. Nonetheless, this strategy rapidly falls brief as a personality sample is perhaps quite common in a single language and in addition seem very steadily in others. Frequency alone doesn’t seize uniqueness. As an alternative, we need to ask:

“How more likely is that this sample to seem in a single language in comparison with all others?”

That is the place statistics is available in! Formally, let:

• L be the set of all 20 studied languages
• S be the set of all noticed character patterns throughout these languages

To find out how strongly a given character sample s∈S identifies a language l∈L, we compute the probability ratio:

[LR_{s,l} = fracl)neg l)]

This compares the likelihood of seeing character sample s in language l, versus in some other language. The upper the ratio, the extra uniquely tied that sample is to that language.

Calculating the Probability Ratio in Apply

To compute the probability ratio for every character sample in follow, we have to translate the conditional possibilities into portions we will really measure. Right here’s how we outline
the related counts:

• c_l(s): the variety of occasions character sample s seems in language l
• c_¬l(s): the variety of occasions character sample s seems in all different languages
• N_l: the whole variety of character sample occurrences in language l
• N_¬l: the whole variety of sample occurrences in all different languages

Utilizing these, the conditional possibilities grow to be:

[P(s|l) = frac{c_l(s)}{N_l},~P(s|neg l)=frac{c_{neg l}(s)}{N_{neg l}}]

and the probability ratio simplifies to:

[LR_{s,l} = fracl)neg l) = frac{c_l(s)cdot N_{neg l}}{c_{neg l}(s)cdot N_l}]

This provides us a numeric rating that quantifies how more likely a personality sample s is to seem in language l versus all others.

Dealing with Zero Counts

There’s sadly an issue with our probability ratio method: what occurs when c_¬l(s) = 0?

In different phrases, what if a sure character sample s seems solely in language l and in no others? This results in a divide-by-zero within the denominator, and a probability ratio of infinity.

Technically, this implies we’ve discovered a superbly distinctive sample for that language. However in follow, it’s not very useful. A personality sample would possibly solely seem as soon as in a single language and by no means wherever else and it could be given an infinite rating. Not very helpful as a robust “fingerprint” of the language.

To keep away from this subject, we apply a method known as additive smoothing. This methodology adjusts the uncooked counts barely to remove zeros and cut back the influence of uncommon occasions.

Particularly, we add a small fixed α to each depend within the numerator, and α|S| to the denominator, with |S| being the whole variety of noticed character patterns. This has the impact of assuming that each character sample has a tiny probability of occurring in each language, even when it hasn’t been seen but.

With smoothing, the adjusted possibilities grow to be:

[P'(s|l) = frac{c_l(s) + alpha},~P'(s|neg l)=frac{c_{neg l}(s) + alpha}{N_{neg l} + alpha|S|}]

And the ultimate probability ratio to be maximized is:

[LR_{s,l} = fracl)neg l) = frac{(c_l(s) + alpha)cdot(N_{neg l} + alpha|S|)}{(N_l + alpha|S|)cdot(c_{neg l}(s) + alpha)}]

This retains issues steady and ensures {that a} uncommon sample doesn’t routinely dominate simply because it’s unique.

The Dataset

Now that we’ve outlined a metric to determine probably the most distinctive character patterns (our linguistic “fingerprints”), it’s time to collect precise language knowledge to investigate.

For this, I used the python library wordfreq, which compiles phrase frequency lists for dozens of languages based mostly on large-scale sources like Wikipedia, books, subtitles, and net textual content.

One significantly helpful operate for this evaluation is top_n_list(), which returns a sorted checklist of the highest n highest frequency phrases in a supplied language. For instance, to get the highest 40 commonest phrases in Icelandic, we’d name:

wordfreq.top_n_list("is", 40, ascii_only=False)

The argument ascii_only=False ensures that non-ASCII characters — like Icelandic’s “ð” and “þ” — are preserved within the output. That’s important for this evaluation, since we’re particularly in search of language-unique character patterns, which incorporates single characters.

To construct the dataset, I pulled the highest 5,000 most frequent phrases in every of the next 20 European languages:

Catalan, Czech, Danish, Dutch, English, Finnish, French, German, Hungarian, Icelandic, Italian, Latvian, Lithuanian, Norwegian, Polish, Portuguese, Romanian, Spanish, Swedish, and Turkish.

This yields a big multilingual vocabulary of 100,000 whole phrases, wealthy sufficient to extract significant statistical patterns throughout languages.

To extract the character patterns used within the evaluation, all attainable substrings of size 1 to five had been generated from every phrase within the dataset. For instance, the phrase language would include patterns akin to l, la, lan, lang, langu, a, an, ang, and so forth. The result’s a complete set S of over 180,000 distinctive character patterns noticed throughout the 20 studied languages.

Outcomes

For every language, the highest 5 most distinctive character patterns, ranked by the probability ratio, are proven. The smoothing fixed was chosen to be α=0.5.

As a result of the uncooked probability ratios will be fairly giant, I’ve reported the base-10 logarithm of the probability ratio (log₁₀(LR)) as an alternative. For instance, a log probability ratio of three signifies that the character sample is 10³ = 1,000 occasions extra prone to seem in that language than in some other. Be aware that attributable to smoothing, these probability ratios are approximate somewhat than precise, and the extremeness of some scores could also be dampened.

Every cell reveals a top-ranked character sample and its log probability ratio.

Dialogue

Under are some attention-grabbing interpretations of the outcomes. This isn’t meant to be a complete evaluation, just some observations I discovered noteworthy:

• Lots of the character patterns with the best probability ratios are single characters distinctive to their language, such because the beforehand talked about Icelandic “ð” and “þ”, Romanian’s “ă”, “ț”, and “ș”, or Turkish’s “ı”, “ş”, and “ğ”. As a result of these characters are primarily absent from all different languages within the dataset, they might have produced infinite probability ratios if not for the additive smoothing.
• In some languages, most notably Dutch, lots of the high outcomes are substrings of each other. For instance, the highest sample “ijk” additionally seems inside the subsequent highest-ranking patterns: “lijk”, “elijk”, and “ijke”. This reveals how sure mixtures of letters are reused steadily in longer phrases, making them much more distinctive for that language.
• English has among the least distinctive character patterns of all of the languages analyzed, with a most log probability ratio of solely 2.79. This can be as a result of presence of English loanwords in lots of different languages’ high 5,000 phrase lists, which dilutes the individuality of English-specific patterns.
• There are a number of instances the place the highest character patterns mirror shared grammatical buildings throughout languages. For instance, the Spanish “-ción”, Italian “-zione”, and Norwegian “-sjon” all operate as nominalization suffixes, just like the English “-tion” — turning verbs or adjectives into nouns. These endings stand out strongly in every language and spotlight how totally different languages can observe related patterns utilizing totally different spellings.

Conclusion

This challenge began with a easy query: What makes a language appear like itself? By analyzing the 5000 commonest phrases in 20 European languages and evaluating the character patterns they use, we uncovered distinctive ‘fingerprints’ for every language — from accented letters like “ş” and “ø” to recurring letter mixtures like “ijk” or “ción”. Whereas the outcomes aren’t meant to be definitive, they provide a enjoyable and statistically grounded technique to discover what units languages aside visually, even with out understanding a single phrase.

See my GitHub Repository for the total code implementation of this system.

Thanks for studying!

References

wordfreq python library:

• Robyn Speer. (2022). rspeer/wordfreq: v3.0 (v3.0.2). Zenodo. https://doi.org/10.5281/zenodo.7199437