A Review of AccentFold: One of the Most Important Papers on African ASR

I loved studying this paper, not as a result of I’ve met a number of the authors earlier than🫣, however as a result of it felt obligatory. Many of the papers I’ve written about to date have made waves within the broader ML neighborhood, which is nice. This one, although, is unapologetically African (i.e. it solves a really African drawback), and I feel each African ML researcher, particularly these focused on speech, must learn it.

AccentFold tackles a selected situation many people can relate to: present Asr programs simply don’t work effectively for African-accented English. And it’s not for lack of making an attempt.

Most present approaches use strategies like multitask studying, area adaptation, or positive tuning with restricted knowledge, however all of them hit the identical wall: African accents are underrepresented in datasets, and gathering sufficient knowledge for each accent is pricey and unrealistic.

Take Nigeria, for instance. We now have lots of of native languages, and many individuals develop up talking multiple. So after we communicate English, the accent is formed by how our native languages work together with it — by way of pronunciation, rhythm, and even switching mid-sentence. Throughout Africa, this solely will get extra complicated.

As a substitute of chasing extra knowledge, this paper provides a wiser workaround: it introduces AccentFold, a technique that learns accent Embeddings from over 100 African accents. These embeddings seize deep linguistic relationships (phonological, syntactic, morphological), and assist ASR programs generalize to accents they’ve by no means seen.

That concept alone makes this paper such an vital contribution.

Associated Work

One factor I discovered fascinating on this part is how the authors positioned their work inside current advances in probing language fashions. Earlier analysis has proven that pre skilled speech fashions like DeepSpeech and XLSR already seize linguistic or accent particular data of their embeddings, even with out being explicitly skilled for it. Researchers have used this to research language variation, detect dialects, and enhance ASR programs with restricted labeled knowledge.

AccentFold builds on that concept however takes it additional. Probably the most carefully associated work additionally used mannequin embeddings to help accented ASR, however AccentFold differs in two vital methods.

• First, quite than simply analyzing embeddings, the authors use them to information the choice of coaching subsets. This helps the mannequin generalize to accents it has not seen earlier than.
• Second, they function at a a lot bigger scale, working with 41 African English accents. That is practically twice the dimensions of earlier efforts.

The Dataset

The authors used AfriSpeech 200, a Pan African speech corpus with over 200 hours of audio, 120 accents, and greater than 2,000 distinctive audio system. One of many authors of this paper additionally helped construct the dataset, which I feel is de facto cool. In response to them, it’s the most numerous dataset of African accented English accessible for ASR to date.

What stood out to me was how the dataset is break up. Out of the 120 accents, 41 seem solely within the take a look at set. This makes it excellent for evaluating zero shot generalization. For the reason that mannequin is rarely skilled on these accents, the take a look at outcomes give a transparent image of how effectively it adapts to unseen accents.

What AccentFold Is

Like I discussed earlier, AccentFold is constructed on the thought of utilizing realized accent embeddings to information adaptation. Earlier than going additional, it helps to elucidate what embeddings are. Embeddings are vector representations of complicated knowledge. They seize construction, patterns, and relationships in a means that lets us examine completely different inputs — on this case, completely different accents. Every accent is represented as a degree in a excessive dimensional area, and accents which can be linguistically or geographically associated are typically shut collectively.

What makes this convenient is that AccentFold doesn’t want specific labels to know which accents are related. The mannequin learns that by way of the embeddings, which permits it to generalize even to accents it has not seen throughout coaching.

How AccentFold Works

The best way it really works is pretty simple. AccentFold is constructed on high of a giant pre skilled speech mannequin known as XLSR. As a substitute of coaching it on only one activity, the authors use multitask studying, which implies the mannequin is skilled to do a number of various things without delay utilizing the identical enter. It has three heads:

1. An ASR head for Speech Recognition, changing speech to textual content. That is skilled utilizing CTC loss, which helps match audio to the proper phrase sequence.
2. An accent classification head for predicting the speaker’s accent, skilled with cross entropy loss.
3. A area classification head for figuring out whether or not the audio is medical or common, additionally skilled with cross entropy however in a binary setting.

Every activity helps the mannequin study higher accent representations. For instance, making an attempt to categorise accents teaches the mannequin to acknowledge how individuals communicate in another way, which is important for adapting to new accents.

After coaching, the mannequin creates a vector for every accent by averaging the encoder output. That is known as imply pooling, and the result’s the accent embedding.

When the mannequin is requested to transcribe speech from a brand new accent it has not seen earlier than, it finds accents with related embeddings and makes use of their knowledge to positive tune the ASR system. So even with none labeled knowledge from the goal accent, the mannequin can nonetheless adapt. That’s what makes AccentFold work in zero shot settings.

What Data Does AccentFold Seize

This part of the paper appears to be like at what the accent embeddings are literally studying. Utilizing a collection of tSNE plots, the authors discover whether or not AccentFold captures linguistic, geographical, and sociolinguistic construction. And truthfully, the visuals communicate for themselves.

1. Clusters Type, However Not Randomly

In Determine 2, every level is an accent embedding, coloured by area. You instantly discover that the factors aren’t scattered randomly. Accents from the identical area are likely to cluster. For instance, the pinkish cluster on the left represents West African accents like Yoruba, Igbo, Hausa, and Twi. On the higher proper, the orange cluster represents Southern African accents like Zulu, Xhosa, and Tswana.

What issues isn’t just that clusters kind, however how tightly they do. Some are dense and compact, suggesting inside similarity. Others are extra unfold out. South African Bantu accents are grouped very carefully, which suggests sturdy inside consistency. West African clusters are broader, doubtless reflecting the variation in how West African English is spoken, even inside a single nation like Nigeria.

2. Geography Is Not Simply Visible. It Is Spatial

Determine 3 reveals embeddings labeled by nation. Nigerian accents, proven in orange, kind a dense core. Ghanaian accents in blue are close by, whereas Kenyan and Ugandan accents seem removed from them in vector area.

There may be nuance too. Rwanda, which has each Francophone and Anglophone influences, falls between clusters. It doesn’t totally align with East or West African embeddings. This displays its combined linguistic identification, and reveals the mannequin is studying one thing actual.

3. Twin Accents Fall Between

Determine 4 reveals embeddings for audio system who reported twin accents. Audio system who recognized as Igbo and Yoruba fall between the Igbo cluster in blue and the Yoruba cluster in orange. Much more distinct mixtures like Yoruba and Hausa land in between.

This reveals that AccentFold isn’t just classifying accents. It’s studying how they relate. The mannequin treats accent as one thing steady and relational, which is what a great embedding ought to do.

4. Linguistic Households Are Bolstered and Typically Challenged
In Determine 9, the embeddings are coloured by language households. Most Niger Congo languages kind one massive cluster, as anticipated. However in Determine 10, the place accents are grouped by household and area, one thing sudden seems. Ghanaian Kwa accents are positioned close to South African Bantu accents.

This challenges widespread assumptions in classification programs like Ethnologue. AccentFold could also be choosing up on phonological or morphological similarities that aren’t captured by conventional labels.

5. Accent Embeddings Can Assist Repair Labels
The authors additionally present that the embeddings can clear up mislabeled or ambiguous knowledge. For instance:

• Eleven Nigerian audio system labeled their accent as English, however their embeddings clustered with Berom, a neighborhood accent.
• Twenty audio system labeled their accent as Pidgin, however had been positioned nearer to Ijaw, Ibibio, and Efik.

This implies AccentFold is just not solely studying which accents exist, but in addition correcting noisy or obscure enter. That’s particularly helpful for actual world datasets the place customers typically self report inconsistently.

Evaluating AccentFold: Which Accents Ought to You Decide

This part is considered one of my favorites as a result of it frames a really sensible drawback. If you wish to construct an ASR system for a brand new accent however don’t have knowledge for that accent, which accents do you have to use to coach your mannequin?

Let’s say you might be focusing on the Afante accent. You don’t have any labeled knowledge from Afante audio system, however you do have a pool of speech knowledge from different accents. Let’s name that pool A. Attributable to useful resource constraints like time, finances, and compute, you may solely choose s accents from A to construct your positive tuning dataset. Of their experiments, they repair s as 20, that means 20 accents are used to coach every goal accent. So the query turns into: which 20 accents do you have to select to assist your mannequin carry out effectively on Afante?

Setup: How They Consider

To check this, the authors simulate the setup utilizing 41 goal accents from the Afrispeech 200 dataset. These accents don’t seem within the coaching or growth units. For every goal accent, they:

• Choose a subset of s accents from A utilizing considered one of three methods
• Effective tune the pre skilled XLS R mannequin utilizing solely knowledge from these s accents
• Consider the mannequin on a take a look at set for that concentrate on accent
• Report the Phrase Error Charge, or WER, averaged over 10 epochs

The take a look at set is identical throughout all experiments and contains 108 accents from the Afrispeech 200 take a look at break up. This ensures a good comparability of how effectively every technique generalizes to new accents.

The authors take a look at three methods for choosing coaching accents:

1. Random Sampling: Decide s accents randomly from A. It’s easy however unguided.
2. GeoProx: Choose accents primarily based on geographical proximity. They use geopy to search out international locations closest to the goal and select accents from there.
3. AccentFold: Use the realized accent embeddings to pick the s accents most much like the goal in illustration area.

Desk 1 reveals that AccentFold outperforms each GeoProx and Random sampling throughout all 41 goal accents.

This leads to a few 3.5 % absolute enchancment in WER in comparison with random choice, which is significant for low useful resource ASR. AccentFold additionally has decrease variance, that means it performs extra constantly. Random sampling has the very best variance, making it much less dependable.

Does Extra Knowledge Assist

The paper asks a basic machine studying query: does efficiency hold bettering as you add extra coaching accents?

Determine 5 reveals that WER improves as s will increase, however solely up to some extent. After about 20 to 25 accents, the efficiency ranges off.

So extra knowledge helps, however solely to some extent. What issues most is utilizing the suitable knowledge.

Key Takeaways

• AccentFold addresses an actual African drawback: ASR programs typically fail on African accented English resulting from restricted and imbalanced datasets.
• The paper introduces accent embeddings that seize linguistic and geographic similarities without having labeled knowledge from the goal accent.
• It formalizes a subset choice drawback: given a brand new accent with no knowledge, which different accents do you have to practice on to get the most effective outcomes?
• Three methods are examined: random sampling, geographical proximity, and AccentFold utilizing embedding similarity.
• AccentFold outperforms each baselines, with decrease Phrase Error Charges and extra constant outcomes
• Embedding similarity beats geography. The closest accents in embedding area aren’t at all times geographically shut, however they’re extra useful.
• Extra knowledge helps solely up to some extent. Efficiency improves at first, however ranges off. You don’t want all the info, simply the suitable accents.
• Embeddings may also help clear up noisy or mislabeled knowledge, bettering dataset high quality.
• Limitation: outcomes are primarily based on one pre skilled mannequin. Generalization to different fashions or languages is just not examined.
• Whereas this work focuses on African accents, the core technique — studying from what fashions already know — may encourage extra common approaches to adaptation in low-resource settings.

Supply Be aware:
This text summarizes findings from the paper AccentFold: A Journey by way of African Accents for Zero Shot ASR Adaptation to Goal Accents by Owodunni et al. (2024). Figures and insights are sourced from the unique paper, accessible at https://arxiv.org/abs/2402.01152.