A Practical Guide to BERTopic for Transformer-Based Topic Modeling

has a variety of use circumstances within the pure language processing (NLP) area, resembling doc tagging, survey evaluation, and content material group. It falls underneath the realm of unsupervised studying method, making it a really cost-effective method that reduces the assets required to gather human-annotated information. We’ll dive deeper into BERTopic, a preferred python library for transformer-based matter modeling, to assist us course of monetary information quicker and reveal how the trending subjects change time beyond regulation.
BERTopic consists of 6 core modules that may be custom-made to swimsuit totally different use circumstances. On this article, we’ll study, experiment with every module individually and discover how they work collectively coherently to provide the top outcomes.

At a excessive stage, a typical BERTopic structure consists of:

• Embeddings: remodel textual content into vector representations (i.e. embeddings) that seize semantic that means utilizing sentence-transformer fashions.
• Dimensionality Discount: cut back the high-dimensional embeddings to a lower-dimensional area whereas preserving essential relationships, together with PCA, UMAP …
• Clustering: group related paperwork collectively primarily based on their embeddings with diminished dimensionality to type distinct subjects, together with HDBSCAN, Ok-Means algorithms …
• Vectorizers: after matter clusters are fashioned, vectorizers convert textual content into numerical options that can be utilized for matter evaluation, together with rely vectorizer, on-line vectorizer …
• c-TF-IDF: calculate significance scores for phrases inside and throughout matter clusters to determine key phrases.
• Illustration Mannequin: leverage semantic similarity between the embedding of candidate key phrases and the embedding of paperwork to seek out essentially the most consultant matter key phrases, together with KeyBERT, LLM-based methods …

Undertaking Overview

On this sensible software, we’ll use Topic Modeling to determine trending subjects in Apple monetary information. Utilizing NewsAPI, we acquire each day top-ranked Apple inventory information from Google Search and compile them right into a dataset of 250 paperwork, with every doc containing monetary information for one particular day. Nevertheless, this isn’t the primary focus of this text so be at liberty to exchange it with your individual dataset. The target is to display methods to remodel uncooked textual content paperwork containing prime Google search outcomes into significant matter key phrases and refine these key phrases to be extra consultant.

BERTopic’s 6 Basic Modules

1. Embeddings

BERTopic makes use of sentence transformer fashions as its first constructing block, changing sentences into dense vector representations (i.e. embeddings) that seize semantic meanings. These fashions are primarily based on transformer architectures like BERT and are particularly skilled to provide high-quality sentence embeddings. We then compute the semantic similarity between sentences utilizing cosine distance between the embeddings. Frequent fashions embody:

• all-MiniLM-L6-v2: light-weight, quick, good common efficiency
• BAAI/bge-base-en-v1.5: bigger mannequin with sturdy semantic understanding therefore offers a lot slower coaching and inference pace.

There are an enormous vary of pre-trained sentence transformers so that you can select from on the “Sentence Transformer” web site and Huggingface model hub. We will use a couple of traces of code to load a sentence transformer mannequin and encode the textual content sequences into excessive dimensional numerical embeddings.

from sentence_transformers import SentenceTransformer

# Initialize mannequin
mannequin = SentenceTransformer("all-MiniLM-L6-v2")

# Convert sentences to embeddings
sentences = ["First sentence", "Second sentence"]
embeddings = mannequin.encode(sentences)  # Returns numpy array of embeddings

On this occasion, we enter a set of economic information information from October 2024 to March 2025 into the sentence transformer “bge-base-en-v1.5”. As proven within the outcome under. these textual content paperwork are reworked into vector embedding with the form of 250 rows and every with 384 dimensions.

We will then feed this sentence transformer to BERTopic pipeline and maintain all different modules because the default settings.

from sentence_transformers import SentenceTransformer
from bertopic import BERTopic

emb_minilm = SentenceTransformer("all-MiniLM-L6-v2")
topic_model = BERTopic(
    embedding_model=emb_minilm,
)

topic_model.fit_transform(docs)
topic_model.get_topic_info()

As the top outcome, we get the next matter illustration.

In comparison with the extra highly effective and bigger “bge-base-en-v1.5” mannequin, we get the next outcome which is barely extra significant than the smaller “all-MiniLM-L6-v2” mannequin however nonetheless leaves giant room for enchancment.

One space for enchancment is lowering the dimensionality, as a result of sentence transformers usually leads to high-dimensional embeddings. As BERTopic depends on evaluating the spatial proximity between embedding area to type significant clusters, it’s essential to use a dimensionality discount method to make the embeddings much less sparse. Due to this fact, we’re going to introduce varied dimensionality discount methods within the subsequent part.

2. Dimensionality Discount

After changing the monetary information paperwork into embeddings, we face the issue of excessive dimensionality. Since every embedding incorporates 384 dimensions, the vector area turns into too sparse to create significant distance measurement between two vector embeddings. Principal Element Evaluation (PCA) and Uniform Manifold Approximation and Projection (UMAP) are frequent methods to scale back dimensionalities whereas preserving the utmost variance within the information. We’ll have a look at UMAP, BERTopic’s default dimensionality discount method, in additional particulars. It’s a non-linear algorithm adopted from topology evaluation that seeks various construction inside the information. It really works by extending a radius outwards from every information level and connecting factors with its shut neighbors. You may dive extra into the UMAP visualization on this web site “Understanding UMAP“.

UMAP n_neighbours Experimentation

An essential UMAP parameter is n_neighbours that controls how UMAP balances native and international construction within the information. Low values of n_neighbors will pressure UMAP to focus on native construction, whereas giant values will have a look at bigger neighborhoods of every level.
The diagram under exhibits a number of scatterplots demonstrating the impact of various n_neighbors values, with every plot visualizing the embeddings in an 2-dimensional area after making use of UMAP dimensionality discount.

With smaller n_neighbors values (e.g. n=2, n=5), the plots present extra tightly coupled micro clusters, indicating a give attention to native construction. As n_neighbors will increase (in direction of n=100, n=150), the factors type extra cohesive international patterns, demonstrating how bigger neighborhood sizes assist UMAP seize broader relationships within the information.

UMAP min_dist Experimentation

The min_dist parameter in UMAP controls how tightly factors are allowed to be packed collectively within the decrease dimensional illustration. It units the minimal distance between factors within the embedding area. A smaller min_dist permits factors to be packed very intently collectively whereas a bigger min_dist forces factors to be extra scattered and evenly unfold out. The diagram under exhibits an experimentation on min_dist worth from 0.0001 to 1 when setting the n_neighbors=5. When min_dist is ready to smaller values, UMAP emphasizes on preserving native construction whereas bigger values remodel the embeddings right into a round form.

We resolve to set n_neighbors=5 and min_dist=0.01 primarily based on the hyperparameter tuning outcomes, because it kinds extra distinct information clusters which are simpler for the following clustering mannequin to course of.

import umap

UMAP_N = 5
UMAP_DIST = 0.01
umap_model = umap.UMAP(
    n_neighbors=UMAP_N,
    min_dist=UMAP_DIST, 
    random_state=0
)

3. Clustering

Following the dimensionality discount module, it’s the method of grouping embeddings with shut proximity into clusters. This course of is prime to matter modeling, because it categorizes related textual content paperwork collectively by their semantic relationships. BERTopic employs HDBSCAN mannequin by default, which has the benefit in capturing buildings with various densities. Moreover, BERTopic supplies the pliability of selecting different clustering fashions primarily based on the character of the dataset, resembling Ok-Means (for spherical, equally-sized clusters) or agglomerative clustering (for hirerarchical clusters).

HDBSCAN Experimentation

We’ll discover how two essential parameters, min_cluster_size and min_samples, influence the conduct of HDBSCAN mannequin.
min_cluster_size determines the minimal variety of information factors allowed to type a cluster and clusters not assembly the edge are handled as outliers. When setting min_cluster_size too low, you may get many small, unstable clusters which is likely to be noise. If setting it too excessive, you may merge a number of clusters into one, dropping their distinct traits.

min_samples calculates the space between a degree and its k-th nearest neighbor, figuring out how strict the cluster formation course of is. The bigger the min_samples worth, the extra conservative the clustering turns into, as clusters can be restricted to type in dense areas, classifying sparse factors as noise.

Condensed Tree is a helpful method to assist us resolve acceptable values of those two parameters. Clusters that persist for a wide range of lambda values (proven because the left vertical axis in a condense tree plot) are thought of steady and extra significant. We want the chosen clusters to be each tall (extra steady) and broad (giant cluster measurement). We use condensed_tree_ from HDBSCAN to check min_cluster_size from 3 to 50, then visualize the info factors of their vector area, colour coded by the expected cluster labels. As we progress by means of totally different min_cluster_size, we will determine optimum values that group shut information factors collectively.

On this experimentation, we chosen min_cluster_size=15 because it generates 4 clusters (highlighted in pink within the condensed tree plot under) with good stability and cluster measurement. Moreover the scatterplot additionally signifies affordable cluster formation primarily based on proximity and density.

We then perform an analogous train to check min_samples from 1 to 80 and chosen min_samples=5. As you possibly can observe from the visuals, the parameters min_samples and min_cluster_size exert distinct impacts on the clustering course of.

import hdbscan

MIN_CLUSTER _SIZE= 15
MIN_SAMPLES = 5
clustering_model = hdbscan.HDBSCAN(
    min_cluster_size=MIN_CLUSTER_SIZE,
    metric='euclidean',
    cluster_selection_method='eom',
    min_samples=MIN_SAMPLES,
    random_state=0
)

topic_model = BERTopic(
    embedding_model=emb_bge,
    umap_model=umap_model,
    hdbscan_model=clustering_model, 
)

topic_model.fit_transform(docs)
topic_model.get_topic_info()

Ok-Means Experimentation

In comparison with HDBSCAN, utilizing Ok-Means clustering permits us to generate extra granular subjects by specifying the n_cluster parameter, consequently, controlling the variety of subjects generated from the textual content paperwork.

This picture exhibits a sequence of scatter plots demonstrating totally different clustering outcomes when various the variety of clusters (n_cluster) from 3 to 50 utilizing Ok-Means. With n_cluster=3, the info is split into simply three giant teams. As n_cluster will increase (5, 8, 10, and many others.), the info factors are break up into extra granular groupings. General, it kinds rounded-shape clusters in comparison with HDBSCAN. We chosen n_cluster=8 the place the clusters are neither too broad (dropping essential distinctions) nor too granular (creating synthetic divisions). Moreover, it’s a correct amount of subjects for categorizing 250 days of economic information. Nevertheless, be at liberty to regulate the code snippet to your necessities if must determine extra granular or broader subjects.

from sklearn.cluster import KMeans

N_CLUSTER = 8
clustering_model = KMeans(
    n_clusters=N_CLUSTER,
    random_state=0
)

topic_model = BERTopic(
    embedding_model=emb_bge,
    umap_model=umap_model,
    hdbscan_model=clustering_model, 
)

topic_model.fit_transform(docs)
topic_model.get_topic_info()

Evaluating the subject cluster outcomes of Ok-Means and HDBSCAN reveals that Ok-Means produces extra distinct and significant matter representations. Nevertheless, each strategies nonetheless generate many cease phrases, indicating that subsequent modules are important to refine the subject representations.

4. Vectorizer

Earlier modules serve the function of grouping paperwork into semantically related clusters, and ranging from this module the primary focus is to fine-tune the subjects by selecting extra consultant and significant key phrases. BERTopic affords varied Vectorizer choices from the fundamental CountVectorizer to extra superior OnlineCountVectorizer which incrementally replace matter representations. For this train, we’ll experiment on CountVectorizer, a textual content processing software that creates a matrix of token counts out of a set of paperwork. Every row within the matrix represents a doc and every column represents a time period from the vocabulary, with the values displaying what number of instances every time period seems in every doc. This matrix illustration allows machine studying algorithms to course of the textual content information mathematically.

Vectorizer Experimentation

We’ll undergo a couple of essential parameters of the CountVectorizer and see how they could have an effect on the subject representations.

• ngram_range specifies what number of phrases to mix collectively into matter phrases. It’s significantly helpful for paperwork consists of brief phrases, which isn’t wanted on this scenario.
  instance output if we set ngram_range=(1, 3)

0                -1_apple nasdaq aapl_apple stock_apple nasdaq_nasdaq aapl   
1  0_apple warren buffett_apple stock_berkshire hathaway_apple nasdaq aapl   
2           1_apple nasdaq aapl_nasdaq aapl apple_apple stock_apple nasdaq   
3              2_apple aapl stock_apple nasdaq aapl_apple stock_aapl inventory   
4           3_apple nasdaq aapl_cramer apple aapl_apple nasdaq_apple inventory 

• stop_words determines whether or not cease phrases are faraway from the subjects, which considerably improves matter representations.
• min_df and max_df determines the frequency thresholds for phrases to be included within the vocabulary. min_df units the minimal variety of paperwork a time period should seem whereas max_df units the utmost doc frequency above which phrases are thought of too frequent and discarded.

We discover the impact of including CountVectorizer with max_df=0.8 (i.e. ignore phrases showing in additional than 80% of the paperwork) to each HDBSCAN and Ok-Means fashions from the earlier step.

from sklearn.feature_extraction.textual content import CountVectorizer
vectorizer_model = CountVectorizer(
		max_df=0.8, 
		stop_words="english"
)

topic_model = BERTopic(
    embedding_model=emb_bge,
    umap_model=umap_model,
    hdbscan_model=clustering_model, 
    vectorizer_model=vectorizer_model
)

Each exhibits enhancements after introducing the CountVectorizer, considerably lowering key phrases continuously appeared in all paperwork and never bringing further values, resembling “appl”, “inventory”, and “apple”.

5. c-TF-IDF

Whereas the Vectorizer module focuses on adjusting the subject illustration on the doc stage, c-TF-IDF primarily have a look at the cluster stage to scale back continuously encountered subjects throughout clusters. That is achieved by changing all paperwork belonging to 1 cluster as a single doc and calculated the key phrase significance primarily based on the normal TF-IDF method.

c-TF-IDF Experimentation

• reduce_frequent_words: determines whether or not to down-weight continuously occurring phrases throughout subjects
• bm25_weighting: when set to True, makes use of BM25 weighting as an alternative of normal TF-IDF, which may help higher deal with doc size variations. In smaller datasets, this variant might be extra strong to cease phrases.

We use the next code snippet so as to add c-TF-IDF (with bm25_weighting=True) into our BERTopic pipeline.

from bertopic.vectorizers import ClassTfidfTransformer

ctfidf_model = ClassTfidfTransformer(bm25_weighting=True)
topic_model = BERTopic(
    embedding_model=emb_bge,
    umap_model=umap_model,
    hdbscan_model=clustering_model, 
    vectorizer_model=vectorizer_model,
    ctfidf_model=ctfidf_model
)

The subject cluster outputs under present that including c-TF-IDF has no main influence to the top outcomes when CountVectorizer has already been added. That is probably as a result of our CountVectorizer has already set a excessive bar of eliminating phrases showing in additional than 80% on the doc stage. Subsequently, this already reduces overlapping vocabularies on the matter cluster stage, which is what c-TF-IDF is meant to realize.

Nevertheless, If we exchange CountVectorizer with c-TF-IDF, though the outcome under exhibits slight enhancements in comparison with when each will not be added, there are too many cease phrases current, making the subject representations much less precious. Due to this fact, it seems that for the paperwork we’re coping with on this state of affairs, c-TF-IDF module doesn’t deliver further worth.

6. Illustration Mannequin

The final module is the illustration mannequin which has been noticed having a big influence on tuning the subject representations. As a substitute of utilizing the frequency primarily based method like Vectorizer and c-TF-IDF, it leverages semantic similarity between the embeddings of candidate key phrases and the embeddings of paperwork to seek out essentially the most consultant matter key phrases. This can lead to extra semantically coherent matter representations and lowering the variety of synonymically related key phrases. BERTopic additionally affords varied customization choices for illustration fashions, together with however not restricted to the next:

• KeyBERTInspired: make use of KeyBERT method to extract matter phrases primarily based semantic similarity.
• ZeroShotClassification: take advantage of open-source transformers within the Huggingface model hub to assign labels to subjects.
• MaximalMarginalRelevance: lower synonyms in subjects (e.g. inventory and shares).

KeyBERTInspired Experimentation

We discovered that KeyBERTInspired is a really cost-effective method because it considerably improves the top outcome by including a couple of further traces of code, with out the necessity of in depth hyperparameter tuning.

from bertopic.illustration import KeyBERTInspired

representation_model = KeyBERTInspired()

topic_model = BERTopic(gh
    embedding_model=emb_bge,
    umap_model=umap_model,
    hdbscan_model=clustering_model, 
    vectorizer_model=vectorizer_model,
    representation_model=representation_model
)

After incorporating the KeyBERT-Impressed illustration mannequin, we now observe that each fashions generate noticeably extra coherent and precious themes.

Take-Residence Message

This text explores BERTopic method and implementation for matter modeling, detailing its six key modules with sensible examples utilizing Apple inventory market information information to display every part’s influence on the standard of matter representations.

• Embeddings: use transformer-based embedding fashions to transform paperwork into numerical representations that seize semantic that means and contextual relationships in textual content.
• Dimensionality Discount: make use of UMAP or different dimensionality discount methods to scale back high-dimensional embeddings whereas preserving each native and international construction of the info
• Clustering: evaluate HDBSCAN (density-based) and Ok-Means (centroid-based) clustering algorithm to group related paperwork into coherent subjects
• Vectorizers: use Depend Vectorizer to create document-term matrices and refine subjects primarily based on statistical method.
• c-TF-IDF: replace matter representations by analyzing time period frequency at cluster stage (matter class) and cut back frequent phrases throughout totally different subjects.
• Illustration Mannequin: refine matter key phrases utilizing semantic similarity, providing choices like KeyBERTInspired and MaximalMarginalRelevance for higher matter descriptions