How I Won the “Mostly AI” Synthetic Data Challenge

I within the Mostly AI Prize and gained each the FLAT and SEQUENTIAL information challenges. The competitors was a improbable studying expertise, and on this submit, I wish to present some insights into my successful resolution.

The Competitors

The objective of the competitors was to generate an artificial dataset with the identical statistical properties as a supply dataset, with out copying the info.

The competitors was cut up into two unbiased challenges:

1. FLAT Information Problem: Generate 100,000 information with 80 columns.
2. SEQUENTIAL Information Problem: Generate 20,000 sequences (teams) of information.

To measure the standard of the artificial information, the competitors used an General Accuracy metric. This rating measures the similarity between the artificial and supply distributions for single columns (univariates), pairs of columns (bivariates), and triples of columns (trivariates) utilizing the L1 distance. Moreover, privateness metrics like DCR (Distance to Closest File) and NNDR (Nearest Neighbor Distance Ratio) have been used to make sure submissions weren’t simply overfitting or copying the coaching information.

Answer Design

Initially, my objective was to create an ensemble of a number of completely different state-of-the-art fashions and mix their generated information. I experimented rather a lot with completely different fashions, however the outcomes didn’t enhance as a lot as I had hoped.

I pivoted my strategy and centered on post-processing. First, I skilled a single generative mannequin from the Mostly AI SDK, and as a substitute of producing the required variety of samples for the submission, I oversampled to create a big pool of candidate samples. From this pool, I then chosen the ultimate output in a means that matches the statistical properties of the supply dataset rather more intently.

This strategy led to a considerable bounce within the leaderboard rating. For the FLAT information problem, the uncooked artificial information from the mannequin scored round 0.96, however after post-processing, the rating jumped to 0.992. I used a modified model of this strategy for the SEQUENTIAL information problem, which yielded an identical enchancment.

My ultimate pipeline for the FLAT problem consisted of three major steps:

1. Iterative Proportional Becoming (IPF) to pick out an outsized, high-quality subset.
2. Grasping Trimming to cut back the subset to the goal dimension by eradicating the worst-fitting samples.
3. Iterative Refinement to shine the ultimate dataset by swapping samples for higher becoming ones.

Step 1: Iterative Proportional Becoming (IPF)

Step one in my post-processing pipeline was to get a robust preliminary subset from the oversampled pool (2.5 million generated rows). For this, I used Iterative Proportional Fitting (IPF).

IPF is a classical statistical algorithm used to regulate a pattern distribution to match a recognized set of marginals. On this case, I needed the artificial information’s bivariate (2-column) distributions to match these of the unique information. I additionally examined uni- and trivariate distributions, however I discovered that specializing in the bivariate relationships yielded the very best efficiency whereas being computationally quick.

Right here’s the way it labored:

1. I recognized the 5,000 most correlated column pairs within the coaching information utilizing mutual data. These are an important relationships to protect.
2. IPF then calculated fractional weights for every of the two.5 million artificial rows. The weights have been adjusted iteratively in order that the weighted sums of the bivariate distributions within the artificial pool matched the goal distributions from the coaching information.
3. Lastly, I used an expectation-rounding strategy to transform these fractional weights into an integer rely of what number of instances every row must be chosen. This resulted in an outsized subset of 125,000 rows (1.25x the required dimension) that already had very robust bivariate accuracy.

The IPF step offered a high-quality start line for the subsequent section.

Step 2: Trimming

Producing an outsized subset of 125,000 rows from IPF was a deliberate alternative that enabled this extra trimming step to take away samples that didn’t match nicely.

I used a grasping strategy that iteratively calculates the “error contribution” of every row within the present subset. The rows that contribute essentially the most to the statistical distance from the goal distribution are recognized and eliminated. This course of repeats till solely 100,000 rows stay, making certain that the worst 25,000 rows are discarded.

Step 3: Refinement (Swapping)

The ultimate step was an iterative refinement course of to swap rows from the subset with higher rows from the a lot bigger, unused information pool (the remaining 2.4 million rows).

In every iteration, the algorithm:

1. Identifies the worst rows throughout the present 100k subset (these contributing most to the L1 error).
2. Searches for the very best alternative candidates from the skin pool that would scale back the L1 error if swapped in.
3. Performs the swap if it leads to a greater general rating.

Because the accuracy of the artificial pattern is already fairly excessive, the extra acquire from this course of is quite small.

Adapting for the Sequential Problem

The SEQUENTIAL problem required an identical strategy, however with two modifications. First, a pattern consists of a number of rows, linked by the group ID. Secondly, the competitors metric provides a measure for coherence. This implies not solely do the statistical distributions have to match, however the sequences of occasions additionally should be much like the supply dataset.

My post-processing pipeline was tailored to deal with teams and in addition optimize for coherence:

1. Coherence-Based mostly Pre-selection: Earlier than optimizing for statistical accuracy, I ran a specialised refinement step. This algorithm iteratively swapped whole teams (sequences) to particularly match the coherence metrics of the unique information, such because the distribution of “distinctive classes per sequence” and “sequences per class”. This ensured that we continued the post-processing with a sound sequential construction.
2. Refinement (Swapping): The 20,000 teams chosen for coherence then went by means of the identical statistical refinement course of because the flat information. The algorithm swapped whole teams with higher ones from the pool to reduce the L1 error of the uni-, bi-, and trivariate distributions. A secret ingredient was to incorporate the “Sequence Size” as a characteristic, so the group lengths are additionally thought-about within the swapping.

This two-stage strategy ensured the ultimate dataset was robust in each statistical accuracy and sequential coherence. Curiously, the IPF-based strategy that labored so nicely for the flat information was much less efficient for the sequential problem. Subsequently, I eliminated it to focus computing time on the coherence and swapping algorithms, which yielded higher outcomes.

Making It Quick: Key Optimizations

The post-processing technique by itself was computationally costly, and making it run throughout the competitors time restrict was a problem in itself. To succeed, I relied on a number of key optimizations.

First, I decreased the info varieties wherever doable to deal with the large pattern information pool with out operating out of reminiscence. Altering the numerical sort of a giant matrix from 64-bit to 32 or 16-bit enormously reduces the reminiscence footprint.

Secondly, when altering the info sort was not sufficient, I used sparse matrices from SciPy. This method allowed me to retailer the statistical contributions of every pattern in an extremely memory-efficient means.

Lastly, the core refinement loop concerned lots of specialised calculations, a few of which have been very sluggish with numpy. To beat this, I used <robust>numba</robust>. By extracting the bottlenecks in my code into specialised capabilities with the @numba.njit decorator, Numba routinely translated them into extremely optimized machine code that runs at speeds similar to C.

Right here is an instance of how I wanted to hurry up the summation of rows in sparse matrices, which was a significant bottleneck within the authentic NumPy model.

import numpy as np
import numba

# This may make the logic run a whole bunch of instances quicker.
@numba.njit
def _rows_sum_csr_int32(information, indices, indptr, rows, Okay):
    """
    Sum CSR rows right into a dense 1-D vector with out creating
    intermediate scipy / numpy objects.
    """
    out = np.zeros(Okay, dtype=np.int32)
    for r in rows:
        begin = indptr[r]
        finish = indptr[r + 1]
        for p in vary(begin, finish):
            out[indices[p]] += information[p]
    return out

Nevertheless, Numba isn’t a silver bullet; it’s useful for numerical, loop-heavy code, however for many calculations, it’s quicker and simpler to stay to vectorized NumPy operations. I counsel you to solely strive it when a NumPy strategy doesn’t attain the required velocity.

Remaining Ideas

Though ML fashions are getting more and more stronger, I believe that for many issues that Information Scientists try to resolve, the key ingredient is usually not within the mannequin. In fact, a robust mannequin is an integral a part of an answer, however the pre- and postprocessing are equally vital. For these challenges, a post-processing pipeline focused particularly for the analysis metric led me to the successful resolution, with none extra ML.

I discovered rather a lot on this problem, and I wish to thank Mostly AI and the jury for his or her nice job in organizing this improbable competitors.

My code and options for each challenges are open-source and will be discovered right here: