Exploratory Data Analysis: Gamma Spectroscopy in Python (Part 3)

objects round us could be barely radioactive. Americium in smoke detectors, radium in some classic watches, or uranium in classic glass; a full checklist could be lengthy. Principally, these objects are secure and can’t trigger a well being danger. Additionally it is attention-grabbing to determine them and examine the matter on the atomic stage. And we are able to do that utilizing a radiation detector. In the first part, I did an exploratory information evaluation of the gamma spectroscopy information. Within the second part, I created a machine studying mannequin for detecting radioactive isotopes. That is the final third half, and it’s time so as to add a created mannequin to the actual app!

On this story, I’ll take a look at two approaches:

• I’ll create a public Streamlit app that might be hosted at no cost on Streamlit Cloud (the app hyperlink is added to the top of the article).
• As a extra versatile and common resolution, I’ll create a Python HTMX-based app that may talk with actual {hardware} and make predictions in actual time.

In the identical approach as within the earlier half, I’ll use a Radiacode scintillation detector to get the info (disclaimer: the gadget used on this take a look at was offered by the producer; I don’t get any business revenue from their gross sales, and I didn’t get any editorial enter about all of the checks). Readers who don’t have a Radiacode {hardware} will be capable of take a look at the app and the mannequin utilizing recordsdata out there on Kaggle.

Let’s get began!

1. Isotopes Classification Mannequin

This mannequin was described within the previous part. It’s primarily based on XGBoost, and I skilled the mannequin utilizing totally different radioactive samples. I used samples that may be legally bought, like classic uranium glass or previous watches with radium dials made within the Fifties. As talked about earlier than, I additionally used a Radiacode scintillation detector, which permits me to get the gamma spectrum of the thing. Solely 10-20 years in the past, these kind of detectors had been out there solely in large labs; in the present day, they are often bought for the worth of a mid-range smartphone.

The mannequin incorporates three parts:

• The XGBoost-based mannequin itself.
• A listing of radioactive isotopes (like Lead-214 or Actinium-228), on which the mannequin was skilled. The Radiacode scintillation detector returns 1024 spectrum values, and 23 of them had been used for the mannequin.
• A label encoder to transform checklist indexes into human-readable names.

Let’s wrap all this right into a single Python class:

from xgboost import XGBClassifier
from sklearn.preprocessing import LabelEncoder


class IsotopesClassificationModel:
    """ Gamma Spectrum Classification Mannequin """

    def __init__(self):
        """ Load fashions """
        path = self._get_models_path()
        self._classifier = self._load_model(path + "/XGBClassifier.json")
        self._isotopes = self._load_isotopes(path + "/isotopes.json")
        self._labels_encoder = self._load_labels_encoder(path + "/LabelEncoder.npy")

    def predict(self, spectrum: Spectrum) -> str:
        """ Predict the isotope """
        options = SpectrumPreprocessing.convert_to_features(
            spectrum, self._isotopes
        )
        preds = self._classifier.predict([features])
        preds = self._labels_encoder.inverse_transform(preds)
        return preds[0]

    @staticmethod
    def _load_model(filename: str) -> XGBClassifier:
        """ Load mannequin from file """
        bst = XGBClassifier()
        bst.load_model(filename)
        return bst

    @staticmethod
    def _load_isotopes(filename: str) -> Record:
        with open(filename, "r") as f_in:
            return json.load(f_in)

    @staticmethod
    def _load_labels_encoder(filename: str) -> LabelEncoder:
        le = LabelEncoder()
        le.classes_ = np.load(filename)
        return le

    @staticmethod
    def _get_models_path() -> str:
        """ Get path to fashions. Mannequin recordsdata are saved in 
            'fashions/V1/' folder """
        parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
        return parent_dir + f"/fashions/{IsotopesClassificationModel.VERSION}"

A Spectrum class incorporates the spectrum information we get from a radiation detector:

@dataclass
class Spectrum:
    """ Radiation spectrum information """

    period: datetime.timedelta
    a0: float
    a1: float
    a2: float
    counts: checklist[int]

Right here, counts is a gamma spectrum, which is represented by a listing of 1024 channel values. Spectrum information could be exported utilizing the official Radiacode Android app or retrieved immediately from a tool utilizing a radiacode Python library.

To load the spectrum into the mannequin, I created a SpectrumPreprocessing class:

class SpectrumPreprocessing:
    """ Gamma Spectrum Preprocessing """

    @staticmethod
    def convert_to_features(spectrum: Spectrum, isotopes: Record) -> np.array:
        """ Convert the spectrum to the checklist of options for prediction """
        sp_norm = SpectrumPreprocessing._normalize(spectrum)
        energies = [energy for _, energy in isotopes]
        channels = [SpectrumPreprocessing.energy_to_channel(spectrum, energy) for energy in energies]
        return np.array([sp_norm.counts[ch] for ch in channels])

    @staticmethod 
    def load_from_xml_file(file_path: str) -> Spectrum:
        """ Load spectrum from a Radiacode Android app file """

Right here, I skip some code blocks that had been already printed within the previous part. Extracting options from the gamma spectrum was additionally defined there, and I extremely suggest studying that half first.

Now, let’s take a look at the mannequin! I took a Radiacode detector and picked up a gamma spectrum inside 10 minutes:

This Chinese language pendant was marketed as “ion-generated,” and it’s barely radioactive. Its gamma spectrum, collected within the official Radiacode Android app, seems to be like this:

After ready for ~10 minutes, I exported the spectrum into an XML file. Now, we are able to run the mannequin:

from spectrum import SpectrumPreprocessing
from ml_models import IsotopesClassificationModel

sp = SpectrumPreprocessing.load_from_file("spectrum.xml")
mannequin = IsotopesClassificationModel()
outcome = mannequin.predict(sp)
print(outcome)

#> Thorium

As we are able to see, the mannequin works nicely. We are able to evaluate the peaks with spectra of identified isotopes (for instance, here or here) and make sure that the spectrum belongs to thorium.

2. Streamlit

The mannequin works; nonetheless, we dwell within the XXI century, and nearly no one will run the console app to get the outcomes. As a substitute, we are able to make the app out there on-line, so all Radiacode customers will be capable of run it.

There are a lot of Python frameworks for making browser-based apps, and Streamlit might be the preferred within the information science group. And what’s necessary for us, a Streamlit Community Cloud platform permits everybody to publish their apps utterly at no cost. To do that, let’s make the app first.

2.1 Streamlit App

A Streamlit framework is comparatively straightforward to make use of, a minimum of if a standard-looking app is nice for us. Personally, I’m not a fan of this strategy. These frameworks conceal all low-level implementation particulars from customers. It’s simple to make a prototype, however the UI logic might be tightly coupled with a really area of interest framework and can’t be reused wherever else. Doing all the pieces non-standard, which isn’t supported by the framework, could be nearly inconceivable or onerous to implement with out digging into tons of abstractions and pages of code. Nonetheless, in our case, the prototype is all we want.

On the whole, a Streamlit code is straightforward, and we simply want to explain the logical hierarchy of our web page:

import streamlit as st
import logging
logger = logging.getLogger(__name__)


def is_xml_valid(xml_data: str) -> bool:
    """ Verify if the XML has legitimate dimension and information """
    return len(xml_data) < 65535 and xml_data.startswith("<?xml")


def get_spectrum(stringio: StringIO) -> Non-compulsory[Spectrum]:
    """ Load spectrum from the StringIO stream """
    xml_data = stringio.learn()
    if is_xml_valid(xml_data):
        return SpectrumPreprocessing.load_from_xml(xml_data)
    return None

def fundamental():
    """ Major app """
    st.set_page_config(page_title="Gamma Spectrum")
    st.title("Radiacode Spectrum Detection")
    st.textual content(
        "Export the spectrum to XML utilizing the Radiacode app, and "
        "add it to see the outcomes."
    )

    # File Add
    uploaded_file = st.file_uploader(
        "Select the XML file", kind="xml", key="uploader",
    )
    if uploaded_file will not be None:
        stringio = StringIO(uploaded_file.getvalue().decode("utf-8"))
        if sp := get_spectrum(stringio):
            # Prediction
            mannequin = IsotopesClassificationModel()
            outcome = mannequin.predict(sp)
            logger.data(f"Spectrum prediction: {outcome}")

            # Present outcome
            st.success(f"Prediction End result: {outcome}")
            # Draw
            fig = get_spectrum_barchart(sp)
            st.pyplot(fig)


if __name__ == "__main__":
    logger.setLevel(logging.INFO)
    fundamental()

As we are able to see, the total app requires a minimal quantity of Python code. Streamlit will render all HTML for us, with title, file add, and outcomes. As a bonus, I can even show a spectrum utilizing Matplotlib:

def get_spectrum_barchart(sp: Spectrum) -> plt.Determine:
    """ Get Matplotlib's barchart """
    counts = SpectrumPreprocessing.get_counts(sp)
    power = [
       SpectrumPreprocessing.channel_to_energy(sp, x) for x in range(len(counts))
    ]

    fig, ax = plt.subplots(figsize=(9, 6))
    ax.spines["top"].set_color("lightgray")
    ax.spines["right"].set_color("lightgray")
    # Bars
    ax.bar(power, counts, width=3.0, label="Counts")
    # X values
    ticks_x = [SpectrumPreprocessing.channel_to_energy(sp, ch) for ch in range(0, len(counts), len(counts) // 20)]
    labels_x = [f"{int(ch)}" for ch in ticks_x]
    ax.set_xticks(ticks_x, labels=labels_x, rotation=45)
    ax.set_xlim(power[0], power[-1])
    ax.set_ylim(0, None)
    ax.set_title("Gamma spectrum")
    ax.set_xlabel("Vitality, keV")
    ax.set_ylabel("Counts")
    return fig

Now we are able to run the app domestically:

streamlit run st-app.py

After that, our app is absolutely operational and could be examined in a browser:

As talked about earlier than, I’m not a fan of very high-level frameworks and like to have a greater understanding of how issues work “below the hood.” Nonetheless, contemplating that I spent solely about 100 traces of code to make a totally useful internet app, I can’t complain – for prototyping, it really works nicely.

2.2 Streamlit Neighborhood Cloud

When the app is examined domestically, it’s time to make it public! A Streamlit Cloud is a free service, and clearly, it has a number of limitations:

• The app runs in a Docker-like container. Your GitHub account have to be linked to Streamlit. When the container begins, it pulls your code from GitHub and runs it.
• On the time of penning this textual content, container assets are restricted to 2 cores and as much as 2,7 GB of RAM. It could be too constrained to run a 70B dimension LLM, however for a small XGBoost mannequin, it’s greater than sufficient.
• Streamlit doesn’t present any everlasting storage. After shutdown or restart, all logs and momentary recordsdata might be misplaced (you should use API secrets and techniques and connect with every other cloud storage out of your Python code if wanted).
• After a interval of inactivity (about half-hour), the container might be stopped, and all momentary recordsdata can even be misplaced. If somebody opens the app hyperlink, it’ll run once more.

As readers can guess, an inactive app prices Streamlit nearly nothing as a result of it shops solely a small configuration file. And it’s a good resolution for a free service – it permits us to publish the app with none prices and provides folks a hyperlink to run it.

To publish the app in Streamlit, we have to carry out three easy steps.

First, we have to commit our Python app to GitHub. A necessities.txt file can be obligatory. Streamlit container makes use of it to put in required Python dependencies. In my case, it seems to be like this:

xgboost==3.0.2
scikit-learn==1.6.1
numpy==1.26.4
streamlit==1.47.0
pillow==11.1.0
matplotlib==3.10.3
xmltodict==0.14.2

Server settings could be modified utilizing a .streamlit/config.toml file. In my case, I restricted the uploaded file dimension to 1 MB as a result of all spectra recordsdata are smaller:

[server]
# Max dimension, in megabytes, for recordsdata uploaded with the file_uploader.
# Default: 200
maxUploadSize = 1

Second, we have to log in to share.streamlit.io utilizing a GitHub account and provides permission to entry the supply code.

Lastly, we are able to create a brand new Streamlit undertaking. Within the undertaking settings, we are able to additionally choose the specified URL and surroundings:

If all the pieces was performed appropriately, we are able to see our app operating:

At this second, customers worldwide may entry our app! In my case, I chosen a gammaspectrumdetection identify, and the app is offered utilizing this URL.

3. FastAPI + HTMX App

As readers can see, Streamlit is a pleasant resolution for a easy prototype. Nonetheless, within the case of the radiation detector, I wish to see information coming from actual Radiacode {hardware}. This might be inconceivable to do in Streamlit; this library simply was not designed for that. As a substitute, I’ll use a number of production-grade frameworks:

• An HTMX framework permits us to make a totally useful internet interface.
• FastAPI will run the server.
• The ML mannequin will course of the info retrieved in real-time from a radiation detector utilizing a Radiacode library.

As talked about earlier than, these readers who don’t have a Radiacode {hardware} will be capable of replay the info utilizing uncooked log recordsdata, saved from an actual gadget. A hyperlink to the app and all recordsdata is offered on the finish of the article.

Let’s get into it!

3.1 HTML/HTMX

The app is linked to a Radiacode detector, and I made a decision to indicate a connection standing, radiation stage, and a spectrum graph on the web page. On the backside, a spectrum assortment time and an ML mannequin prediction might be displayed.

An index.html file for this structure seems to be like this:

<!DOCTYPE html>
<head>
    <meta identify="viewport" content material="width=device-width, initial-scale=1.0">
    <title>Gamma Spectrum & Monitoring</title>
    <script src="https://cdn.jsdelivr.web/npm/[email protected]/dist/htmx.min.js" crossorigin="nameless"></script>
    <script src="https://cdn.jsdelivr.web/npm/chart.js"></script>
    <hyperlink rel="stylesheet" href="{{ url_for('static', path='types.css') }}">
</head>

<physique>
    <script kind="textual content/javascript">
        perform createChart() {
            const ctx = doc.getElementById('gammaChart').getContext('2nd');
            window.chart = new Chart(ctx, {
                 ...
            });
        }

        perform updateChartData(labels, information) {
            window.chart.information.labels = labels;
            window.chart.information.datasets[0].information = information;
            window.chart.replace();
        }

        doc.addEventListener('DOMContentLoaded', perform() {
            // console.log('Web page loaded');
            createChart();
        }, false);
    </script>

    <div class="dashboard">
        <div id="reload-status" hx-get="/gadget/ballot" hx-target="#device-info" hx-swap="outerHTML" hx-trigger="load, each 1s"/>
    
        <div class="info-row-top">
            <div class="info-box-left">
                <span id="device-info">Machine: n/a</span><br/>
            </div>
            <div class="info-box-right">
                <span id="device-cps">CPS: n/a</span>
            </div>
        </div>

        <canvas id="gammaChart" class="gamma-chart"></canvas>
        <div id="gammaChartData"></div>

        <div class="info-row-bottom">
            <div class="info-box-left">
                <span>Assortment Time:</span><br/>
                <span id="collection-time">n/a</span>
            </div>
            <div class="info-box-right">
                <span>Prediction:</span><br/>
                <span id="prediction-result">n/a</span>
           </div>
        </div>
        <button class="reset-button" hx-post="/gadget/reset_spectrum" hx-swap="none">Reset Spectrum</button>
    </div>
</physique>
</html>

Right here, I used Chart.js to make a graph, and all UI controls are positioned within the div part. HTMX permits us to make an interactive internet app with out utilizing JavaScript. This HTMX string performs the precise replace:

hx-get="/gadget/ballot" hx-target="#device-info" hx-swap="outerHTML" hx-trigger="load, each 1s"

Right here, HTMX is configured to name the /gadget/ballot endpoint each second, and the outcome might be positioned into the management with a #device-info ID.

Readers are welcome to alter the HTML if wanted. I’m not a frontend developer, and possibly this UI could be performed extra successfully. Nonetheless, the web page works and does its job:

3.2 FastAPI

As talked about earlier than, I made a decision to make use of FastAPI for the backend half.

First, we have to create a FastAPI occasion and add the required endpoints:

from fastapi import FastAPI, Request, Response
from fastapi.responses import FileResponse
from contextlib import asynccontextmanager


@asynccontextmanager
async def lifespan(app: FastAPI):
    logging.data("FastAPI::Began")
    app.state.gadget = RadiaCodeDevice()
    app.state.gadget.start_in_background()
    yield
    app.state.gadget.cease()
    logging.data("FastAPI::Ended")

app = FastAPI(lifespan=lifespan)
app.mount("/static", StaticFiles(listing="static"), identify="static")

@app.get("/gadget/ballot")
async def device_poll():
    """ Get gadget information """
    # The code is positioned beneath

@app.publish("/gadget/reset_spectrum")
async def device_reset_spectrum():
    """ Reset gamma spectrum information """
    app.state.gadget.reset_spectrum()
    return "OK"

@app.get("/")
def read_index(request: Request):
    context = {"request": request}
    return templates.TemplateResponse("index.html", context)

@app.get("/favicon.ico", include_in_schema=False)
async def favicon():
    return FileResponse("static/favicon.ico")


if __name__ == '__main__':
    import uvicorn
    uvicorn.run("app:app", host='0.0.0.0', port=8000, reload=True)

As readers can see, I used a lifespan callback to run the Radiacode connection when the FastAPI server is began. The app.state is a world state object in FastAPI. Now we have solely a single gadget occasion, so using a world state is an effective place for that. FastAPI itself is predicated on asyncio, and the Radiacode connection thread works within the background. In asyncio, I can’t block the primary thread, so I used two queues (one for information and one for instructions) for information trade.

The way in which of updating the web page in HTMX is attention-grabbing. Let’s say I wish to replace the CPS (counts per second) worth. As a reminder, in HTML, I’ve these controls:

<span id="device-info">Machine: n/a</span><br/>

<div class="info-box-left">
    <span id="device-info">Machine: n/a</span><br/>
</div>
<div class="info-box-right">
    <span id="device-cps">CPS: n/a</span>
</div>

<div id="reload-status" hx-get="/gadget/ballot" hx-target="#device-info"
     hx-swap="outerHTML" hx-trigger="load, each 1s"
/>

In a response to /gadget/ballot, I must return a correct HTML for the corresponding IDs:

def get_connection_data(self) -> str:
     """ Get gadget information for the HTML ballot request """
     standing = "Related"
     return f'<span id="device-info">{standing}</span>'

def get_cps_data(self) -> str:
    """ Get CPS information for the HTML ballot request """
    return f'<span id="device-cps" hx-swap-oob="true">CPS: {self.radiation_cps:.2f}</span>'

def get_spectrum_data(self) -> str:
    """ Get spectrum information for the HTML ballot request """
    return f'<script>updateChartData([], [])</script>'

def get_spectrum_prediction(self) -> str:
    ...

@app.get("/gadget/ballot")
async def device_poll():
    """ Get gadget information """
    ballot = app.state.gadget.get_connection_data()
    cps = app.state.gadget.get_cps_data()
    spectrum = app.state.gadget.get_spectrum_data()
    prediction = app.state.gadget.get_spectrum_prediction()
    return Response(
        content material=ballot + cps + prediction + spectrum,
        media_type="utility/textual content"
    )

HTMX will mechanically set off the /gadget/ballot endpoint and exchange the controls on an online web page with new information. The hx-swap-oob key permits us to replace a number of controls in a single request. As we are able to see, HTMX permits us to seamlessly join a frontend web page in a browser with a Python backend.

A get_spectrum_data technique is barely totally different. Updating a Chart.js bar chart will not be supported immediately by HTMX. Nonetheless, I can return a JavaScript code block that might be executed on the web page. I exploit this characteristic to replace the graph by calling the updateChartData technique, which was positioned in index.html earlier than.

The complete supply code is longer, and right here I displayed solely the essential elements. A hyperlink to a full undertaking is offered on the finish of the article.

4. Testing

Lastly, let’s see each apps in motion! My mannequin can classify isotopes, and I wished to check it utilizing an object that was not “seen” by the mannequin earlier than. To do that, I visited the minerals store within the metropolis middle:

I do know that some minerals could be barely radioactive, so I selected the most effective one (or the worst one, relying in your choice) utilizing a radiation detector. It was a blue apatite. I examined it at dwelling, and my GMC radiation meter reveals that its radioactivity stage is 0,71 µSv/h, which was thought of by the gadget’s algorithm as “Excessive”:

Right here, “excessive” doesn’t imply “harmful” – this worth is simply excessive for the typical background radiation stage, which is often about 0,1 µSv/h. Nonetheless, on this case, the upper stage is barely about 1-2cm across the mineral. It isn’t harmful and can’t trigger any hurt (as a comparability, the radiation within the airplane, brought on by cosmic rays, is ~1.5 instances larger than that). And as described within the earlier half, a Geiger counter can present us the worth, however can’t inform us why this mineral is radioactive. So, I’ll use a Radiacode gamma detector to search out it out.

As a reminder, first I created a Streamlit app and printed it on the Streamlit Neighborhood Cloud platform. The app is available online, however whereas no one makes use of it, the app is in “sleep mode”:

I saved a spectrum utilizing a Radiacode Android app. When the Streamlit app began (the container spin-up takes 2-3 minutes), I uploaded the file:

As a second take a look at, let’s run the native app. It’s extra useful, it will probably talk with the Radiacode detector by way of USB, and may work as a UI to manage the {hardware}. One other essential distinction is that we are able to see information from the detector in actual time.

I pressed the “Reset Spectrum” button to restart the buildup, and after 2-3 minutes of amassing the info, the mineral was recognized as thorium:

This was an attention-grabbing discover, as a result of in accordance with Wikipedia, apatite could comprise uranium, however thorium was not talked about there. Nonetheless, in accordance with the USGS (U.S. Geological Survey) web page, thorium could typically be present in apatites. On this case, the Wikipedia article was simply not full sufficient.

Conclusion

On this article, I examined the machine studying mannequin for detecting radioactive isotopes. I examined two methods of utilizing the mannequin – a Streamlit app, printed in Streamlit Neighborhood Cloud, and a totally useful FastAPI app that will get information from the {hardware} and can be utilized as an online interface for an actual radiation detector.

As we are able to see, the mannequin works nicely, and I used to be capable of finding hint quantities of thorium in a blue apatite. This reality was not even talked about in Wikipedia, and it’s all the time good to be taught one thing new. Nonetheless, as was talked about within the earlier half, the mannequin’s capabilities are restricted. I’m not a nuclear establishment, and the mannequin was skilled solely on radioactive objects that may be legally bought, like a classic watch with radium dials. I don’t have take a look at sources like Cesium or Plutonium. Nonetheless, it was enjoyable to coach the mannequin and to make use of it within the app. Most likely, tasks like this don’t have any business worth, and there’s a 0.00% market demand for the isotope classification mannequin. Nonetheless, whereas doing this, I used to be ready to make use of instruments like FastAPI, HTMX, or CSS, which could be helpful in different tasks. Final however not least, I had enjoyable, which is crucial half 😉

All information recordsdata had been collected utilizing a Radiacode scintillation detector. It’s a pleasant and compact gadget with a worth of a mid-range smartphone, that, as we are able to see, permits us to carry out some enjoyable experiments (disclaimer: I don’t have any revenue or different business curiosity from its gross sales). For these readers who don’t have a Radiacode {hardware}, all collected information is freely available on Kaggle. The archive incorporates spectra of various objects used to coach the mannequin. Spectra in XML format can be utilized to check a Streamlit app. The “replay” folder incorporates the log, saved from a Radiacode gadget, and can be utilized to check the HTMX app.

The complete supply code for this text is offered on my Patreon page. This assist helps me to purchase gear and electronics for future checks like this. And readers are additionally welcome to attach by way of LinkedIn, the place I periodically publish smaller posts that aren’t sufficiently big for a full article.

Thanks for studying.