From Python to JavaScript: A Playbook for Data Analytics in n8n with Code Node Examples

my first n8n workflow, as a knowledge scientist, it felt like I used to be dishonest.

I may connect with APIs with out studying 30-page docs, set off workflows from Gmail or Sheets, and deploy one thing helpful in minutes.

Nevertheless, the numerous downside is that n8n shouldn’t be natively optimised to run a Python setting within the cloud situations utilized by our clients.

Like many knowledge scientists, my each day toolbox for knowledge analytics is constructed on NumPy and Pandas. 

To remain in my consolation zone, I usually outsourced calculations to exterior APIs as an alternative of utilizing n8n JavaScript code nodes.

As an illustration, that is what is finished with a Manufacturing Planning Optimisation software, which is orchestrated by way of a workflow that features an Agent node that calls a FastAPI microservice.

This method labored, however I had shoppers who requested to have full visibility of the information analytics duties on their n8n consumer interface.

I realised that I must be taught simply sufficient JavaScript to carry out knowledge processing with the native code nodes of n8n.

On this article, we are going to experiment with small JavaScript snippets inside n8n Code nodes to carry out on a regular basis knowledge analytics duties.

For this train, I’ll use a dataset of gross sales transactions and stroll it by way of to an ABC and Pareto evaluation, that are extensively utilized in Provide Chain Administration.

I’ll present side-by-side examples of Pandas vs. JavaScript in n8n Code nodes, permitting us to translate our acquainted Python knowledge evaluation steps immediately into automated n8n workflows.

The concept is to implement these options for small datasets or fast prototyping inside the capabilities of a cloud enterprise n8n occasion (i.e. with out group nodes).

I’ll finish the experiment with a fast comparative examine of the efficiency versus a FastAPI name.

You’ll be able to comply with me and replicate your complete workflow utilizing a Google Sheet and a workflow template shared within the article.

Let’s begin!

Constructing a Information Analytics Workflow utilizing JavaScript in n8n

Earlier than beginning to construct nodes, I’ll introduce the context of this evaluation.

ABC & Pareto Charts for Provide Chain Administration

For this tutorial, I suggest that you just construct a easy workflow that takes gross sales transactions from Google Sheets and transforms them right into a complete ABC and Pareto charts.

This can replicate the ABC and Pareto Evaluation module of the LogiGreen Apps developed by my startup, LogiGreen.

The objective is to generate a set of visuals for the stock groups of a grocery store chain to assist them perceive the distribution of gross sales throughout their shops.

We are going to give attention to producing two visuals.

The primary chart exhibits an ABC-XYZ evaluation of gross sales objects:

• X-axis (Proportion of Turnover %): the contribution of every merchandise to whole income.
• Y-axis (Coefficient of Variation): demand variability of every merchandise.
• Vertical pink traces break up objects into A, B, and C courses primarily based on turnover share.
• The horizontal blue line marks secure vs variable demand (CV=1)

Collectively, it highlights which objects are high-value & secure (A, low CV) versus these which are low-value or extremely variable, guiding prioritisation in stock administration.

The second visible is a Pareto evaluation of gross sales turnover:

• X-axis: share of SKUs (ranked by gross sales).
• Y-axis: cumulative share of annual turnover.
• The curve illustrates how a small fraction of things contributes to the vast majority of income.

In brief, this highlights (or not) the basic Pareto rule, which affirms that 80% of gross sales can come from 20% of the SKUs.

How did I generate these two visuals? I merely used Python.

On my YouTube channel, I shared a complete tutorial on the way to do it utilizing Pandas and Matplotlib.

The target of this tutorial is to organize gross sales transactions and generate these visuals in a Google Sheet utilizing solely n8n’s native JavaScript nodes.

Constructing a Information Analytics Workflow in n8n

I suggest to construct a workflow that’s manually triggered to facilitate debugging throughout improvement.

To comply with this tutorial, that you must

Now you can join your duplicated sheet utilizing the second node, which is able to extract the dataset from the worksheet: Enter Information.

This dataset consists of retail gross sales transactions on the each day granularity:

• ITEM: an merchandise that may be bought in a number of shops
• SKU: represents an `SKU` bought in a selected retailer 
• FAMILY: a gaggle of things
• CATEGORY: a product class can embody a number of households
• STORE: a code representing a gross sales location
• DAY of the transaction
• QTY: gross sales amount in models
• TO: gross sales amount in euros

The output is the desk’s content material in JSON format, able to be ingested by different nodes.

Python Code

import pandas as pd
df = pd.read_csv("gross sales.csv") 

We will now start processing the dataset to construct our two visualisations.

Step 1: Filter out transactions with out gross sales

Allow us to start with the straightforward motion of filtering out transactions with gross sales QTY equal to zero.

We don’t want JavaScript; a easy Filter node can do the job.

Python Code

df = df[df["QTY"] != 0]

Step 2: Put together knowledge for Pareto Evaluation

We first must mixture the gross sales per ITEM and rank merchandise by turnover.

Python Code

sku_agg = (df.groupby("ITEM", as_index=False)
             .agg(TO=("TO","sum"), QTY=("QTY","sum"))
             .sort_values("TO", ascending=False))

In our workflow, this step shall be finished within the JavaScript node TO, QTY GroupBY ITEM:

const agg = {};
for (const {json} of things) {
  const ITEM = json.ITEM;
  const TO = Quantity(json.TO);
  const QTY = Quantity(json.QTY);
  if (!agg[ITEM]) agg[ITEM] = { ITEM, TO: 0, QTY: 0 };
  agg[ITEM].TO += TO;
  agg[ITEM].QTY += QTY;
}
const rows = Object.values(agg).kind((a,b)=> b.TO - a.TO);
return rows.map(r => ({ json: r }));

This node returns a ranked desk of gross sales per ITEM in amount (QTY) and turnover (TO): 

1. We provoke agg as a dictionary keyed by ITEM
2. We loop over n8n rows in objects

• Changing TO and QTY to numbers
• Add the QTY and TO worth into the working totals of every ITEM

3. We lastly rework the dictionary into an array sorted by TO desc and return objects

We now have the information able to carry out a Pareto Evaluation on gross sales amount (QTY) or turnover (TO).

For that, we have to calculate cumulative gross sales and rank SKUs from the best to the bottom contributor.

Python Code

abc = sku_agg.copy()  # from Step 2, already sorted by TO desc
whole = abc["TO"].sum() or 1.0
abc["cum_turnover"] = abc["TO"].cumsum()
abc["cum_share"]    = abc["cum_turnover"] / whole             
abc["sku_rank"]     = vary(1, len(abc) + 1)
abc["cum_skus"]     = abc["sku_rank"] / len(abc)               
abc["cum_skus_pct"] = abc["cum_skus"] * 100                    

This step shall be finished within the code node Pareto Evaluation:

const rows = objects
  .map(i => ())
  .kind((a, b) => b.TO - a.TO);

const n = rows.size; // variety of ITEM
const totalTO = rows.cut back((s, r) => s + r.TO, 0) || 1;

We gather the dataset objects from the earlier node

1. For every row, we clear up the fields TO and QTY (in case we have now lacking values)
2. We kind all SKUs by turnover in descending order.
3. We retailer in variables the variety of objects and the entire turnover

let cumTO = 0;
rows.forEach((r, idx) => {
  cumTO += r.TO;
  r.cum_turnover = cumTO;                     
  r.cum_share = +(cumTO / totalTO).toFixed(6); 
  r.sku_rank = idx + 1;
  r.cum_skus = +((idx + 1) / n).toFixed(6);   
  r.cum_skus_pct = +(r.cum_skus * 100).toFixed(2);
});

return rows.map(r => ({ json: r }));

Then we loop over all objects in sorted order.

1. Use the variable cumTO to compute the cumulative contribution
2. Add a number of Pareto metrics to every row:

• cum_turnover: cumulative turnover as much as this merchandise
• cum_share: cumulative share of turnover
• sku_rank: rating place of the merchandise
• cum_skus: cumulative variety of SKUs as a fraction of whole SKUs
• cum_skus_pct: identical as cum_skus, however in %.

We’re then finished with the information preparation of the pareto chart.

This dataset shall be saved within the worksheet Pareto by the node Replace Pareto Sheet.

And with a little bit of magic, we are able to generate this graph within the first worksheet:

We will now proceed with the ABC XYZ chart.

Step 3: Calculate the demand variability and gross sales contribution

We may reuse the output of the pareto chart for the gross sales contribution, however we are going to contemplate every chart as unbiased.

I’ll break up the code for the node Demand Variability’ and ‘Gross sales x Gross sales % into a number of segments for readability.

Block 1: outline features for imply and normal deviation

operate imply(a)
operate stdev_samp(a){
  if (a.size <= 1) return 0;
  const m = imply(a);
  const v = a.cut back((s,x)=> s + (x - m) ** 2, 0) / (a.size - 1);
  return Math.sqrt(v);
}

These two features shall be used for the coefficient of variation (Cov)

• imply(a): computes the common of an array.
• stdev_samp(a): computes the pattern normal deviation

They take as inputs the each day gross sales distributions of every ITEM that we construct on this second block.

Block 2: Create the each day gross sales distribution of every ITEM

const sequence = {};  // ITEM -> { day -> qty_sum }
let totalQty = 0;

for (const { json } of things) {
  const merchandise = String(json.ITEM);
  const day  = String(json.DAY);
  const qty  = Quantity(json.QTY || 0);

  if (!sequence[item]) sequence[item] = {};
  sequence[item][day] = (sequence[item][day] || 0) + qty;
  totalQty += qty;
}

Python Code

import pandas as pd
import numpy as np
df['QTY'] = pd.to_numeric(df['QTY'], errors='coerce').fillna(0)
daily_series = df.groupby(['ITEM', 'DAY'])['QTY'].sum().reset_index()

Now we are able to compute the metrics utilized to the each day gross sales distributions.

const out = [];
for (const [item, dayMap] of Object.entries(sequence)) {
  const each day = Object.values(dayMap); // each day gross sales portions
  const qty_total = each day.cut back((s,x)=>s+x, 0);
  const m = imply(each day);               // common each day gross sales
  const sd = stdev_samp(each day);        // variability of gross sales
  const cv = m ? sd / m : null;        // coefficient of variation
  const share_qty_pct = totalQty ? (qty_total / totalQty) * 100 : 0;

  out.push({
    ITEM: merchandise,
    qty_total,
    share_qty_pct: Quantity(share_qty_pct.toFixed(2)),
    mean_qty: Quantity(m.toFixed(3)),
    std_qty: Quantity(sd.toFixed(3)),
    cv_qty: cv == null ? null : Quantity(cv.toFixed(3)),
  });
}

For every ITEM, we calculate

• qty_total: whole gross sales
• mean_qty: common each day gross sales.
• std_qty: normal deviation of each day gross sales.
• cv_qty: coefficient of variation (variability measure for XYZ classification)
• share_qty_pct: % contribution to whole gross sales (used for ABC classification)

Right here is the Python model in case you have been misplaced:

abstract = daily_series.groupby('ITEM').agg(
    qty_total=('QTY', 'sum'),
    mean_qty=('QTY', 'imply'),
    std_qty=('QTY', 'std')
).reset_index()

abstract['std_qty'] = abstract['std_qty'].fillna(0)

total_qty = abstract['qty_total'].sum()
abstract['cv_qty'] = abstract['std_qty'] / abstract['mean_qty'].exchange(0, np.nan)
abstract['share_qty_pct'] = 100 * abstract['qty_total'] / total_qty

We’re practically finished.

We simply must kind by descending contribution to organize for the ABC class mapping:

out.kind((a,b) => b.share_qty_pct - a.share_qty_pct);
return out.map(r => ({ json: r }));

We now have for every ITEM, the important thing metrics wanted to create the scatter plot.

Solely the ABC courses are lacking at this step.

Step 4: Add ABC courses

We take the output of the earlier node as enter.

let rows = objects.map(i => i.json);
rows.kind((a, b) => b.share_qty_pct - a.share_qty_pct);

Simply in case, we kind ITEMS by descending by gross sales share (%) → most necessary SKUs first.

(This step may be omitted as it’s usually already accomplished on the finish of the earlier code node.)

Then we are able to apply the category primarily based on hardcoded circumstances:

• A: SKUs that collectively signify the primary 5% of gross sales
• B: SKUs that collectively signify the following 15% of gross sales
• C: Every part after 20%.

let cum = 0;
for (let r of rows) {
  cum += r.share_qty_pct;

  // 3) Assign class primarily based on cumulative %
  if (cum <= 5) {
    r.ABC = 'A';   // high 5%
  } else if (cum <= 20) {
    r.ABC = 'B';   // subsequent 15%
  } else {
    r.ABC = 'C';   // relaxation
  }

  r.cum_share = Quantity(cum.toFixed(2));
}

return rows.map(r => ({ json: r }));

This may be finished that approach utilizing Python Code.

df = df.sort_values('share_qty_pct', ascending=False).reset_index(drop=True)
df['cum_share'] = df['share_qty_pct'].cumsum()
def classify(cum):
    if cum <= 5:
        return 'A'
    elif cum <= 20:
        return 'B'
    else:
        return 'C'
df['ABC'] = df['cum_share'].apply(classify)

The outcomes can now be used to generate this chart, which may be discovered within the first sheet of the Google Sheet:

I struggled (in all probability as a consequence of my restricted information of Google Sheets) to discover a “handbook” answer to create this scatter plot with the proper color mapping.

Subsequently, I used a Google Apps Script obtainable within the Google Sheet to create it.

As a bonus, I added extra nodes to the n8n template that carry out the identical sort of GroupBy to calculate gross sales by retailer or a pair of ITEM-store.

They can be utilized to create visuals like this one:

To conclude this tutorial, we are able to confidently declare that the job is finished.

For a reside demo of the workflow, you’ll be able to take a look at this brief tutorial

Our clients, who run this workflow on their n8n cloud occasion, can now achieve visibility into every step of the information processing.

However at which value? Are we loosing in efficiency?

That is what we are going to uncover within the subsequent part.

Comparative Examine of Efficiency: n8n JavaScript Node vs. Python in FastAPI

To reply this query, I ready a simple experiment.

The identical dataset and transformations have been processed utilizing two completely different approaches inside n8n:

1. All in JavaScript nodes with features immediately inside n8n.
2. Outsourcing to FastAPI microservices by changing the JavaScript logic with HTTP requests to Python endpoints.

These two endpoints are linked to features that may load the information immediately from the VPS occasion the place I hosted the microservice.

@router.publish("/launch_pareto")
async def launch_speedtest(request: Request):
    strive:
        session_id = request.headers.get('session_id', 'session')

        folder_in = f'knowledge/session/speed_test/enter'
        if not path.exists(folder_in):
                makedirs(folder_in)

        file_path = folder_in + '/gross sales.csv'
        logger.data(f"[SpeedTest]: Loading knowledge from session file: {file_path}")
        df = pd.read_csv(file_path, sep=";")
        logger.data(f"[SpeedTest]: Information loaded efficiently: {df.head()}")

        speed_tester = SpeedAnalysis(df)
        output = await speed_tester.process_pareto()
        
        consequence = output.to_dict(orient="data")
        consequence = speed_tester.convert_numpy(consequence)
        
        logger.data(f"[SpeedTest]: /launch_pareto accomplished efficiently for {session_id}")
        return consequence
    besides Exception as e:
        logger.error(f"[SpeedTest]: Error /launch_pareto: {str(e)}n{traceback.format_exc()}")
        elevate HTTPException(status_code=500, element=f"Did not course of Pace Check Evaluation: {str(e)}")
    
@router.publish("/launch_abc_xyz")
async def launch_abc_xyz(request: Request):
    strive:
        session_id = request.headers.get('session_id', 'session')

        folder_in = f'knowledge/session/speed_test/enter'
        if not path.exists(folder_in):
                makedirs(folder_in)

        file_path = folder_in + '/gross sales.csv'
        logger.data(f"[SpeedTest]: Loading knowledge from session file: {file_path}")
        df = pd.read_csv(file_path, sep=";")
        logger.data(f"[SpeedTest]: Information loaded efficiently: {df.head()}")

        speed_tester = SpeedAnalysis(df)
        output = await speed_tester.process_abcxyz()
        
        consequence = output.to_dict(orient="data")
        consequence = speed_tester.convert_numpy(consequence)
        
        logger.data(f"[SpeedTest]: /launch_abc_xyz accomplished efficiently for {session_id}")
        return consequence
    besides Exception as e:
        logger.error(f"[SpeedTest]: Error /launch_abc_xyz: {str(e)}n{traceback.format_exc()}")
        elevate HTTPException(status_code=500, element=f"Did not course of Pace Check Evaluation: {str(e)}")

I need to focus this take a look at solely on the information processing efficiency.

The SpeedAnalysis consists of all the information processing steps listed within the earlier part

• Grouping gross sales by ITEM
• Sorting ITEM by descending order and calculate cumulative gross sales
• Calculating normal deviations and technique of gross sales distribution by ITEM

class SpeedAnalysis:
    def __init__(self, df: pd.DataFrame):
        config = load_config()
        
        self.df = df
        
    def processing(self):
        strive:
            gross sales = self.df.copy()
            gross sales = gross sales[sales['QTY']>0].copy()
            self.gross sales = gross sales

        besides Exception as e:
            logger.error(f'[SpeedTest] Error for processing : {e}n{traceback.format_exc()}')
            
    def prepare_pareto(self):
        strive:
            sku_agg = self.gross sales.copy()
            sku_agg = (sku_agg.groupby("ITEM", as_index=False)
             .agg(TO=("TO","sum"), QTY=("QTY","sum"))
             .sort_values("TO", ascending=False))
            
            pareto = sku_agg.copy()  
            whole = pareto["TO"].sum() or 1.0
            pareto["cum_turnover"] = pareto["TO"].cumsum()
            pareto["cum_share"]    = pareto["cum_turnover"] / whole              
            pareto["sku_rank"]     = vary(1, len(pareto) + 1)
            pareto["cum_skus"]     = pareto["sku_rank"] / len(pareto)               
            pareto["cum_skus_pct"] = pareto["cum_skus"] * 100
            return pareto                    
        besides Exception as e:
            logger.error(f'[SpeedTest]Error for prepare_pareto: {e}n{traceback.format_exc()}')
            
    def abc_xyz(self):
            each day = self.gross sales.groupby(["ITEM", "DAY"], as_index=False)["QTY"].sum()
            stats = (
                each day.groupby("ITEM")["QTY"]
                .agg(
                    qty_total="sum",
                    mean_qty="imply",
                    std_qty="std"
                )
                .reset_index()
            )
            stats["cv_qty"] = stats["std_qty"] / stats["mean_qty"].exchange(0, np.nan)
            total_qty = stats["qty_total"].sum()
            stats["share_qty_pct"] = (stats["qty_total"] / total_qty * 100).spherical(2)
            stats = stats.sort_values("share_qty_pct", ascending=False).reset_index(drop=True)
            stats["cum_share"] = stats["share_qty_pct"].cumsum().spherical(2)
            def classify(cum):
                if cum <= 5:
                    return "A"
                elif cum <= 20:
                    return "B"
                else:
                    return "C"
            stats["ABC"] = stats["cum_share"].apply(classify)
            return stats
        
    def convert_numpy(self, obj):
        if isinstance(obj, dict):
            return {okay: self.convert_numpy(v) for okay, v in obj.objects()}
        elif isinstance(obj, listing):
            return [self.convert_numpy(v) for v in obj]
        elif isinstance(obj, (np.integer, int)):
            return int(obj)
        elif isinstance(obj, (np.floating, float)):
            return float(obj)
        else:
            return obj

    async def process_pareto(self):
        """Important processing operate that calls all different strategies so as."""
        self.processing()
        outputs = self.prepare_pareto()
        return outputs
    
    async def process_abcxyz(self):
        """Important processing operate that calls all different strategies so as."""
        self.processing()
        outputs = self.abc_xyz().fillna(0)
        logger.data(f"[SpeedTest]: ABC-XYZ evaluation accomplished {outputs}.")
        return outputs

Now that we have now these endpoints prepared, we are able to start testing.

The outcomes are proven above:

• JavaScript-only workflow: The entire course of was accomplished in a bit greater than 11.7 seconds.
  More often than not was spent updating sheets and performing iterative calculations inside n8n nodes.
• FastAPI-backed workflow: The equal “outsourced” course of was accomplished in ~11.0 seconds.
  Heavy computations have been offloaded to Python microservices, which dealt with them quicker than native JavaScript nodes.

In different phrases, outsourcing advanced computations to Python truly improves the efficiency.

The reason being that FastAPI endpoints execute optimised Python features immediately, whereas JavaScript nodes inside n8n should iterate (with loops).

For big datasets, I’d think about a delta that’s in all probability not negligible.

This demonstrates that you are able to do easy knowledge processing inside n8n utilizing small JavaScript snippets.

Nevertheless, our Provide Chain Analytics merchandise can require extra superior processing involving optimisation and superior statistical libraries.

For that, clients can settle for coping with a “black field” method, as seen within the Production Planning workflow offered in this Towards Data Science article.

However for gentle processing duties, we are able to combine them into the workflow to supply visibility to non-code customers.

For an additional venture, I exploit n8n to attach Provide Chain IT techniques for the switch of Buy Orders utilizing Digital Information Interchange (EDI).

This workflow, deployed for a small logistics firm, solely parses EDI messages utilizing JavaScript nodes.

As you’ll be able to uncover on this tutorial, we have now carried out 100% of the Digital Information Interchange message parsing utilizing JavaScript nodes.

This helped us to enhance the robustness of the answer and cut back our workload by handing over the upkeep to the client.

What’s the greatest method?

For me, n8n needs to be used as an orchestration and integration software linked to our core analytics merchandise.

These analytics merchandise require particular enter codecs that won’t align with our clients’ knowledge.

Subsequently, I’d advise utilizing JavaScript code nodes to carry out this preprocessing.

For instance, the workflow above connects a Google Sheet (containing enter knowledge) to a FastAPI microservice that runs an algorithm for Distribution Planning Optimisation.

The concept is to plug our optimisation algorithm right into a Google Sheet utilized by Distribution Planners to organise retailer deliveries.

The JavaScript code node is used to remodel the information collected from the Google Sheet into the enter format required by our algorithm.

By doing the job contained in the workflow, it stays underneath the management of the client who runs the workflow in their very own occasion.

And we are able to maintain the optimisation half in a microservice hosted on our occasion.

To higher perceive the setup, be happy to take a look at this brief presentation

I hope this tutorial and the examples above have given you adequate perception to grasp what may be finished with n8n when it comes to knowledge analytics.

Be at liberty to share your feedback in regards to the method and your ideas on what could possibly be improved to boost the workflow’s efficiency with me.

About Me

Let’s join on Linkedin and Twitter. I’m a Provide Chain Engineer who makes use of knowledge analytics to enhance logistics operations and cut back prices.

For consulting or recommendation on analytics and sustainable provide chain transformation, be happy to contact me by way of Logigreen Consulting.

Discover your full information for Provide Chain Analytics: Analytics Cheat Sheet.

If you’re considering Information Analytics and Provide Chain, have a look at my web site.

Samir Saci | Data Science & Productivity