If me for some time, you in all probability know I began my profession as a QA engineer earlier than transitioning into the world of knowledge analytics. I didn’t go to highschool for it, didn’t have a mentor, and didn’t land in a proper coaching program. Every thing I do know right this moment—from SQL to modeling to storytelling with knowledge—is self-taught. And consider me, it’s been a journey of trial, error, studying, and re-learning.
The Dilemma That Modified My Profession
A number of years in the past, I began fascinated with switching organizations. Like many individuals in fast-evolving tech roles, I confronted a surprisingly tough query:
What function am I really doing? Which roles ought to I apply for?
On paper, I used to be a Information Analyst. However in actuality, my function straddled a number of capabilities: writing SQL pipelines, constructing dashboards, defining KPIs, and digging into product analytics. I wasn’t positive whether or not I needs to be making use of for Analyst roles, BI roles, or one thing fully completely different.
To make issues worse, again then, job titles had been imprecise, and job descriptions had been bloated with buzzwords. You’d discover a posting titled “Information Analyst” that listed necessities like:
- Construct ML pipelines
- Write complicated ETL scripts
- Keep knowledge lakes
- Create dashboards
- Current executive-level insights
- And oh, by the way in which, be nice at stakeholder administration
It was overwhelming and complicated. And I do know I’m not alone on this.
Quick ahead to right this moment: fortunately, issues are evolving. There’s nonetheless overlap between roles, however organizations have began to outline them extra clearly. On this article, I need to break down the actual variations between knowledge roles, by way of the lens of a real-world instance.
A Actual-World Situation: Meet Quikee
Let’s think about a fictional quick-commerce startup known as Quikee, launching throughout a number of Indian cities. Their worth proposition? Ship groceries and necessities inside 10 minutes.
Prospects place orders by way of the app or web site. Behind the scenes, there are micro-warehouses (additionally known as “darkish shops”) throughout cities, and a fleet of supply companions who make these lightning-fast deliveries.
Now, let’s stroll by way of the information wants of this firm—from the second an order is positioned, to the dashboards executives use of their Monday morning conferences.
Step 1: Capturing and Storing Uncooked Information
The second a buyer locations an order, transactional knowledge is generated:
- Timestamps
- Order ID
- Objects ordered
- Value
- Low cost codes
- Buyer location
- Fee methodology
- Assigned supply associate
Let’s assume Quikee makes use of Amazon Kinesis to stream this knowledge in actual time to an S3 knowledge lake. That stream is high-volume, time-sensitive, and essential for enterprise monitoring.
However right here’s the catch: uncooked knowledge is messy. You possibly can’t use it instantly for decision-making.
So what occurs subsequent?
Step 2: Constructing Information Pipelines
Enter the Information Engineers.
They’re answerable for:
- Ingesting real-time knowledge
- Validating schema consistency
- Dealing with failures and retries
- Writing pipelines to maneuver knowledge from S3 into a knowledge warehouse (say, Snowflake or Redshift)
That is the place ETL (Extract, Remodel, Load) or ELT pipelines come into play. Information engineers clear, format, and construction the information to make it queryable.
For instance, an order desk may get cut up into:
- Orders → One row per order
- Order_Items → One row per merchandise in an order
- Funds → One row per fee try
At this stage, uncooked logs are become structured tables that analysts can work with.
Step 3: Dimensional Modeling & OLAP
As management begins asking strategic questions like:
- “Which metropolis brings in essentially the most income?”
- “Which retailer is underperforming?”
- “What’s our common supply time by zone?”
…it turns into clear that querying transactional knowledge instantly gained’t scale.
That’s the place dimensional modeling is available in.
As an alternative of flat, uncooked tables, knowledge is structured into Truth and Dimension Tables.
🔸 Truth Tables
- Giant, quantitative knowledge tables which comprise overseas keys together with measures and metrics (Properly, more often than not. There are factless reality tables as effectively which shouldn’t have any measures).
- Examples:
fact_orders,fact_payments,fact_deliveries - Comprise metrics like income, order rely, supply time
🔹 Dimension Tables
- Smaller, descriptive tables that assist perceive the information in a reality desk
- Examples:
dim_store,dim_product,dim_customer,dim_delivery_agent - Assist filter, group, and be part of info for deeper insights
This construction allows OLAP—quick, analytical querying throughout a number of dimensions. For instance, now you can run queries like:
“Present me common supply time by retailer and hour of day, during the last 7 days.”
This step is completed by Information Engineers at many of the organisations however I did construct few Dim and Truth tables once I was working as a Business Intelligence Engineer at Amazon.
Step 4: Defining KPIs and Metrics
That is the place Analytics Engineers (or BI Engineers) shine.
They sit between the technical knowledge layer and enterprise customers. Their obligations typically embody:
- Defining KPIs (e.g., churn charge, repeat buy %, time-to-fulfillment)
- Writing logic for complicated metrics (e.g., cohort retention, energetic customers)
- Creating semantic fashions or metrics layers in instruments like dbt or Looker
- Guaranteeing constant definitions throughout the corporate
For instance, at Amazon, our staff didn’t question uncooked knowledge to calculate income each time. As an alternative, we created pre-aggregated reality tables at each day, weekly, and month-to-month grains. That manner, dashboards loaded quicker, and metrics stayed constant throughout groups.
Analytics Engineers act as translators between engineering and the enterprise—defining what we measure and how we measure it.
Step 5: Evaluation, Reporting & Storytelling
Now comes the function of the Data Analyst.
Armed with clear, modeled knowledge, they deal with answering actual enterprise questions like:
- “Why did retention drop in Bangalore final month?”
- “Which coupon codes drive essentially the most new customers?”
- “What are the highest merchandise reordered within the first 30 days?”
They construct dashboards in instruments like Tableau, Energy BI, or Looker. They run ad-hoc SQL queries. They dive into A/B take a look at outcomes, person conduct traits, and marketing campaign effectiveness.
However above all, they inform tales with knowledge—making complicated numbers comprehensible and actionable for stakeholders.
Who’s Who?
TL;DR: The place Do You Match?
Right here’s how I give it some thought:
- Love constructing sturdy pipelines and fixing scalability issues? → You’re a Information Engineer
- Love defining enterprise metrics and organizing complicated datasets? → You’re an Analytics Engineer
- Love uncovering insights and storytelling with knowledge? → You’re a Information Analyst
In fact, real-world roles typically mix these. Particularly at smaller firms, it’s possible you’ll put on a number of hats. And that’s okay.
The secret’s not the title—however the place you add essentially the most worth and what energizes you.
Ultimate Ideas
It took me a very long time to grasp what I really do—not simply what my job title says. And in the event you’ve ever felt that confusion, you’re not alone.
In the present day, I can clearly say I function on the intersection of knowledge modeling, enterprise logic, and storytelling—a candy spot between analytics and engineering. And I’ve discovered that the flexibility to attach the dots is extra essential than becoming into an ideal field.
For those who’ve walked an analogous path—or put on a number of hats in your function—I’d love to listen to your story.
Drop a remark 👇 or share this with somebody figuring it out too.
