How AI Is Rewriting the Day-to-Day of Data Scientists

articles, I’ve explored and in contrast many AI instruments, for instance, Google’s Data Science Agent, ChatGPT vs. Claude vs. Gemini for Data Science, DeepSeek V3, and so on. Nonetheless, that is solely a small subset of all of the AI instruments out there for Data Science. Simply to call just a few that I used at work:

• OpenAI API: I take advantage of it to categorize and summarize buyer suggestions and floor product ache factors (see my tutorial article).
• ChatGPT and Gemini: They assist me draft Slack messages and emails, write evaluation experiences, and even efficiency critiques.
• Glean AI: I used Glean AI to seek out solutions throughout inner documentation and communications rapidly.
• Cursor and Copilot: I get pleasure from simply urgent tab-tab to auto-complete code and feedback.
• Hex Magic: I take advantage of Hex for collaborative information notebooks at work. In addition they supply a function referred to as Hex Magic to write down code and repair bugs utilizing conversational AI.
• Snowflake Cortex: Cortex AI permits customers to name Llm endpoints, construct RAG and text-to-SQL providers utilizing information in Snowflake.

I’m positive you may add much more to this listing, and new AI instruments are being launched each day. It’s nearly not possible to get an entire listing at this level. Due to this fact, on this article, I wish to take one step again and give attention to a much bigger query: what do we actually want as information professionals, and the way AI will help. 

Within the part under, I’ll give attention to two fundamental instructions — eliminating low-value duties and accelerating high-value work. 

1. Eliminating Low-Worth Duties

I grew to become a knowledge scientist as a result of I actually get pleasure from uncovering enterprise insights from advanced information and driving enterprise choices. Nonetheless, having labored within the trade for over seven years now, I’ve to confess that not all of the work is as thrilling as I had hoped. Earlier than conducting superior analyses or constructing machine studying fashions, there are lots of low-value work streams which are unavoidable day by day — and in lots of circumstances, it’s as a result of we don’t have the fitting tooling to empower our stakeholders for self-serve analytics. Let’s take a look at the place we’re at the moment and the best state:

Present state: We work as information interpreters and gatekeepers (generally “SQL monkeys”)

• Easy information pull requests come to me and my crew on Slack each week asking, “What was the GMV final month?” “Are you able to pull the listing of consumers who meet these standards?” “Are you able to assist me fill on this quantity on the deck that I must current tomorrow?” 
• BI instruments don’t assist self-service use circumstances nicely. We adopted BI instruments like Looker and Tableau so stakeholders can discover the info and monitor the metrics simply. However the actuality is that there’s all the time a trade-off between simplicity and self-servability. Typically we make the dashboards simple to grasp with just a few metrics, however they’ll solely fulfill just a few use circumstances. In the meantime, if we make the instrument very customizable with the potential to discover the metrics and underlying information freely, stakeholders might discover the instrument complicated and lack the boldness to make use of it, and within the worst case, the info is pulled and interpreted within the flawed manner.  
• Documentation is sparse or outdated. This can be a frequent scenario, however might be brought on by totally different causes — perhaps we transfer quick and give attention to delivering outcomes, or there isn’t a nice information documentation and governance insurance policies in place. In consequence, tribal information turns into the bottleneck for folks exterior of the info crew to make use of the info.

Superb state: Empower stakeholders to self-serve so we will reduce low-value work

• Stakeholders can do easy information pulls and reply fundamental information questions simply and confidently.
• Knowledge groups spend much less time on repetitive reporting or one-off fundamental queries.
• Dashboards are discoverable, interpretable, and actionable with out hand-holding.

So, to get nearer to the best state, what function can AI play right here? From what I’ve noticed, these are the frequent instructions AI instruments are going to shut the hole:

1. Question information with pure language (Textual content-to-SQL): One technique to decrease the technical barrier is to allow stakeholders to question the info with pure language. There are many Textual content-to-SQL efforts within the trade:
   • For instance, Snowflake is one firm that has made plenty of progress in Text2SQL models and began integrating the potential into its product. 
   • Many corporations (together with mine) additionally explored in-house Text2SQL options. For instance, Uber shared their journey with Uber’s QueryGPT to make information querying extra accessible for his or her Operations crew. This text defined intimately how Uber designed a multi-agent structure for question technology. In the meantime, it additionally surfaced main challenges on this space, together with precisely deciphering consumer intent, dealing with giant desk schemas, and avoiding hallucinations and so on. 
   • Actually, to make Textual content-to-SQL work, there’s a very excessive bar as it’s a must to make the question correct — even when the instrument fails simply as soon as, it might spoil the belief and finally stakeholders will come again to you to validate the queries (then it is advisable learn+rewrite the queries, which just about double the work 🙁). Up to now, I haven’t discovered a Textual content-to-SQL mannequin or instrument that works completely. I solely see it achievable if you find yourself querying from a really small subset of well-documented core datasets for particular and standardized use circumstances, however it is extremely exhausting to scale to all of the out there information and totally different enterprise situations. 
   • However in fact, given the massive quantity of funding on this space and speedy growth in AI, I’m positive we are going to get nearer and nearer to correct and scalable Textual content-to-SQL options. 
2. Chat-based BI assistant: One other frequent space to enhance stakeholders’ expertise with BI instruments is the chat-based BI assistant. This really takes one step additional than Textual content-to-SQL — as a substitute of producing a SQL question primarily based on a consumer immediate, it responds within the format of a visualization plus textual content abstract. 
   • Gemini in Looker is an instance right here. Looker is owned by Google, so it is extremely pure for them to combine with Gemini. One other benefit for Looker to construct their AI function is that information fields are already documented within the LookML semantic layer, with frequent joins outlined and widespread metrics inbuilt dashboards. Due to this fact, it has plenty of nice information to study from. Gemini permits customers to regulate Looker dashboards, ask questions in regards to the information, and even construct customized information brokers for Conversational Analytics. Although primarily based on my restricted experimentation with the instrument, it instances out typically and fails to reply easy questions generally. Let me know in case you have a unique expertise and have made it work…
   • Tableau additionally launched the same function, Tableau AI. I haven’t used it myself, however primarily based on the demo, it helps the info crew to arrange information and make dashboards rapidly utilizing pure language, and summarise information insights into “Tableau Pulse” for stakeholders to simply spot metric adjustments and irregular tendencies.   
3. Knowledge Catalog Instruments: AI may assist with the problem of sparse or outdated information documentation. 
   • Throughout one inner hackathon, I keep in mind one undertaking from our information engineers was to make use of LLM to extend desk documentation protection. AI is ready to learn the codebase and describe the columns accordingly with excessive accuracy normally, so it could assist enhance documentation rapidly with restricted human validation and changes. 
   • Equally, when my crew creates new tables, we now have began to ask Cursor to write down the desk documentation YAML recordsdata to save lots of us time with high-quality output. 
   • There are additionally plenty of information catalogs and governance instruments which were built-in with AI. After I google “ai information catalog”, I see the logos of information catalog instruments like Atlan, Alation, Collibra, Informatica, and so on (disclaimer: I’ve used none of them..). That is clearly an trade development. 

2. Accelerating high-value work

Now that we’ve talked about how AI might assist with eliminating low-value duties, let’s focus on the way it can speed up high-value information tasks. Right here, high-value work refers to information tasks that mix technical excellence with enterprise context, and drive significant impression by way of cross-functional collaboration. For instance, a deep dive evaluation that understands product utilization patterns and results in product adjustments, or a churn prediction mannequin to determine churn-risk clients and leads to churn-prevention initiatives. Let’s evaluate the present state and the best future:

Present state: Productivity bottlenecks exist in on a regular basis workflows 

• EDA is time-consuming. This step is essential to get an preliminary understanding of the info, however it might take a very long time to conduct all of the univariate and multivariate analyses.
• Time misplaced to coding and debugging. Let’s be trustworthy — nobody can keep in mind all of the numpy and pandas syntax and sklearn mannequin parameters. We continually must search for documentation whereas coding.
• Wealthy unstructured information is just not totally utilized. Enterprise generates plenty of textual content information each day from surveys, assist tickets, and critiques. However how one can extract insights scalably stays a problem.

Superb state: Knowledge scientists give attention to deep considering, not syntax 

• Writing code feels quicker with out the interruption to search for syntax.
• Analysts spend extra time deciphering outcomes, much less time wrangling information.
• Unstructured information is not a blocker and may be rapidly analyzed.

Seeing the best state, I’m positive you have already got some AI instrument candidates in thoughts. Let’s see how AI can affect or is already making a distinction: 

1. AI coding and debugging assistants. I believe that is by far probably the most adopted kind of AI instrument for anybody who codes. And we’re already seeing it iterating.
   • When LLM chatbots like ChatGPT and Claude got here out, engineers realized they might simply throw their syntax questions or error messages to the chatbot with high-accuracy solutions. That is nonetheless an interruption to the coding workflow, however a lot better than clicking by way of a dozen StackOverflow tabs — this already seems like final century. 
   • Later, we see increasingly built-in AI coding instruments popping up — GitHub Copilot and Cursor combine along with your code editor and may learn by way of your codebase to proactively recommend code completions and debug points inside your IDE. 
   • As I briefly talked about firstly, information instruments like Snowflake and Hex additionally began to embed AI coding assistants to assist information analysts and information scientists write code simply. 
2. AI for EDA and evaluation. That is considerably just like the Chat-based BI assistant instruments I discussed above, however their purpose is extra bold — they begin with the uncooked datasets and intention to automate the entire evaluation cycle of information cleansing, pre-processing, exploratory evaluation, and generally even modeling. These are the instruments normally marketed as “changing information analysts” (however are they?).
   • Google Data Science Agent is a really spectacular new instrument that may generate an entire Jupyter Pocket book with a easy immediate. I lately wrote an article exhibiting what it could do and what it can not. Briefly, it could rapidly spin up a well-structured and functioning Jupyter Pocket book primarily based on a customizable execution plan. Nonetheless, it’s lacking the capabilities of modifying the Jupyter Pocket book primarily based on follow-up questions, nonetheless requires somebody with strong information science information to audit the strategies and make handbook iterations, and desires a transparent information drawback assertion with clear and well-documented datasets. Due to this fact, I view it as an ideal instrument to free us a while on starter code, as a substitute of threatening our jobs.
   • ChatGPT’s Data Analyst tool may also be categorized beneath this space. It permits customers to add a dataset and chat with it to get their evaluation completed, visualizations generated, and questions answered. You’ll find my prior article discussing its capabilities here. It additionally faces related challenges and works higher as an EDA helper as a substitute of changing information analysts.
3. Simple-to-use and scalable NLP capabilities. LLM is nice at conversations. Due to this fact, NLP is made exponentially simpler with LLM at the moment.
   • My firm hosts an inner hackathon yearly. I keep in mind my hackathon undertaking three years in the past was to attempt BERT and different conventional matter modeling strategies to research NPS survey responses, which was enjoyable however truthfully very exhausting to make it correct and significant for the enterprise. Then two years in the past, in the course of the hackathon, we tried OpenAI API to categorize and summarise those self same suggestions information — it labored like magic as you are able to do high-accuracy matter modeling, sentiment evaluation, suggestions categorization all simply in a single API name, and the outputs nicely match into our enterprise context primarily based on the system immediate. We later constructed an inner pipeline that scaled simply to textual content information throughout survey responses, assist tickets, Gross sales calls, consumer analysis notes, and so on., and it has grow to be the centralized buyer suggestions hub and knowledgeable our product roadmap. You’ll find extra in this tech blog.
   • There are additionally plenty of new corporations constructing packaged AI buyer suggestions evaluation instruments, product overview evaluation instruments, customer support assistant instruments, and so on. The concepts are all the identical — using the benefit of how LLM can perceive textual content context and make conversations to create specialised AI brokers in textual content analytics. 

Conclusion

It’s simple to get caught up chasing the newest AI instruments. However on the finish of the day, what issues most is utilizing AI to get rid of what slows us down and speed up what strikes us ahead. The hot button is to remain pragmatic: undertake what works at the moment, keep inquisitive about what’s rising, and by no means lose sight of the core function of information science—to drive higher choices by way of higher understanding.