Observe Makes Passing
in pc science was something however straightforward. I vividly bear in mind reaching a breaking level across the finish of the tenth week of my first semester. With just some weeks till my first remaining, I sat watching Calc 1 observe issues, spiraling into despair. I’d at all times been good at math. I did all of the homework and paid consideration in all of the lectures. So how may it’s that I didn’t even know the place to start out? Why wasn’t something clicking?
I usually joked with mates about dropping out of this system, even properly into my remaining semester. Week 10 of Semester 1 was the one time I very critically thought-about it.
It was January 2022, proper on the heels of the COVID tech hiring growth. I’d tried my hand at frontend improvement and had a fairly good grasp of React. Not one of the introductory math programs I used to be taking made any sense. Loads of acquaintances and mates of mates had gotten soft tech jobs with out levels, so why couldn’t I? What use was understanding the best way to show a operate was steady out in the true world?
Looking back, I understood that that was precisely what I used to be alleged to really feel. That was after I really determined to pursue my diploma, not after I utilized a yr earlier. That feeling of impending doom was what lit a hearth beneath me and drove me to review like a person possessed for the following few months.
To this present day, I’ve by no means been happier to get again a grade than after I opened the scan of my graded Calc 1 examination to see “61/100” staring me again within the face: a passing grade with a cool margin of two factors above failing. However all that mattered was that it was a passing grade, particularly when virtually half the scholars had failed the category, many for the second or third time.
By all accounts, my first semester of undergrad was tough. Sure, this was by design, and sure, I realized rather a lot from it, each by way of the fabric itself and (principally) about resilience and perseverance. Nevertheless it took shifting to Germany and beginning my grasp’s for me to know how good I actually had it again then, at the least in a single explicit regard.
The Human Coaching Knowledge Drawback
One of many greatest surprises to me at my new college was that previous exams are a lot much less of a factor right here. For all of the stress and nervousness I had throughout my bachelor’s, one factor I knew I may at all times rely on was the existence of plentiful and easily-accessible scans of previous exams and exam-relevant drawback units, particularly for introductory programs.
For Discrete Math, I solved all the handfuls of previous exams going again virtually a decade. I distinctly bear in mind warming up for Linear Algebra 1 with questions from the Nineties. This was so ingrained within the tradition of my program that I utterly took it with no consideration. The one cause I managed to move Calc 1 (by the pores and skin of my tooth) was as a result of I had spent hours on finish fixing a whole bunch of questions from exams.
I used to be so accustomed to exams from previous years being available that skimming over them had develop into a part of my course of for vetting lessons I used to be contemplating taking. This meant that my impolite awakening got here pretty early on in my first semester of grad college, whereas making an attempt to determine my schedule.
So stunning was the revelation that I can map my response to the 5 levels of grief. At first, I used to be in denial, completely satisfied that there should be some secret platform the place all of the previous exams have been hiding. Anger, bargaining, and despair quickly adopted. Acceptance didn’t actually, however I used to be keen to postpone my considerations till finals got here nearer on the finish of the semester.
As my first two finals (on back-to-back days, no much less) approached in a rush, I discovered myself confronted with what I prefer to name the Human Coaching Knowledge Drawback. Granted, the human mind and machines are (very!) considerably totally different. However I couldn’t assist however liken my state of affairs to that of a machine studying mannequin with inadequate coaching knowledge. I used to be utterly stumped on the best way to bridge the hole between lecture notes and potential examination questions.
My undergrad expertise had granted me the perception of what human underfitting appears to be like like, each at coaching time (learning) and take a look at time (on examination day). I vividly bear in mind a couple of class the place, for one cause or one other, I most popular extra in-depth overview of lecture slides or notes to fixing observe issues.
This was an strategy I rapidly dropped throughout my freshman yr, and for good cause: even in theory-heavy lessons, it yielded disastrous outcomes. Understanding the proofs for all 40 theorems the professor required was a lot much less assist in passing Linear Algebra 2 than training making use of them to resolve issues would have been. That’s to not say an enough grasp of the speculation isn’t mandatory; it completely is. However having the ability to recite the lecture notes by coronary heart received’t prevent for those who can’t reply questions like those on the ultimate.
And so, armed with a whole bunch of slides and a imprecise thought of the construction of every examination, I racked my mind for tactics to keep away from the pitfall of entering into blind with none observe issues. Denial crept again in, and I desperately looked for previous exams I knew didn’t exist. Ultimately, I shifted my consideration from discovering the Holy Grail to turning my drawback into one an LLM may have the ability to clear up.
Artificial Coaching Knowledge for People
Researchers at IBM outline artificial knowledge as “data that’s been generated on a pc to reinforce or exchange actual knowledge to enhance AI fashions” [1]. It has many advantages, from mitigating privateness considerations to reducing prices, resulting in its widespread adoption for makes use of as various as tooling for monetary establishments [1] and 3D content material technology [2].
In my case, the motivation was easy: the real-world (human) coaching knowledge I wanted to review simply wasn’t accessible within the wild.
After all, utilizing artificial knowledge solely is smart if that knowledge precisely imitates the information our skilled mannequin will encounter in the true world. I knew I needed to be very intentional about how I generated the mock exams I needed to make use of. Simply telling Claude to put in writing a observe take a look at or two wouldn’t lower it, even when I gave it all of the slides and materials I needed to work with. Solely when getting down to write an examination does one understand what number of choices there are to be made, properly past what’s in and what’s out by way of the fabric.
Fortunately, I wasn’t flying utterly blind on that entrance. For one class, I had details about the examination’s construction and the sorts of questions there have been on it from college students who had taken it the yr prior. For the opposite, the professor offered a breakdown of the examination into sections and a small handful of open-ended overview questions.
Each lessons had Q&A periods after their respective remaining lectures. I paid particular consideration to something that appeared like a touch as to what they may ask, which later proved to be very useful.
Simple Mode: Replicating a Template
The primary examination was easy since I had way more to work with. It additionally had a status for being comparatively formulaic. I gave Claude the instance questions and construction I had and requested it to stay to the identical model.
Most of the questions lent themselves properly to slight adjustments that made them novel sufficient to be price fixing for observe with out straying too removed from what was typical for the precise examination. Other than a couple of LaTeX formatting hiccups, which have been pretty simply resolved, it was easy crusing.
To insure myself in opposition to any surprises, I additionally had it generate some trickier questions primarily based on the lecture slides and my notes from the Q&A session. Despite the fact that nothing surprising was requested ultimately, doing a little focused overview tailor-made to my very own private blind spots was an awesome confidence booster.
Though I undoubtedly would have been in a position to examine for the primary examination with out the assistance of LLMs, I nonetheless felt like I gained rather a lot through the use of Claude. I may completely think about how useful it could have been for among the newer or extra superior programs I took in undergrad, the place there have been solely a small handful of previous exams accessible.
Laborious Mode: Development from Scratch
The second examination was a a lot harder nut to crack. To start with, the breadth of the fabric was a lot wider. Secondly, the slides solely very loosely mirrored what was mentioned at school. Most significantly, there was far much less data accessible on what the examination would appear like. What particulars there have been have been exhausting to seek out and imprecise.
The primary two considerations have been at the least partially mitigated by the truth that I made an effort to take complete notes all through the semester. As for hints on the construction and elegance of the examination, I scoured each potential platform and picked up something that appeared even remotely related. In that vein, the Q&A session ended up being a godsend. Transcribing the professor’s solutions and feedback left me with a significantly better (albeit nonetheless incomplete) thought of what to anticipate.
Admittedly, I used to be initially pessimistic in regards to the prospect of Claude having the ability to generate mock exams of a lot worth. Although I had used it pretty extensively for guided materials overview, I had my doubts about how it could fare with the uncertainty at play. Nonetheless, I gave it all the things I knew in regards to the examination and hoped for one of the best.
I used to be pleasantly shocked on the outcomes. Though the primary few makes an attempt produced exams that didn’t really feel fairly proper, the core did appear promising. They did seem to adequately cowl the fabric and to be difficult sufficient. After some backwards and forwards, Claude began producing checks that I may have been satisfied have been actual.
I solved the improved checks and requested Claude to appropriate my options. The very act of fixing observe checks made me really feel nice about my grasp of the fabric. Claude’s traditional sycophancy was the cherry on high. (It did level out errors, however was exceptionally gentle on deducting factors and overly-excited about appropriate solutions.) Finally, nevertheless, I wouldn’t understand how properly Claude had executed coaching me till take a look at time. With the fateful day quick approaching, I hoped for one of the best.
Generalizing to Take a look at Knowledge and Stopping Dataset Air pollution
When Artificial Knowledge Alone Doesn’t Lower It
Whereas artificial knowledge definitely has its advantages, it has a important downside. What a mannequin learns primarily based on artificial knowledge will, at finest, mannequin the simulated world from which that knowledge is drawn. That simulated world may diverge from actuality in methods we’re utterly unaware of till it’s too late [3].
As Dani Shanley places it in “Synthetic data, real harm,“
“… simply as generative AI fashions can produce believable (however false) textual content or photos, artificial knowledge mills might create datasets that seem statistically legitimate, whereas introducing delicate, hard-to-catch distortions and synthetic patterns, or lacking essential real-world complexities.” [3]
Shanley additionally attracts consideration to the hidden and disproportionate impression of the people tasked with synthesizing knowledge on how fashions in the end behave. Largely arbitrary choices on their half may have important, probably dangerous, downstream results [3].
I noticed this impression in motion whereas learning for my second examination. Slowly however certainly, I had unintentionally skewed Claude’s outputs primarily based on my private interpretation of what the professor had stated. My intestine feeling on what the examination ought to appear like grew to become the arbiter of which questions have been related and which weren’t.
It additionally grew to become clearer as time went on that my coaching dataset was veering ever additional right into a biased tackle actuality. After the sixth mock examination, it was apparent that Claude had simply settled on a hard and fast set of a number of dozen questions.
Even when prompted to introduce extra selection, each output from there on out was just a few cobbling collectively of questions I had already seen. Granted, these did embody many key questions it was closely implied would seem on the precise examination.
On take a look at day, I used to be shocked at how a lot the examination resembled those I had solved for observe. The gimmes the professor had hinted at have been certainly there, however so have been a formidable variety of non-trivial questions I had solved whereas learning. Roughly 60% of the questions have been an identical or similar to ones I had practiced. Most of the relaxation have been on subjects I had at the least touched on.
Nonetheless, one a part of the examination ended up being a major blind spot. It was a piece on subjects we had mentioned solely briefly at the start of the semester. Whereas learning, I used to be unreasonably assured in swiftly dismissing sure kinds of questions, be it as a result of they appeared uncharacteristic (e.g., too mathematical) or as a result of they have been about issues I had deemed too insignificant to incorporate within the notes I took at school.
Sadly, these turned out to the precise kinds of questions that have been requested in that part. Some have been about subjects that solely appeared on a single slide all semester. Others have been deeply technical in a approach I simply didn’t count on. Although I did my finest to reply them, I hadn’t skilled my psychological mannequin on knowledge that will allow it to generalize to those questions properly sufficient.
The tablet was all of the extra bitter to swallow because the sorts of questions I struggled with have been ones Claude included in its first makes an attempt at mock exams. These have been exactly those I did away with early on primarily based on little greater than hunches.
On this case, the slip up was removed from catastrophic. In my view, it wasn’t even near undoing the advantages of learning utilizing artificial mock exams. Even so, it serves as a cautionary story that hearkens again to Shanley’s warnings about how artificial knowledge can insidiously exacerbate mannequin subjectivity and bias [3].
Overcoming Overfitting: Learn how to Make the Better of Artificial Human Coaching Knowledge
For a lot of real-world functions, an artificial dataset that yields a mannequin with solely 60% accuracy would in all probability be thought-about subsequent to ineffective. With enough real-world knowledge (i.e., precise previous exams), there isn’t any doubt in my thoughts that 90%+ accuracy could be achievable.
To be honest, although, the (human) mannequin into consideration has flaws that machines don’t and is, in some ways, a lot more durable to coach. I can say with confidence that that 60% would virtually definitely surpass the accuracy of every other methodology I may have tried.
I’ll completely follow this methodology for future exams, with three key takeaways I plan to implement:
- Separate chats are the way in which to go. The suggestions loop that led Claude to converge on particular questions undoubtedly had rather a lot to do with me operating the complete cycle of producing checks and checking solutions in a single massive, lengthy context. This meant any new mock examination was instantly primarily based on the entire earlier ones. Past that, Claude tried to be useful by tailoring the inquiries to what it thought have been my weak spots, main it to develop into much more entrenched in what it thought needs to be requested. Normal context rot(1) was additionally in all probability an essential issue.
- Hold an open thoughts. As talked about above, the main blind spot I developed was largely the results of placing an excessive amount of inventory in my subjective evaluation of what materials would or ought to make the lower. As an alternative of difficult my assumptions and devoting a while to masking minor subjects that appeared like lengthy photographs, I leaned into my biases.
- Increase with real-world coaching knowledge! That is, after all, simpler stated than executed. It considerably contradicts the very premise of this text. However what you are able to do as a scholar (or as an educator) is enrich the financial institution of recognized questions for future college students. I managed to recollect a lot of the questions that have been on my second examination and doc them for future college students to make use of when learning.
Afterword: My Ideas on LLMs as a Studying Help
The elephant within the room is that not one of the examination preparation workflow I described would have been even remotely possible after I began my bachelor’s in late 2021. Possibly that is what made the method really feel virtually magical to me.
I bear in mind wishing I had a technique to routinely examine and proper my solutions on mock exams when learning in my freshman yr.Should you would have informed me again then that an AI device, not to mention a free one, would have the ability to try this (nevertheless imperfectly) in 2026, I’d have thought you have been loopy.
A lot has been written in regards to the new issues LLMs have led to. Most of the factors which have been made are particularly related to college students. And certainly, I can’t argue that claims like “AI is making folks dumber” are utterly unfounded. I’ve seen firsthand how these instruments let an individual outsource pondering and eradicate any mental discomfort. For an ever-growing vary of complicated duties, they characterize the last word shortcut [4].
Concerningly, I imagine individuals who resist the temptation to take these shortcuts are more and more being penalized, at the least within the brief run. A pal who was the one one to not vibe-code assignments in a sure class involves thoughts. Others cruised to excellent grades on their homework regardless of threats about how AI-generated submissions would supposedly be rejected. He put within the work and ended up being docked important factors for minor errors, with little in the way in which of constructive suggestions or recourse.
Nonetheless, in the long term, it’s a well-established proven fact that progress, in its myriad kinds, entails some form of stress. A kind of kinds is studying, and the required stress comes within the type of lively engagement with the fabric. Few issues are extra rewarding for my part than the lightbulb second of lastly understanding a troublesome idea after battling it for hours or days. Experiencing such moments with Fourier sequence, reductions, metric areas, and plenty of different ideas was a serious a part of what led me to decide on to pursue a grasp’s diploma within the area.
LLMs undoubtedly allow would-be learners to deprive themselves of this stress and, in flip, of precise studying. Typically, although, I feel too little consideration is paid to the opposite facet of the coin: with the fitting strategy, they’ll personalize and democratize studying like no invention because the web has.
Having skilled greater schooling each pre- and post-ChatGPT, I really feel enormously lucky to have instruments like Claude and Gemini at my fingertips. Their utility for examination preparation was simply the tip of the iceberg. It felt like my productiveness was boosted tenfold all through the semester. Issues clicked a lot sooner than they ever would have in any other case. LLMs have been a sport changer for all the things from technique (when and the best way to examine what) to reviewing slides and notes to growing real curiosity and curiosity within the materials.
To summarize with a platitude: “With nice energy comes nice duty.” LLMs are what you make of them. With the fitting strategy, they’ll coach you to tackle the heavy lifting as an alternative of doing it for you.
Should you loved this text, please take into account following me on LinkedIn to maintain up with future articles and initiatives.
Footnotes
(1) Engineering at Anthropic defines context rot as a phenomenon the place “because the variety of tokens within the context window will increase, the mannequin’s skill to precisely recall data from that context decreases.” [5]
References
[1] Ok. Martineau and R. Feris, “What’s artificial knowledge?,” IBM Analysis Weblog, Feb. 7, 2023. https://research.ibm.com/blog/what-is-synthetic-data.
[2] Y. Shi, P. Wang, J. Ye, M. Lengthy, Ok. Li, and X. Yang, “MVDream: Multi-view diffusion for 3D technology,” arXiv preprint arXiv:2308.16512, 2023. https://doi.org/10.48550/arXiv.2308.16512.
[3] D. Shanley, “Artificial knowledge, actual hurt,” Ada Lovelace Institute Weblog, Sep. 18, 2025. https://www.adalovelaceinstitute.org/blog/synthetic-data-real-harm/.
[4] S. Bogdanov, “In the long term, LLMs make us dumber,” @desunit (Sergey Bogdanov), Aug. 12, 2025. https://desunit.com/blog/in-the-long-run-llms-make-us-dumber/.
[5] P. Rajasekaran, E. Dixon, C. Ryan, and J. Hadfield, “Efficient context engineering for AI brokers,” Engineering at Anthropic, Sep. 29, 2025. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents.
