Lately I've been experimenting with using LLMs to build a kind of tool I still hesitate to even think about, because it's so foreign to anything we could build before. Internal tooling for analyzing our own teams: work that used to be too manual to do at any real scale without a large group of people, or without standing up sentiment analysis and custom ML models. Access to natural language as a building block has been such a paradigm shift that I'm still catching myself not ideating big enough. There is so much raw capacity here that I don't think any of us have fully mapped yet.
// A Simple Example
Here's one that would have been too cumbersome to build before: bug trend analysis.
As part of our PR process, we now require a section that captures the details of the change that caused the bug being fixed: who authored it, when it happened, and what they were trying to accomplish with the original change. (Something I only just thought of while writing this: add whether that same PR shipped with corresponding unit tests.)
The important part is that a human doesn't write any of that. The LLM generates it as part of its normal output while it works the fix. Getting that data from people would have meant research on every PR, plus coaching to get everyone filling it in correctly and consistently. Now we get it almost for free, or cheaply enough that it may as well be free, and it takes very little mentoring or shaping to keep it consistent.
// Then Ask Questions of It
Once that data exists on every PR, the rest is straightforward. Pull all the PRs for a time period, hand them to a model, and ask questions. What kinds of changes keep producing bugs? Does it correlate with missing tests? With certain areas of the codebase? Are we trending in the right direction over time, or just feeling like we are?
That last question is the one that matters. It turns a gut feeling about quality into something we can actually measure and watch move.
// The Part That Made Me Shudder
What stuck with me afterward wasn't the result. It was how little work the whole thing took to build and use. I started thinking about what it would have required before LLMs: designing a structured form, enforcing it, training people to fill it out, auditing the answers, building the pipeline, and probably a classification model to make sense of the free-text parts. Then I shuddered. A project like that would have died in a prioritization meeting, and nobody would have faulted us for it.
The tools we skipped weren't bad ideas. They were ideas whose cost was out of proportion to the insight. Natural language collapsed the cost, and a whole category of ideas just became viable.
// Why This Matters
I think ideas like this are the foundation of a lot of native AI tooling. They're also why AI-native products are trouncing legacy systems that are trying to bolt AI on and become AI-native after the fact. The legacy approach asks, "where can we add AI to what we already do?" The native approach asks, "what can we do now that was never worth attempting?" Those are very different questions, and over the next few years I expect the gap between the people asking each of them to be what separates the winners from the losers.
So my challenge to myself, and to you, is to go looking for the things you've quietly written off as too manual, too fuzzy, or too expensive to measure. Review quality, team sentiment, onboarding friction, incident patterns: anywhere the data exists as messy human language, there's probably a tool waiting to be built in an afternoon.
As my friend Brendan says, "Dare to dream a little bigger."