The Starting Point
In 2023, I started using AI as a tool for sports prediction analysis. The initial idea was straightforward: ask the AI to retrieve information about a game, analyze that information, and make a recommendation based on the data. The workflow was simple - give the AI a question, get an answer.
What followed became one of the most instructive experiences I've had working with AI systems. Sports events have clearly defined inputs (rosters, recent performance, matchup stats) and clearly defined outcomes, which made it a useful feedback environment. When the AI was wrong, the failure was visible and specific. That specificity made the failures fixable - and fixing them repeatedly is what turned this into a real study in prompt engineering and AI fluency.
This project has been running and evolving since 2023. The sports analysis is ongoing. The prompt system is still being improved.
The First Failure: Hallucinated Rosters
The first major problem showed up quickly. The AI was referencing players who were no longer on the teams being analyzed - athletes who had been traded, released, or retired. The model was pulling from training data that was months or years out of date and presenting it with complete confidence. There was no flag that this information might have changed. No acknowledgment of uncertainty. It just filled in the gaps from what it "knew" - and what it knew was wrong.
The impact was immediate: analysis built on wrong roster data produces wrong conclusions. But the more important realization was that this failure wasn't a fluke - it was structural. The model had no mechanism built in to distinguish between current and outdated information. That gap had to be addressed at the prompt level, not by the model itself.
Updating the System
I started modifying the prompt structure to address the specific failures I was seeing. This wasn't one fix - it was a series of targeted adjustments, each tied to an observed failure mode:
- Added explicit instructions to verify current roster status before producing any analysis
- Required the model to flag uncertainty rather than silently fill gaps
- Structured prompts to require step-by-step reasoning instead of direct conclusions
- Tested different prompt versions side by side to see which produced more reliable output
Each iteration followed the same pattern: identify what specifically went wrong, determine whether it was a data problem or a reasoning structure problem, modify the prompt to address that exact failure, test the result, and document what changed. The sports analysis was the feedback loop. Every incorrect output pointed at something fixable in the prompt design.
Closing the Loop: Controlling What the AI Knows
The most significant shift in the project came when I stopped relying on the AI to retrieve information itself and started providing the data directly. Instead of "research this team and analyze," the workflow became: I source the current stats, rosters, and relevant context myself, put that data into the session, and tell the model explicitly to use only what's provided.
This closed the loop on the biggest source of unreliable output.
When the AI controlled its own information retrieval, any data point it used could come from any source, any time period, with no verification on my end. By sourcing the data myself before it entered the prompt:
- Every data point in the analysis was current and verified before the AI touched it
- The AI's job became analysis, not retrieval - a much more controllable task
- I could see exactly what information the model was working from
- The model had no path to introduce outdated or fabricated data
This is about data integrity - the same principle that applies to any analytical system. Garbage in, garbage out. Controlling what goes in is the most direct lever on output quality.
The Iterative Engineering Cycle
Over multiple years, the prompt system grew progressively more sophisticated. Techniques that emerged through iteration and testing:
Step-by-step reasoning gates: Rather than asking for conclusions directly, prompts require the model to complete specific steps - roster verification, form analysis, matchup context - before reaching a recommendation. This prevents the model from skipping validation and jumping straight to an answer.
Hard information source constraints: Explicit instructions on what sources the model can draw from. If it wasn't provided in the session, it cannot be used. This constraint is stated clearly at the start of each analysis prompt.
Verified vs. assumed separation: The model must clearly distinguish what it knows from provided data versus what it is inferring. Any inference must be labeled as such. This makes uncertainty visible rather than buried in confident-sounding output.
Prompt versioning: Multiple versions of the same prompt structure, tested side by side. Versions that produce inconsistent, overconfident, or unverified outputs are revised. Versions that reliably produce structured, well-reasoned output are kept and built on.
Failure documentation: Every significant failure mode was logged: what the model did, what it should have done, and what specific prompt change addressed it. This documentation became the foundation for the next iteration.
The result over multiple years is a prompt system where the AI's reasoning is structured, its information sources are controlled, and its uncertainty is explicit. The sports prediction accuracy improved - but the more lasting output is a practical framework for making AI workflows reliable.
Core Lesson: AI Is Only As Good As Its Data
The clearest thing this project established is that AI capability and AI reliability are not the same thing. A model can be sophisticated and still produce wrong output if it's working from wrong, outdated, or uncontrolled information.
The gap between "can analyze" and "will analyze accurately" is determined almost entirely by the quality, currency, and sourcing of inputs. This framework now shapes how I evaluate any AI tool or workflow:
- What information does the AI have access to? And who controls that access?
- How current and verified is that information? Is it sourced and checked, or retrieved without validation?
- Is the reasoning transparent? Can I see the steps, or just the conclusion?
- Is the model constrained from filling gaps with assumptions? Or will it confidently produce output even when it should be flagging uncertainty?
These aren't abstract questions. They are the difference between AI output you can act on and AI output that creates false confidence in wrong conclusions.
Cybersecurity Relevance
These questions matter directly in security contexts, where AI tools are increasingly being integrated into SOC workflows, threat analysis, and incident response. The same failure modes that showed up in sports prediction appear in AI security tools - and the stakes are considerably higher.
- Hallucination in security tools: An AI-assisted SOC tool that produces wrong MITRE ATT&CK classifications, incorrect threat actor attribution, or fabricated IOCs creates dangerous blind spots. Understanding hallucination at a practical level - what causes it, how to detect it, how to reduce it - is directly applicable to evaluating and building AI security tools.
- Input validation: The discipline of sourcing and verifying data before it enters an AI analysis workflow maps directly to security data pipelines. Unvalidated log data, stale threat feeds, or unverified context introduced into an AI workflow produces the same class of failures as unvalidated sports stats.
- Behavior control through prompting: Controlling what a model can reason from, what steps it must complete, and what it must flag as uncertain is equivalent to access control for AI. It determines what the system can and cannot produce - and who is responsible for what it outputs.
- Iterative improvement culture: The test-identify-fix-document cycle developed through this project is directly applicable to any AI tool integration in a security environment. AI tools in security should be tested and iterated on, not deployed and trusted blindly.