Policy admin system at work. Java, around 420 thousand lines, twenty years old, most of the original decisions undocumented and the people long gone.
Anything inside one package is fine. Anything that crosses packages gets me an answer that reads well and is wrong, because it never opened the file where the behaviour actually lives. I have watched it describe a validation rule confidently, based on nothing but the class name.
What I have tried. Feeding it the file tree, too big to be useful. Pointing it at specific files, which works but requires me to already know the answer. Asking it to search first, which helps until it stops after two or three hits and decides it has enough.
What are people doing on genuinely large old systems? I am not asking for it to understand the whole thing. I want it to reliably notice when it has not read enough to answer.
Nothing that solves it, but two habits that move the needle.
I keep a map of the system, one page, written by me, saying which area does what and where the surprising parts are. It goes into every session. It is not documentation for humans, it is orientation, and it stops the class name guessing because it now has something better to guess from.
The other one: ask for a file and line reference for every claim. When it cannot produce one, that is your signal. Claims without references are the invented ones, fairly reliably.
The reference rule matches what I do with the makefiles. Ask the question, then ask which file that came from, and half the time the second question is where it falls over.
On old code I also have it summarise one package into a paragraph, I check that paragraph myself, and the checked paragraphs go in a file I reuse. Slow to build up. But they are mine once corrected, and handing it twenty checked paragraphs is much cheaper than twenty thousand lines.
Pushing back gently on the map, not because it is wrong but because on a system that old, the map is precisely the thing nobody has. That is what Daniel is asking how to get.
The version I have seen work is building it off real questions. Every time you answer a cross-package question the hard way, write down what you learned in the map file. After a month it covers the parts people actually ask about, which is a far smaller system than the whole one.
Thanks both. Sofia, yes, that is the bind exactly. Started the map anyway, page and a half so far.
File and line references are good and I have caught two invented answers with it already. But the thing I actually asked, getting it to know when it has not seen enough, I still do not have. It will happily hand me a well referenced answer that is correct about the two files it read and wrong about the system.
Leaving this open. If somebody has cracked it I would like to hear how.