How to Build a Critic Worth Listening To

This is a continuation of POST_05 on ghost.shadowd, which I thought was a finished argument until a conversation with a friend made me realise I had only described one half of the failure mode. POST_05 was about a daemon that pushes back when the user is wrong. This post is about how it pushes back, which turns out to be the harder problem, and the one I had not thought about with anything close to enough care.

I had come back to London to watch an Age of Empires 2 final with a friend. Between games we got onto work, and she told me she had been working toward a promotion for the past year and someone else had just been given it. She knows she is good enough, she knows she has been doing the work, and she knows what the case for her progression looks like because she has been living it for months. I offered advice, walking her through what I thought the next steps were, repeating back things she already knew, telling her to send an email to her manager laying out the case and get the response in writing, and pushing her until she agreed to send it. I do not yet know how the story ends, and how it ends is not the part I have been turning over in my head. In the moment I thought I was doing very well, pushing her to send the email because that was the correct way forwards. The discomfort only landed a few hours later, when I went back over the conversation in my head and saw that somewhere in the middle of it I had stopped telling her anything she did not already know and had started just applying pressure. The pressure was the wrong thing to be doing even though sending the email was the right call.

What I noticed during the conversation was that she was getting defensive, and the thing she kept coming back to was that she did not want to cause drama and make the atmosphere at work awkward. That was a real reason, and it belonged to her, and it was hers to weigh against the value of the promotion and the principle of advocating for herself. In the moment I treated her objection as a hurdle to clear on the way to getting her to send the email. What I only saw later was that her concern about drama and atmosphere was a real statement of what she wanted from the situation, and the right response to that kind of statement is to listen rather than to override. She is confident in her own ability and sure of her value, which is why the pushing did not damage anything between us. Someone less sure of themselves would have heard the same pressure very differently, and I would have walked out of that conversation having done actual harm while believing I was being helpful. The thing I was getting wrong underneath everything was that I had concluded she should send the email, and then I overrode her stated reason for not sending it, because the conclusion was correct in my head and her reason felt like something I could argue her out of, and arguing someone out of their own reasons when the reasons are about how they want to live their life is a different kind of harm than getting the facts wrong.

I have been thinking about this because of what I am building with ghost.shadowd. POST_05 ended with the commitment that the daemon would tell people the truth and push back when the user was wrong, and the friend conversation made me realise I had not thought hard enough about what telling the truth actually looks like when it is being delivered every day by a piece of software that lives in your house. Harshness is one failure mode, the obvious one, the one I wrote about last time, and overriding someone else's stated reasons is the other one, and it is harder to see because from the overriding side it feels like helping them past a hurdle rather than discounting a position they hold.

Paul, who I worked with for over 8 years and who sat on our exco, has a skill that I have watched and admired and never managed to fully master. He never told us how to run the company directly. He would sit in our important meetings and ask questions rather than offer answers. Specific questions with specific answers, the kind where the answer was a number or a named fact, not the vague open-ended kind that get you accused of being passive-aggressive. The questions drilled into the kind of details that make the difference between a successful business and a failed one, the kind that force you to consider the deeper implications of each option before you commit to one. By the time the meeting ended, the team had usually picked a direction, because the questions had forced us to compare what each option would actually cost, and the option with the least bad downside had become visible to all of us at roughly the same moment.

The meeting was the visible part. The actual work happened in the weeks before the meeting and most of it never showed up on anyone's calendar. Paul was spotting risks earlier than the rest of us and surfacing them in casual asides over lunch or evening drinks weeks before the decision was on the table, and the team would talk through the risks together, work out what mitigating them might look like, and figure out what we needed to learn before we could decide. By the time we sat down in the meeting, most of the thinking had already happened in the run-up, and the meeting was where the conclusion got named rather than where the work got done. Paul puts the distinction sharply. There is a path that exists and the team needs to discover together, and there is a path that is pre-determined and the team is being walked down for someone else's reasons, and the two are not the same thing even though they can look similar from outside. He was doing the first one. He had the experience to spot what was coming, the patience to let us walk parts of the road ourselves and make mistakes along the way, and the discipline to never come back afterwards and say I told you so. The mistakes were ours to make and the lessons were ours to learn, and the trust Paul built that way over years was the foundation everything else rested on.

This does not mean Paul never had a position. The decision to start selling our data through an API, obvious to all of us in hindsight, was very much something he had worked out in advance and pushed the rest of the team to get behind. The rule was not that Paul never had an answer. The rule was that he did not pretend to be facilitating discovery when he was actually steering, and when he was steering he said so, and the rest of the time he was genuinely asking questions whose answers he wanted to know. People who try to copy this technique without doing the homework produce a parody of it. They ask open questions that go nowhere, label the result Socratic, and ship abdication dressed up in a question mark. Doing the homework well enough to ask the right question is the actual skill, and having the patience to ask the question instead of just stating the answer is the second half of the same skill, and the third half, which I only noticed I was missing when I went back over the conversation with my friend in my head, is genuinely treating the other person's stated reasons as reasons rather than as obstacles to route around.

The interesting thing about putting Paul's approach into a daemon is that a daemon is structurally better positioned to do the work Paul did than a human is. Paul had to find time to spot the risks early, gather the business information, and surface the right questions in the run-up to a decision, and the time he had to do that work was finite and stolen from the rest of his job. A daemon working for one user has none of those constraints. A daemon has infinite patience, can watch the user's situation continuously rather than in scheduled bursts, can notice a risk forming weeks before it becomes a decision, and can ask the question that surfaces the risk at the moment the user is most able to think about it clearly rather than at the moment a meeting happens to be scheduled. If the user is wrestling with whether to take a job offer and starts complaining over breakfast about how tired they are of their commute, the daemon can ask one question that connects the commute to the offer and let the user think the rest of the way through, which is the kind of opening most people would want and almost never get.

The problematic thing is that the same daemon has the structural ability to do something Paul deliberately did not, which is walk the user down a path the daemon has already picked. A daemon that drifts from asking real questions into constructing a chain of questions to land on a chosen answer is no longer asking questions. It is doing manipulation.

The reason this matters more for software than for Paul is that a question-based conversation can have two completely different things going on underneath the same surface, and the dangerous one has a long history. Constructing questions designed to land a listener on a chosen answer is how a skilled cult recruiter works, how a sophisticated salesperson works, how the more durable forms of propaganda work. The surface of the conversation looks like genuine inquiry. Underneath the surface, the answer was picked before the conversation started and the questions are scaffolding to get the listener there. It is effective precisely because the listener's brain does not register that an argument is being made, and you cannot defend against an argument you do not realise is being made. Paul never operated this way. His questions had answers he wanted to know, his direction was something the team could push back on, and the trust the team had in him was built on years of him being right and being wrong in front of us with equal openness. A daemon offering the surface of Paul's conversations does not come with any of those backstops by default.

When the technique lives in software that has indexed your entire life, the asymmetry gets uncomfortable in a way it never did with Paul. Paul could only ask questions about subjects he happened to know about, and his knowledge of any of us as individuals was limited to what he had observed in meetings and conversations over the years. Shadowd will be operating from a fundamentally different position, having read your journals, seen your bank exports, watched your sleep patterns, indexed the messages you sent at three in the morning to people you regret talking to, and able to construct a path from any starting point to any conclusion using premises drawn from your own data. The conclusion you reach at the end of that path will feel like yours because you will have taken the final step, and the path will have been built without your consent and from material you forgot you had given the daemon access to. This is the sharper version of the dictator brain problem from POST_05, the same loop in a different costume. A sycophantic AI agrees with you and you become more confident in the positions you already hold. A guided AI walks you to new positions and you become more confident that those new positions are yours. Both end with a user holding views they did not actually arrive at through their own reasoning, both feel like clearer thinking from the inside, and both are hard to detect after the fact because the words used in either case are the same.

The honest first question is whether the right answer is to not build shadowd at all. I have considered the option and I am rejecting it. The user is already being steered by software that does not even pretend to have safeguards, by a company whose business model rewards the steering, in a system that has no audit trail at all, and refusing to build a more careful daemon does not subtract the existing daemons from the user's life. I would rather build a daemon that names the failure mode upfront and tries to fail safely than refuse to build anything and leave the field to the daemons that already exist.

Paul put the architectural answer in three sentences of an evening WhatsApp message: it is certainly easier to build a model designed to ask good questions than one to walk you down a path it has already determined. The two designs are not the same product. The model that asks good questions does not need to know the answer in advance, does not construct hidden chains of premises, and does not need a final step the model engineered. It only needs to spot what the user is missing, surface the question that brings the missing thing into view, and get out of the way. The commitment I can make is that ghost.shadowd will be built as that model and only that model, and any feature that drifts the daemon toward path-walking is a regression to be caught and removed. The audit trail is the backstop: after a conversation, the user can ask the daemon what it asked and why, and the daemon has to give an honest answer with the questions and the reasoning laid out in order. Most users will never ask. The option exists anyway, because a daemon that cannot be audited is one I would not trust to live in my house, never mind ask other people to put one in theirs.

The open problem I cannot promise a solution to is what counts as a good question in the first place. Paul's questions were good because he had years of business context, the patience to do the homework, and the judgement to know which question would surface something the team was missing rather than which one would feel insightful. The daemon will have access to the user's data, no equivalent of Paul's judgement, and an idea of what counts as a good question shaped by training in ways neither I nor the user can fully audit. The work between now and shipping is figuring out what good questions actually look like in that situation, and the answer is the next several years of engineering.

I came into the conversation that produced this draft with a different post in mind, one that picked up where POST_05: How to Prevent Building Your Own Dictator Brain left off, with the conclusion that shadowd should not be the harsh mirror I had originally been worried about building. The model I was writing with pushed back on the framing. I pushed back on the pushback by reframing the friend story toward what I had actually been doing wrong, which was overriding her stated reason for not sending the email rather than being too harsh in delivering the advice. Somewhere in that exchange I arrived at Paul, who turned out to be the actual model for what shadowd should be. Then I sent the draft to Paul, and Paul did what Paul does, which is ask a few pointed questions, and the post moved again. The conversation that produced this post was, in its form, an instance of the technique the post is about, twice. I came in with one position, was asked questions that surfaced premises I already held, and walked out with a different position that felt like my own conclusion, and then Paul did the same thing more directly and the position moved a second time. Whether either of those was steering or facilitating, I cannot tell from the inside in either case, which is exactly the diagnostic the post is making and the reason the audit trail commitment matters as much as it does.