Why Does the AI Keep Forgetting What I Told It?: From "I'll Remember That" to Letting It Run Itself

What this article covers

You explain something to an AI, revise it three times, and it finally says "I understand, I'll remember that". Next time, on the same task, it goes back to the original way. The crux is the word "remember": it has three meanings for an AI, and only one of them survives to the next conversation.

This piece comes in three parts, and you can read only the part you need:

• Sections one to two: how to tell the three kinds of fake remembering apart, plus seven follow-up questions you can use today
• Section three: turning those questions into a checklist the AI runs itself, with a copy-ready example
• Section four onward: the advanced part. How two AIs share one set of material, what to look at before copying someone else's skill package, how to set acceptance criteria, the three depths of mutual review, the four rules I write into a loop, when to stop, and finally a real case from a participant

1. "I'll remember" has three meanings, and only one survives to next time

That is what I opened the talk with. Learning a tool and learning to direct a tool are two different things. And the first gate of directing is whether it actually took in what you said.

I was full of suspicion from my first day with AI, but my suspicion pointed the wrong way at first. I thought what I had to guard against was it answering wrongly. Later I found what I had to guard against was it answering correctly and then not doing it.

The earliest time, I explained a rule many times over, and it said it had remembered. I asked: where did you remember it?

Its answer was honest: sorry, my remembering only exists inside this conversation window.

In plain language: close this conversation and it has forgotten. That is the first kind of fake remembering.

What I did at the time was to state the requirement clearly: I want your remembering to mean that from now on, in a new task, a new window, you still remember. Only then did it propose two approaches, one being a work log in a fixed directory, the other a skill package.

The file got written, and I thought that settled it. But next time it still did not follow. Because a file existing and a file being read are two different things. If nobody tells it to read, it does not read. That is the second kind of fake remembering.

The last one is the hardest to spot. I wrote the rule as a skill package, so surely now it would start up automatically. Until one time I found it still had not, and after I pressed, it said: I did save it as a skill package, but that trigger phrase was not set up properly, and I did not consider this workflow to be the same thing as this task.

It was not lying to me. The package was there and the content was right. It simply judged that "this task now" and "the task that package handles" were not the same thing, so it never called it up.

I fell into all three of these, one at a time.

2. Seven follow-up questions, copy them and use them

When an AI says "I finally get it, I understand, I'll remember", I do not reply "good". I ask all of the following in one go.

Group one: what exactly did you record

This group tests understanding. Plenty of times it can recite the words but cannot say why it should be done this way, and that means it will drift the moment the situation changes.

Group two: where did you record it, can you find it

Question three knocks out "remembered in the window". Question four knocks out "file exists but never activated" and "trigger phrase not set". These two I added later, because all three of the pits above leaked here.

Group three: what if it goes wrong

This group matters most and gets asked least. The first six questions are still about "will it follow through this time". Questions five to seven ask "when I am not here, does it have a way to notice, fix, and accumulate on its own". Only when it can answer these three do you have a chance of handing the work over.

3. Turning the questions into a checklist it runs itself

I had very little confidence about this. I am not an engineer, I cannot write automated workflows, and most of my work is abstract analysis and judgement, which is hard to break into steps. My daily reality back then was: the AI runs one step, I correct it, it runs another, I correct it again.

Later I wrote it into a skill package, and it could run three to five steps at a time, but every three to five steps it still needed me to stop and correct. The real turning point was noticing one thing: those correction points were the same every time.

In the same workflow, I stopped at the same few places to check, and the standard never changed either. If the standard is that clear, could it review itself? And if it does not meet the standard, correct itself until it does?

That is what loop engineering means: writing the check you keep doing into a rule it follows.

What a checklist looks like

Plenty of people get stuck here: they know what they check every time, but they do not know what "written as a rule" is supposed to look like. Here is one office example, walked through from start to finish.

Suppose every month you review a departmental expense summary, and every time you look at these five things: whether dates cross into another month, whether invoice numbers repeat, whether department codes are right, whether the total matches the number of receipts, and whether any approval field is blank.

Step one, write the five items as a passage of rule text. Plain language exactly like this, no syntax needed:

Step two, decide where it lives. Browser only: paste it into custom instructions, or open a "month-end" project and put it in the project instructions. Using a desktop agent: save it as a file in a fixed folder, or make it a skill package. The location does not matter. What matters is that you can open it next time and that it goes and reads it on its own.

Step three, confirm it actually ran. This step gets skipped most often. Open a brand-new conversation, say only the sentence you would normally say ("take a look at this report for me"), and see whether it reports back on the five items by itself. If it does, the trigger phrase is written correctly. If it does not, it did not recognise that this sentence relates to this rule, so go back and add what you actually say to the "when to use me" section.

The difference between a trigger that fires and one that does not

Almost all cases of the third kind of fake remembering come from the "when to use me" section being written too abstractly. Compare:

You can read the left-hand style perfectly well, but it has to decide whether "what the user just said counts as finance-related document processing", and that makes it hesitate. The right-hand style writes out the actual sentences you say, so the decision becomes a comparison.

One more pit: when two rules have overlapping trigger conditions, it may only call up one of them. So do not keep adding more and more rule files. Keep one kind of work in one file where you can.

There are two other causes of a trigger not firing, less common but worth knowing: one is that this conversation is stuffed with so much material that the rule got pushed out of its reading range (the context being full, as above); the other is that the rule file sits somewhere it never looks. So the step-three test has to happen in a brand-new, clean conversation, which is how you tell a phrasing problem from a location problem.

An example I run myself

I have a publishing workflow: while out, I send a YouTube link from my phone with a voice instruction, and the desktop machine analyses the content, drafts a short post in my own voice, makes the image cards, and publishes. For the images I hard-coded four checks: whether the format is 4 by 5, whether it has been compressed and converted, whether the text on the image has drifted, and whether the character design has drifted.

Those four are exactly what I used to look over by hand every time. Written into the workflow, it checks and fixes them itself, and I only guard the copy gate.

To be honest about it, the most recent run of this loop only half worked. One image never got checked, because my rule only said "check twice" and anything from the third onward was not covered; the short-post platform published fine, the other two social platforms did not go out, and it did not go and find that itself. What I did afterwards was simple: went back and had it publish the missing ones, then added "every single image has to be checked" and "all three platforms report success or failure" to the rule.

The five stages of this workflow, the complete wording of the four image checks, and the other gap that failed silently are written up in Can I Get the Computer at Home Started While I Am Out?: One QR Code Scan and the Desktop Begins.

When a loop drops a stage, it is usually because that stage's checkpoint was never written in. Find one missed check, add one checkpoint, and the workflow moves forward one notch.

4. Before two AIs can work together, give them the same material

By this point you have a checklist that runs itself. The next move is bringing in a second AI to find the faults. Before that there is one prerequisite: both sides need to be able to read the same thing.

The benefit is direct: when the two review each other, both can find the same material, and you are not pasting things back and forth in the middle.

Someone at the talk asked whether piling everything into one folder gets messy. My answer was yes.

The fix is that this folder needs a set of conventions inside it, so every AI reading it knows what goes where, how to find things, how to use them. That can be handled with a classification and retrieval scheme, which is outside the scope of this piece, but you should at least know this: making material interoperable is fast, and the price is that it keeps piling up, so the conventions have to grow along with the material.

How to pick the tools

My main setup at present is two desktop agents: Claude and Codex. The reasoning is practical. One monthly subscription with each, and on the desktop they can call each other with no separate API wiring, which is the lowest barrier for someone doing document work.

My work is mainly document analysis and reasoning, qualitative report research and cross-domain integration. It is not about who operates faster, and it is not programming. On that line, the strongest model is of course useful, but the price is high too; for my important decisions now, I run these two.

5. Whether to copy someone else's skill package: workflow conflict first, then the source

There are plenty of ready-made skill packages online now, including the kind that lets two AIs call each other. Grabbing one is the fastest route, but two things need looking at first.

This section is about the order of judgement. The three-layer source check, the four things that actually matter about permissions, and how to audit the packages you already installed have a full treatment in Will Someone Else's Skill Package Mess Up My AI?: Three Things I Do Before Copying One.

More common than malicious code is workflow conflict

The other person usually has no bad intent. Their way of working is simply very different from yours. Add their approach in wholesale and your AI gets confused: your original workflow wants to go east, this skill package says go west, so which way?

So I ask one question first: does its mechanism and workflow conflict with what I have now? If it does, do I skip it entirely, take part as reference, integrate part of it, or take only its concept and keep my own workflow?

That question is usually worth more time than the security check, because conflicts do not raise errors. They just let your workflow drift slowly, and you only find out much later.

What a source check can stop, and what it cannot

My habit is to hand the article or the link to the AI first, ask it to check for malicious code and to look up whether the online claims are real, and only ask it to install once that is clear. That part I read out directly in class, and I added one line to it:

So what I do has three layers:

1. Look at who the source is. Someone with a name, a public record, other users, versus an anonymous link: very different risk
2. Then ask the AI to check, and look up the online discussion at the same time
3. Only once both layers pass does it get permission to act; for anything whose source is unclear, I would rather take the concept and rewrite it myself

Three things you hit during installation

1. Access: exactly enough, starting from read-only. If the package has been checked and you can name the files it needs, give it read and write on those; withdraw temporary permissions when the job is done.
2. Account authorisation: it will ask for authorisation from the other vendor. Be explicit here: I do not want to pay extra for API usage, I want to use my subscription account's quota.
3. Who installs it: this is fairly mechanical work, so hand it to whichever vendor is strongest at execution. There is no need to spend your main model on it.

One more thing: it can be installed without a ready-made package. I demonstrated it once on a friend's computer without giving it the skill package at all, only the promotional copy for my course, and the AI got it installed by itself. The only difference was a few more attempts. When a workflow is genuinely complex, my suggestion is to use someone else's as a reference and reshape it into your own.

6. "It's installed" is not the same as "it works"

Once it is done, the AI usually says: I have installed it.

That sentence also needs reviewing. I installed one for someone once, the AI said it was installed, and I pressed as usual: are you sure? How did you verify? Have you run it? Are there no problems?

Only then did it say: right, I only installed it, I have not actually tested it, it has not been run through.

So the acceptance standard has to come from you. You cannot let it define what counts as done. The standard has two layers:

First layer, whether the function itself produced a result. If you installed something that makes image cards, you should see a real image; if you installed a report checker, you should see a real check report. This layer cannot be skipped.

Second layer, if what you installed is the connection kind that "lets two AIs call each other", you also have to verify the connection itself. What I said at the time was this:

7. Why one model cannot review its own problems

Imagine a cooking contest where each chef is free to make whatever they themselves like to eat. The chef who loves spicy hotpot loves chilli, so of course they will make something spicy. And when it comes to judging, which dish will they think is best? The spicy one.

They are not trying to deceive anyone, and they are not covering for themselves. They just like it hot.

Models are the same. When it makes a decision it thought this was good, which is why it produced that decision; so when it reviews that decision, of course it still thinks it is good, because this is what it considers good. So having the same model self-review three or five times, in my view, does not do much.

The three depths of mutual review

The first was my earliest approach, and after a while I noticed a blind spot: what if the main model judged wrongly right at the beginning? Or what if there is a completely different route that also works well? Reviewing only the output means you never see either of those.

Running dual track has one crucial setting, simple to state and easy to forget: both sides use the same material, each works from scratch, and neither is allowed to peek at the other's answer. Look at the other side's result first and the second track has lost its independence, and what you have is one answer phrased two ways.

Roughly how the cost works

Fixed cost first: one monthly subscription with each of two vendors, and the desktop tools can call each other with no separate API wiring. That is how I use it.

Then variable cost. For the same job, reviewing the output only is roughly twice the work of a single pass, and a full dual-track mutual review is noticeably more again, because both sides have to read the whole thing from the start. Three-way mutual review I worked out and did not do: if every vendor has to review every other vendor, A reviews B then C, B reviews A then C, C reviews A and B, and then you run it all again, the workload multiplies several times over and the time is not there either. These figures are my own rough sense of the relative scale, not measurements.

The copy-ready prompts for each of the three levels, the three criteria for choosing, and what is left when you only have one vendor are in How Do You Design Mutual Review Between Two AIs?: Three Levels of Review Mechanism.

8. The four rules I write into a loop

Once the two can review each other, the next thing to decide is what they do when they hit a problem. Running loop engineering now, there are four things I always write in.

These four assume your material is clear enough. In my own experience, as long as past cases, my standards, my principles and the relevant examples are all written into that shared material, the two discussing it between themselves handles the large majority of situations. That is a judgement from experience, not a statistic.

What this rule solves is very practical: getting stuck and doing nothing is worse than finishing most of it and reporting. Coming back half an hour later to a half-drawn line, versus coming back to something ninety percent complete plus a note saying "stuck on step five", is a big difference in value.

The fourth I added on the spot that day. Halfway through I realised I had not covered it: you tell it to revise to a standard, so how many times? Twice, five times, ten, or a thousand? Without a ceiling, it will revise without limit, the bill climbs, or it drifts further off.

9. How many rounds before you stop

There is one more thing missed more easily than "should I run a mutual review": how many revisions before you stop.

If you only tell an AI "revise according to my standard" without an upper limit, it will keep revising until it thinks the result is good. That usually goes one of two ways: the bill keeps climbing, or the result drifts further off.

My own work is mainly document analysis and reasoning, and my rule of thumb is two to three rounds. One round here means:

• Round one: A reviews B, and B reviews A
• Round two: A revises once following B's suggestions, B revises once following A's, and then they review each other again

Anything not solved in two or three rounds usually will not be solved in ten or a hundred. That is normally something missing from the rule itself, and no amount of extra rounds fills it in.

How to tell it is time to stop: an extra ball on the desk

You give the AI a reference photo of a tidy desk plus four indicators: documents squared up, cup back in its fixed place, small items gathered together, seventy percent of the desk clear. It tidies one round, compares the result against the standard, and tidies again if it falls short.

But what if there is suddenly a ball on the desk, and the reference photo has no ball and says nothing about where a ball goes? It could run a hundred more times and still never fully match the reference. What it should do at that point is stop and report: everything else is in place, this ball has no rule, please add one.

When the other side is unreachable: the four fields a handoff needs

I have one desktop machine and two laptops, the accounts log each other out, and the mutual review drops out often. My rule used to say "a mutual review is mandatory", and the result was that one disconnection froze the whole flow, and what I came back to was half-finished.

My rule now says: if the other side does not respond, the main track finishes first, then tells me the other vendor did not review it, and produces a handoff instruction I can copy in one click and paste to the other vendor. That handoff has to be clear about four things:

1. What this task is doing
2. What it is being asked to review
3. What it must not touch (the key point: this is going over to be reviewed, not continued)
4. Where the result should be pasted back

The main track finishing first means most of what you get is usable, which beats the whole line stopping there. That is my own sense of it, not a measured proportion.

10. A real case from the room: 150 pages, two models, different results

A participant on the day raised a very good case. He had made a 150-page training deck at work, first using one model to check content, structure and chapter order repeatedly, revising it more than ten times; then he had another model review it once, and it turned up a standards error in the content that the first model had missed even with deep thinking on. He later traced it to the difference between two international standards.

He asked how I saw it. My first judgement was not about model capability. It was about the material.

The three things below are the answer I gave on the day. How each of the three layers of material, slicing and perspective is done, including how to write a rule that pins the standard and version, is in The AI Revised It Ten Times and It Is Still Wrong. Now What?: Cleaning the Data Is Only the First Layer.

First thing: PPT and PDF have to be cleaned first

For serious standards and serious content, I always suggest converting to a plain-text Markdown file first, pulling the chapter structure and a content index out together, and then handing it to the AI. Of the cases I have met where documents produce hallucination, the large majority saw enormous improvement after data cleaning. That is from the cases I have encountered, not a statistic.

Second thing: chunk it, and not with the same slicing

• Do not analyse ten chapters in one go. One analysis per chapter, then one big consolidation at the end
• Switch to a different slicing and run again: first have it group the ten chapters into five themes, then run once per theme
• Two slicings covering each other is what reaches the blind spots a single slicing leaves
• Only then cross them: use the chapter slicing to have both vendors run one round each, then cross the two sets of results

Uncleaned material also makes it very easy to blow past the context limit, and chunking is partly about that too.

Third thing: discuss the mechanism itself with the AI

This is the step I think most people miss. Rather than rerunning endlessly, discuss why it went wrong directly:

Problems usually come from one of three places: the instruction was not worded well, the mechanism was not set up, or it genuinely exceeds the model's current ceiling. All three are handled completely differently, so work out which one first.

Last step: from skill package to loop

That participant then said: what is missing now is any way to let it run itself. That was precisely put, and the answer is usually simple too: it does not know what to do when something is wrong, and does not know how to correct it. Write those two things in and it will run.

11. The order: review first, then mutual review, then the loop

Laid out in order, the whole thing is only three parts:

These three parts have an order. Skipping straight to automation before the review mechanism is sound means handing over a standard you cannot articulate yourself, and no number of rounds will improve it.

So when have you reached part three? There is a simple test. When you find this standard has run once or twice with no problems, you can tell it:

If it can, that is loop engineering. When it cannot, it is usually because it does not know what to do when something goes wrong. So go back to part one and add "how to notice it went wrong, how to fix it, when to stop".

How you can start

None of the three steps needs programming. What you need is to state clearly the check you do every single time, and that is something only you can do. The AI cannot do it for you.

Which one to read next

This piece is the overview. Each of four sections has its own full treatment:

• The AI Revised It Ten Times and It Is Still Wrong. Now What?: Cleaning the Data Is Only the First Layer: the 150-page case from section ten, and the complete method for material, slicing and perspective.
• How Do You Design Mutual Review Between Two AIs?: Three Levels of Review Mechanism: copy-ready prompts for the three levels in section seven, the criteria for choosing, and the fallback when the other side drops out.
• Will Someone Else's Skill Package Mess Up My AI?: Three Things I Do Before Copying One: the full version of section five, including the four permission items and how to audit what you already installed.
• Can I Get the Computer at Home Started While I Am Out?: One QR Code Scan and the Desktop Begins: what the publishing loop in section three actually looks like, including where it did not work.