AI Makes Technical Execution Cheap. Judgment Stays Scarce.
Hiring more developers or replacing them with AI agents leaves the same question unanswered: who will make the decisi...
What the Last 20% of an AI-Built SaaS Looks Like
10 min read
The Screens Work. The System Has Questions.
17.10.2026, By Stephan Schwab
An AI-built application can look almost finished while the difficult questions have barely begun. Building Nilo, our SaaS, with AI assistance keeps making that gap concrete. The screens work. Then two releases disagree about an event, a renewal forgets the customer's chosen plan, or an optional database failure takes down a useful page. The remaining work hides in interactions that a demo never exercises. Seeing those interactions, and questioning plausible fixes, takes more than knowing how to ask for another feature.
In When Vibe Coding Meets a Real Developer, I described applications people had built with AI before asking an experienced developer to look at them. They had got impressively far. Their applications also contained problems they had no reason to recognize.
Nilo gives me the view from the other side. We use AI to build it too. This is 2026. Having a developer laboriously type everything would prove very little beyond a preference for typing.
The interesting evidence sits in my conversations with Codex: the questions after the implementation, the objections after the explanation, the apparently small changes that reach across the product.
The “last 20%” is shorthand, not a measurement of Nilo’s progress. It describes the work people discover after the application already looks convincing. There is no promise that it takes 20% of the effort. Percentages are wonderfully comforting when nobody has measured the denominator.
Blue/Green, Until the Two Colors Talk to Each Other
A healthy new release can already be breaking the old one before it receives a single customer request.
On 3 October, more errors appeared after a deployment. I asked Codex to investigate the centralized logs, where messages from the running application instances come together.
We already had blue/green deployment. In the familiar explanation, one version serves customers while a second starts alongside it. Check the candidate, switch traffic, retire the old version. Very tidy. Two colors, one arrow, an excellent slide.
The investigation found that the new Nilo release published a new kind of internal event, tasks.board. The old release did not recognize that event type. Both versions shared the event infrastructure, so the old instances received the new events and rejected them.
Both versions could still pass their health checks. The new version had started successfully. The old version still answered requests. Those facts did not establish that they could safely run together.
The errors stopped when the old instances shut down. Codex initially presented that as reassuring: the burst had ended, and the current instances were working.
I asked:
That isn’t how blue/green deployment is supposed to work - is it?
That question changed the conclusion. The disappearance of the errors proved that the overlap had ended. It did not prove that the handover was correct.
Our implementation controlled incoming web traffic. It did not adequately control the background activity connecting the two versions. Starting the candidate also started work that could affect the running system.
Someone looking only at the website might never see this. A developer focused on the traffic switch might also miss it. The failure crossed a boundary between deployment, background processing, and application events.
“Finished” Has to Include the Work You Cannot See
Switching web traffic does not settle work already underway. An interrupted email operation can still send a message the application has not recorded.
I then asked how to improve the handover without making frequent deployments and fast development painful.
That brought us to draining: stop starting new work, let work already underway reach a known outcome, and then retire the process.
For a simple website, counting active requests might get you much of the way there. Nilo also has scheduled work, mailbox activity, asynchronous operations, and live connections. A request can finish while the work it initiated continues elsewhere.
Consider an operation that sends an email. The external mail service may accept it just before the application stops, while the application has not yet recorded the result. Restarting and trying again could send a duplicate. Killing the thread tells you nothing about whether the recipient will receive the message.
The incompatible event was the observed failure. The email is a failure scenario the handover must address: establish whether an interrupted operation succeeded before retrying it. Traffic routing alone cannot answer that.
The candidate needs to start without prematurely doing production work. The old process needs to stop admitting work and finish what it has already accepted. Event consumers and live connections need an orderly end. Timeouts need an explicit consequence.
A fixed delay cannot establish those facts. Waiting a minute proves that a minute passed. The application may have finished in two seconds, or still be waiting on an external service.
The blue/green label does not make incompatible versions safe to run together. You need compatible events during overlap or a controlled handover that prevents the two versions from processing them together. A short pause in writes may be the honest price of keeping internal contracts easy to evolve.
Those are product and operational decisions hiding inside something that looked like a deployment script.
The Fix Can Be Too Clever as Well
The first plausible fix deserves the same scrutiny as the original implementation.
Codex proposed explicit release states, work permits, and a deployment-generation record in Mongo. The explanation was coherent. It also pushed deployment orchestration into the application.
I asked:
So the application becomes aware of the deployment process? That doesn’t sound right
Then:
This is not a new thing. I would assume in the Spring ecosystem this has already been solved
The investigation turned toward Spring’s existing lifecycle facilities, in the framework Nilo uses. The useful boundary became clearer: deployment infrastructure coordinates releases and routing; application components participate in orderly startup and shutdown.
That still requires integration. A background listener that starts automatically must participate in the lifecycle. Existing framework support does not wire itself into every custom component. But examining that support first changes how much machinery we need to invent, and where it belongs.
As of 3 October, the work was still underway. A sounder direction still needed implementation and production verification.
This exchange matters because the AI was useful throughout. It traced the failure and developed concrete proposals. It also gave an initially reassuring interpretation and then a solution whose architectural consequences needed challenging.
The work included evaluating the explanation, not merely accepting the patch.
A Subscription Is a Promise That Survives Next Month
A successful payment proves the money moved. It does not prove the next renewal will preserve the plan the customer chose.
An earlier conversation began with making some credits non-expiring. I even called it “probably a very small change.” Famous words.
Nilo has paid credits, operator grants, and recurring services that consume credits. A trial customer may receive a grant and later become a paying customer. The displayed balance alone cannot explain what should expire, what an operator may revoke, or what a renewal must provide.
That required distinguishing the sources and recording grants and their use. Then came another question:
Is there anything that avoids running totally low on credits? We have recurring charges
And the business rule:
It should not be possible to choose a too small subscription while recurring charge products are active
Disabling a button helps the person using the billing page. The rule also has to hold in the service that changes the subscription, before it changes anything at the payment provider.
The investigation exposed a related renewal problem: the customer’s chosen tier needed to be stored and used at renewal, instead of falling back to the lowest tier.
A signup demonstration does not exercise next month’s renewal. A payment can succeed while the product fails to preserve the commitment the customer thought they bought.
The pricing page was visible. The promise spread across the credit model, subscription changes, payment notifications, and renewal processing. That is where the remaining work lived.
Making an Error Disappear Can Be the Wrong Fix
Keep independent functions working when optional data fails. Do not erase the error by rewriting records or weakening access boundaries.
One contact-page problem came from unavailable personal storage. Nilo keeps personal contact-related data separately from company-owned contacts. The company contacts could still be available even when the personal database was not.
During the investigation, Codex took a repair direction that I rejected. The development database contained mixed data and inconsistent references. The required behavior was to contain the unavailable dependency, not rewrite other stored records to make the exception disappear.
My correction included:
But NEVER alter the Mongo database
The required behavior was straightforward from the customer’s perspective: display the company contacts, explain that personal information is unavailable, and keep independent functions working.
Getting there required preserving the storage boundary, handling the specific failure where the page consumed optional data, and checking the import and detail pages too. Fixing one visible page did not cover every consumer.
A missing membership must not become an excuse to invent access to another database. Equally, unavailable optional data should not prevent the customer from using unrelated company data.
The instinct to “repair” things can be actively harmful when the system is correctly refusing access. Experience helps distinguish a broken reference, an unavailable dependency, and a boundary that must remain intact. They can all arrive looking like the same red error message.
A Visit Does Not Tell You Who Visited
A tracked visit establishes that someone or something opened the link. Recording it as human interest makes the CRM more confident than the evidence allows.
Campaign tracking produced another deceptively simple feature: send a personalized link and show activity on the contact’s timeline.
I asked what happens when an email security scanner follows the link to check the page for malware. If we record that as the human recipient having read it, our CRM tells a story the evidence does not support.
Requiring a deliberate action before recording anything would avoid part of that ambiguity, but defeat the purpose of observing visits. I objected to that too.
We changed the model to retain separate observations: a visit, then reported scrolling, clicking, or other activity. Later signals can add context without rewriting the first observation into a stronger claim.
They remain observations. Even a reported click does not give us magical certainty about a person’s intentions.
Then the contact timeline remained empty. I asked Codex to trace what should have arrived.
The investigation compared production logs, the documented API, and the code actually running on the external page. That caller used a different endpoint and omitted parts of Nilo’s required request and authentication. Nilo could have a working API and the other system could have tracking code, while the integration between them remained incomplete.
A page saying it sends events is weak evidence. A log mentioning the campaign domain is weak evidence too; it may describe the outgoing email. The meaningful proof is the request reaching the right service, passing the intended checks, producing a stored event, and appearing for the right contact.
“Both sides implemented it” leaves quite a lot of room between the sides.
Experience Shows Up in the Second Question
Much of the remaining work begins with asking what a successful result actually proves.
None of these examples required abandoning AI. We used it to investigate and change the system. The speed is real.
So is the need to question the result. The deployment finished; did incompatible versions do work together? The payment succeeded; will renewal preserve the chosen plan? The page loads; did the fix quietly alter data it had no business touching? An event appears; what does it establish about a human?
A non-developer has little reason to anticipate these questions. A capable developer with experience concentrated in one layer may miss them too. Years spent on screens do not automatically teach payment lifecycles. Knowing container deployment does not automatically explain the asynchronous work inside the container.
Breadth matters because the failures travel. Recognizing a familiar smell gives you a place to investigate; it does not exempt you from proving the diagnosis.
The questions can become executable checks: the chosen tier survives renewal; an unsuitable subscription causes no payment-provider change; unavailable personal storage leaves company contacts intact; a candidate release performs no production work before the handover permits it.
Then the next AI-assisted change has something concrete to satisfy. Tests give those decisions a memory that runs. Deployment checks and production evidence cover what isolated tests cannot.
That is what the “last 20%” looks like. Some of it is missing code. Much of it is finding a promise the visible feature does not yet keep, then following that promise across the whole system.
The interface may barely change. The customer gets a service that continues to work through renewal, failure, and the next release. Quite a lot of product for something the demo could not show.
Talk It Through
Tell me what is happening. I listen, ask a few practical questions, and reflect back what I see: where the risk may sit, what may be blocking delivery, and what looks worth checking next. No pitch, no obligation. Confidential and direct.
Talk it through. Practical reflection, no pitch.
Start a Conversation
🎭 This Week's Episodes
Telenovelas show what we can't say in client meetings. The drama is heightened, but the patterns are real.
Documentation Avalanche
The forty-seven questions from Passau become a daily occupation. Stefan finds a gap the navigation team cannot investigate without Marcus's approva...
Technical Consultancy
Embedded Delivery Partner
Embedded into your team as an active contributor, reducing delivery friction and helping important work move cleanly.
Technical Advisor
Peer-style technical assessments before far-reaching decisions; reduce architectural and product risk early.
Product & Delivery
Ship working software to real users earlier. Measure impact and adapt based on evidence rather than assumption.
Custom Software Development
High quality, maintainable software. Short-term augmentation that leaves lasting capability in your own team.
Recent Articles
- What the Last 20% of an AI-Built SaaS Looks Like (17.10.2026)
- AI Computer Use Is Not a Business Contract (09.10.2026)
- When Everything Is Finally Under Control (05.10.2026)
- AI Makes Technical Execution Cheap. Judgment Stays Scarce. (26.09.2026)
- The Agent Escaped. Who Left the Door Open? (24.09.2026)