San Francisco’s AI industry is moving toward a new kind of software. Instead of simply answering questions or following instructions, AI agents are being built to recognize when something goes wrong, examine what happened, and use that information to perform better the next time.
This could change how businesses use software. A customer service agent could learn which responses lead to successful resolutions. A coding agent could remember why a previous fix failed. A sales agent could identify mistakes in its research process and avoid repeating them across hundreds of future tasks.
The important shift is not that AI is suddenly becoming perfect. It is that developers can now build systems where mistakes become useful information rather than events that disappear when a task ends.
For San Francisco startups, this creates opportunities far beyond building more powerful chatbots. Companies that help AI agents remember useful experiences, test their decisions, improve workflows, and prove that their performance is getting better could become an important part of the next generation of business software.
But there is a problem that founders cannot ignore. An AI agent that changes its behavior is not necessarily an AI agent that improves. If the feedback is wrong, the memory is poor, or the tests are too easy, an agent may become more confident while making the same mistakes.
This article examines how self-improving AI agents actually work, what published research shows about their potential, where San Francisco companies are building the technology, and how businesses can use it safely. It also includes original SFTechScene calculations based on public research and clearly defined models to identify where learning from mistakes could create the greatest business value.
The Short Version: AI Agents Are Moving From Completing Tasks to Getting Better at Them
The first generation of business AI assistants focused on producing useful answers. The next generation of AI agents began taking actions, such as searching company records, updating software, writing code, and completing business tasks.
Self-improving agents add another layer. They can record the outcome of an action, examine feedback, identify a possible mistake, and apply an adjustment to future work.
Consider an AI agent that processes customer refunds. In a simple system, the agent reads a request, checks the order, and decides what should happen. If it incorrectly recommends a refund, a human employee fixes the problem, but the agent may make the same mistake tomorrow.
In a learning system, the correction can become a test case. Developers can identify why the mistake happened, improve the agent’s instructions or tools, and check whether the new version handles similar requests correctly.
The change sounds small, but its business impact could be significant. Companies would no longer judge AI only by how well it performs on launch day. They could measure whether the software becomes more reliable after hundreds or thousands of real tasks.
Three findings help explain why this matters.
First, published research already shows that agents can improve performance by using feedback and stored experiences without changing the underlying AI model. The 2023 Reflexion study reported a coding benchmark result of 91%, compared with 80% for a GPT-4 reference result in the study.
NeurIPS Proceedings
+1
Second, learning can take the form of reusable skills. The Voyager research project demonstrated how an agent could build a library of successful actions, reuse those skills, and develop more complex abilities over time.
ML Anthology
+1
Third, improvement must be tested. Research and engineering guidance from Anthropic and OpenAI emphasize structured evaluations, recorded agent actions, and repeatable testing as essential parts of building reliable agent systems.
Anthropic
+1
For San Francisco startups, the opportunity is not limited to creating agents that learn. It also includes building the tools that decide what an agent should remember, which mistakes matter, whether a proposed improvement works, and when a new behavior is safe to deploy.
What Is a Self-Improving AI Agent?
A self-improving AI agent is a software system that uses information from earlier attempts to improve how it performs future tasks.
The agent might change its approach while completing a single task. It might save a useful lesson for the next task. Or its developers might use recorded outcomes to improve the agent’s instructions, tools, and model.
These methods are related, but they are not the same.
The Difference Between an AI Agent and a Self-Improving AI Agent
A traditional AI agent receives a goal, chooses actions, and works toward completing that goal. It may use company software, search for information, or ask another AI system for help.
A self-improving agent adds a feedback process. Instead of treating every task as an isolated event, the system uses earlier results to help guide later decisions.
| Capability | Standard AI agent | Self-improving AI system |
|---|---|---|
| Completes assigned tasks | Yes | Yes |
| Uses business tools | Often | Often |
| Detects some errors | Possible | Designed to capture errors |
| Saves useful lessons | Not necessarily | Often |
| Tests improved approaches | Optional | Core requirement |
| Changes future behavior | Limited or temporary | Through memory, updates, or training |
| Proves improvement over time | Not necessarily | Should measure it |
The key point is that an agent does not need to retrain its AI model to improve its performance.
A customer support agent might become more accurate simply because it has access to a better policy document. A coding agent might perform better because it remembers a tested solution. A scheduling agent might avoid repeated mistakes after developers update its instructions.
These are all possible forms of system-level improvement, even when the model’s underlying weights remain unchanged.
The Five Stages of Learning From Mistakes
A practical self-improvement system needs a complete feedback cycle.
How the improvement loop works
1. Attempt the taskAgent performs a business action
2. Inspect the resultCheck what succeeded or failed
3. Identify a possible lessonFind the cause and propose a change
4. Test the changeCompare old and new behavior on fixed tests
5. Preserve or rejectKeep proven improvements and discard failures
Repeat with fresh tasks and feedback
The last two stages are especially important. Without testing and a clear decision about whether to keep a change, an agent is merely experimenting with its behavior.
Businesses need controlled improvement rather than uncontrolled change.
Why San Francisco Is Becoming an Important Center for Self-Improving AI Agents
San Francisco has a strong position in agent development because several important parts of the technology are being built within its broader AI ecosystem.
The region has model developers, coding-agent companies, enterprise software teams, and businesses working on AI evaluation and monitoring. Together, they create an environment where researchers can test new learning methods and startups can turn those methods into products.
However, it would be premature to claim that every San Francisco agent startup has developed genuinely autonomous learning. Many products currently focus on monitoring behavior, collecting feedback, and helping developers improve software.
That distinction matters because the commercial opportunity is broader than the phrase self-improving AI suggests.
OpenAI: Building the Tools to Measure and Improve Agents
San Francisco-based OpenAI provides tools for building agents and evaluating how they behave.
Its agent evaluation systems support recorded execution traces, datasets, grading, and prompt optimization. Developers can examine which actions an agent selected, where it failed, and whether a change improved results across repeated tests.
OpenAI API
+1
This is important because a company cannot build a trustworthy improvement process without recording what happened.
Suppose an AI support agent incorrectly promises a refund. The final message alone may not explain the problem. Developers need to know whether the agent used an outdated policy, selected the wrong customer account, misunderstood a tool result, or simply made an unsupported decision.
Once the cause becomes visible, teams can create an evaluation case and check proposed fixes.
Anthropic: Treating Feedback as a Core Part of Agent Design
San Francisco-based Anthropic has published engineering guidance describing an evaluator-optimizer pattern.
In this approach, one AI system produces a result while another evaluates it and provides feedback. The producing system can then revise its work when the task has clear quality standards and additional attempts are worthwhile.
Anthropic
This pattern is useful for coding, research, document processing, and other work where mistakes can be identified before the final result reaches a customer.
Anthropic’s January 2026 guidance also stresses that evaluation must reflect the full complexity of agent behavior. Checking whether an answer sounds convincing is not enough when the system can make tool calls, change records, and take actions across several steps.
Anthropic
Arize AI: Making Agent Mistakes Easier to Understand
Arize AI is part of the San Francisco AI infrastructure ecosystem and focuses on AI monitoring and evaluation.
Its Phoenix platform lets developers trace agent activity, inspect outputs, run evaluations, build datasets, and compare changes. The company describes a workflow in which teams observe failures, label outcomes, form an explanation, test a proposed fix, and measure the results.
Phoenix
+1
That provides a practical foundation for systems that improve through repeated testing.
Arize should not be confused with an agent that automatically learns every lesson by itself. Its role is to provide infrastructure that makes measured improvement possible.
Why the Supporting Tools May Become a Major Startup Opportunity
Agent companies face a difficult problem as they grow. The number of possible failures increases when agents interact with more applications, customer records, payment systems, and external websites.
A developer may be able to inspect ten failures manually. Reviewing thousands of agent tasks every day is much harder.
This creates room for specialized businesses that identify repeating mistakes, organize feedback, generate useful test cases, and help teams determine which changes are safe.
The opportunity is similar to the growth of tools for monitoring traditional software, except AI agents introduce a new challenge: the same request can produce different actions and results.
Original SFTechScene Research #1: How Much Can Learning From Mistakes Actually Improve AI Performance?
To assess whether self-improvement is more than a marketing claim, SFTechScene examined published benchmark results from two research projects: Reflexion and Voyager.
The purpose is to separate improvements demonstrated under experimental conditions from predictions about what businesses might achieve.
Research Methodology
We used publicly reported outcomes from the peer-reviewed 2023 Reflexion research and the 2024 journal version of Voyager.
For Reflexion, we calculated the absolute improvement in benchmark accuracy, the relative increase in successful results, and the relative reduction in failed results.
For Voyager, we compared the published performance ratios against prior systems. Because those results cover different measures, we did not combine them into one overall improvement score.
We also avoided treating research benchmarks as enterprise customer performance. The studies use different agents, tasks, environments, and evaluation procedures, so they cannot support a single industry-wide estimate of how much self-improving AI improves business productivity.
Finding #1: Reflexion Reported an 11-Percentage-Point Coding Improvement
The Reflexion paper reported a 91% result on HumanEval, compared with an 80% GPT-4 reference result.
Published HumanEval benchmark results
Results reported by the Reflexion researchers; percentage of problems passed.
Source: Shinn et al., Reflexion, NeurIPS 2023. The reported comparison is specific to the study’s setup, not a controlled estimate for all agents.
NeurIPS Proceedings
The absolute change is 11 percentage points. That means 11 more successful outcomes per 100 problems when using the reported rates.
The relative increase in success rate is 13.75%, calculated by dividing the 11-point increase by the original 80% rate.
More interestingly, the implied failure rate moves from 20% to 9%.
Failure rates implied by reported success rates
SFTechScene calculations from the Reflexion paper.
That represents a 55% relative reduction in failure rate.
This is an original calculation from the published figures, not an additional result measured by the researchers.
The commercial implication is that changes that appear modest when expressed as accuracy gains can be much larger when expressed as reductions in failed work.
For an AI system already completing most tasks correctly, eliminating a portion of the remaining mistakes may create more value than simply making responses faster.
However, the Reflexion study does not establish that a business agent will achieve the same gain. Its result depends on the benchmark, the feedback available, and how the system uses that feedback.
Finding #2: Reusable Skills Can Produce Gains Across Different Measures
Voyager provides a different kind of evidence.
Instead of focusing on coding benchmark accuracy, the researchers developed an agent that explored Minecraft, learned useful actions, and stored executable skills for later reuse.
The journal version reported that Voyager obtained 3.1 times as many unique items, traveled 2.3 times farther, and reached selected technical milestones up to 15.3 times faster than the previous systems used for comparison.
ML Anthology
Voyager’s reported performance ratios
Multiples of prior best systems; higher is better for the first two metrics and denotes faster completion for the third.
Source: Wang et al., Voyager, Transactions on Machine Learning Research, 2024. Each ratio measures a different outcome; they should not be averaged.
These findings suggest that a stored library of successful actions can improve an agent’s ability to complete future tasks.
For businesses, the parallel is straightforward.
A finance agent might save a reliable method for handling a particular invoice format. A recruiting agent might reuse a successful process for checking interview availability. A coding agent might preserve tested steps for fixing a recurring integration problem.
The skill library reduces the need to solve every problem from the beginning.
Finding #3: Two Different Studies Point Toward Two Different Business Products
Our comparison suggests that companies should distinguish between learning from unsuccessful attempts and preserving successful methods.
| Research approach | Main learning method | Observed benefit | Possible business product |
|---|---|---|---|
| Reflexion | Feedback and written reflections | Higher reported coding benchmark accuracy | Error analysis and correction |
| Voyager | Reusable stored skills | Better performance on exploration and task milestones | Agent memory and skill libraries |
| Combined approach | Feedback plus tested reusable skills | Not independently measured in this analysis | Controlled agent improvement platform |
The combined approach is a product hypothesis, not a reported finding from these studies.
A startup that builds a shared memory system without reliable evaluation may preserve bad decisions. A startup that evaluates every mistake but never saves reusable solutions may spend too much money rediscovering the same answer.
The strongest design is likely to connect both processes while requiring evidence before changes are promoted into general use.
How Self-Improving AI Agents Actually Learn
There is no single method that makes software self-improving.
Different businesses need different learning systems depending on the complexity of their work, the quality of available feedback, and the risk of making a mistake.
Learning Through Reflection
Reflection is one of the simplest methods.
An agent attempts a task, receives feedback, and creates a short record describing why the attempt succeeded or failed. It can use that record during a later attempt.
Suppose a coding agent changes a payment application and the tests fail. The agent examines the error and discovers that it updated a function without changing a related test.
On the next attempt, it uses that observation to check the function and its tests together.
The benefit is that the system can change its behavior without needing to update the underlying model. The limitation is that a written explanation can be inaccurate, so it needs validation.
Learning Through Memory
Memory lets an agent retain information that may matter beyond its current task.
A useful business memory system might store approved procedures, earlier error patterns, customer preferences, successful tool sequences, or lessons from reviewed work.
However, saving everything is a poor strategy.
An agent that stores thousands of irrelevant notes may retrieve the wrong information or spend more time processing memories than completing tasks.
Strong memory systems need rules for relevance, freshness, accuracy, and permission. They also need ways to remove outdated information.
Learning Through Repeated Testing
Repeated testing lets developers compare different agent versions on the same work.
For example, a startup could create 200 representative support cases and measure whether a revised agent correctly resolves each case.
When the team changes the instructions, it reruns the test cases. The update should be accepted only if the overall results improve without introducing unacceptable failures.
This is more dependable than deciding that an agent seems smarter after a handful of successful demonstrations.
Learning Through Better Tools
Sometimes an agent’s problem is not its ability to reason. It simply lacks the right information or access.
An agent might repeatedly fail to answer customer delivery questions because the shipping data is unavailable.
Adding a reliable shipping lookup tool could eliminate the problem without changing the model or creating an elaborate memory system.
This is an important lesson for founders: Fixing the environment can be more valuable than making the agent think harder.
Learning Through Model Training
The most complex approach involves updating the model itself.
Companies can use reviewed examples, successful task records, and feedback to improve a model through further training. This may help when the required behavior cannot be achieved reliably through instructions, memory, or tools alone.
Training introduces additional demands around data quality, cost, privacy, and evaluation.
For many early-stage San Francisco startups, improving the workflow and its feedback systems will be a more practical first step than training a model from scratch.
Original SFTechScene Research #2: Why Small Error Rates Become a Major Problem in Multi-Step AI Agents
One of the most important challenges in autonomous software is that business tasks often involve several dependent actions.
A customer service agent might identify a customer, locate an order, check the purchase date, read the refund rules, calculate the eligible amount, and update a company system.
Even if the agent completes each action correctly most of the time, the chance of completing the entire process without mistakes can be much lower.
Our Research Question
How much does improving the reliability of individual actions improve the reliability of a complete business workflow?
To answer this, SFTechScene developed a mathematical model using three possible per-step success rates: 95%, 98%, and 99%.
We examined workflows containing five, ten, and twenty dependent steps.
These percentages are illustrative assumptions, not measured success rates for any particular commercial agent.
Methodology: Modeling the Chance of a Fully Correct Workflow
The model assumes that each step must succeed for the full task to be considered successful. It also assumes that failures are independent and every step has the same success probability.
Under those assumptions:
\[ P(\text{complete success})=p^n \]
Here, \(p\) represents the probability that one step succeeds, and \(n\) represents the number of required steps.
We calculated this probability for all nine combinations.
Finding #1: A 95%-Reliable Step Is Not Enough for a Long Workflow
| Required steps | 95% step success | 98% step success | 99% step success |
|---|---|---|---|
| 5 | 77.4% | 90.4% | 95.1% |
| 10 | 59.9% | 81.7% | 90.4% |
| 20 | 35.8% | 66.8% | 81.8% |
Modeled probability of completing every step correctly
SFTechScene calculation assuming independent steps of equal reliability. Not observed agent performance.
■ 95% per step
■ 98% per step
■ 99% per step
The result is striking. With a 95% success rate at each step, a twenty-step workflow has only a 35.8% chance of completing without a mistake under this model.
Increasing step reliability to 99% raises that figure to 81.8%.
That is an increase of approximately 46 percentage points in the chance of completing the entire workflow correctly.
This helps explain why agent developers should care about small, repeatable improvements. Improving a single action from 95% to 99% may sound modest, but across a long series of dependent actions, the modeled impact becomes much larger.
Finding #2: Workflow Design Can Matter as Much as Agent Intelligence
The same analysis also reveals an opportunity that receives less attention.
If a company can reduce the number of dependent steps, it may improve the chance of successful task completion without changing the AI model.
At 98% per-step reliability, reducing a process from twenty steps to ten increases modeled end-to-end success from 66.8% to 81.7%.
That is a gain of approximately 15 percentage points purely from shortening the workflow.
A San Francisco startup building business agents should therefore measure two things: how well each action works and whether every action is necessary.
For example, an agent might use six separate tools to gather customer information that could be returned through one approved application interface. Replacing those calls with a single reliable operation may reduce the number of opportunities for failure.
Finding #3: Self-Improvement Should Target Repeated Failure Points
Not every mistake deserves the same amount of engineering effort.
If an agent fails only when handling a rare formatting issue, fixing that issue may provide limited benefit. If it regularly selects the wrong customer record, the same mistake could affect thousands of tasks.
The best improvement strategy begins by identifying failures that are frequent, expensive, and preventable.
Teams can then prioritize changes based on expected business value rather than trying to improve every part of the system equally.
What This Model Cannot Tell Us
Real agent workflows do not always follow the independence assumption.
Some failures are correlated. An incorrect customer identity may cause several later actions to fail. Other systems can recover from mistakes before the final task is completed.
Steps also differ in their importance and their success probabilities.
Our analysis therefore describes a simplified system where every action must succeed. It is not a forecast of commercial AI failure rates.
Its purpose is to show why improving small actions, shortening workflows, and adding error recovery can have significant effects on complete-task reliability.
For real deployments, companies should replace the assumed percentages with observed data from their own workflows.
Why AI Agents Need Better Memory, Not Just More Memory
Memory has become an attractive feature in the AI industry because it promises that agents will remember earlier experiences.
But memory can also create problems.
Imagine an AI agent that reviews a company policy and incorrectly concludes that every customer can receive a full refund within sixty days.
If the agent saves this conclusion as a trusted rule, it could repeat the error across thousands of future conversations.
The learning system has made the original mistake more persistent.
The Difference Between Useful Memory and Dangerous Memory
A reliable memory system should distinguish between facts, observations, proposed lessons, and verified procedures.
A fact might be an approved refund policy that has a source and an effective date. An observation might be that a recent refund request failed because the order could not be located.
A proposed lesson might suggest searching by transaction identifier rather than customer name.
The lesson should not become an approved procedure until it has been tested or reviewed.
| Memory type | Example | Recommended control |
|---|---|---|
| Verified fact | Approved refund deadline | Source and expiration check |
| Task history | Previous customer request | Privacy and retention rules |
| Unverified reflection | Possible reason for a failed search | Mark as tentative |
| Tested skill | Approved order lookup sequence | Versioning and regression tests |
| User preference | Preferred report format | Permission and correction controls |
| Sensitive information | Personal or financial details | Strict access limits and minimization |
The commercial opportunity is not simply to build bigger memory databases.
It is to build memory systems that make useful information available at the right time without turning every past response into a permanent instruction.
Why Forgetting Is Sometimes an Improvement
Businesses change their policies, software, prices, and approval procedures.
An agent that remembers everything may continue applying rules that were correct six months ago but are wrong today.
A well-designed system should be able to retire an outdated procedure, replace a failed lesson, and recognize when old information no longer applies.
This means forgetting needs to be treated as part of learning.
For startups building persistent AI agents, the ability to correct and remove memory may become as important as the ability to store it.
Original SFTechScene Research #3: The Economics of a Self-Improving AI Agent
Higher accuracy is valuable, but founders also need to know whether improving accuracy creates enough financial benefit to justify the additional software, monitoring, and review costs.
To explore this question, SFTechScene developed a transparent business-cost model.
This is an illustrative economic analysis, not a survey or measurement of San Francisco companies. Every assumption is shown so businesses can replace the values with their own operational data.
Research Methodology and Assumptions
We modeled a business completing 10,000 AI-assisted tasks per month.
The original agent has a 90% task success rate. After introducing a learning and evaluation system, its success rate increases to 95%.
We assume the underlying agent costs $0.25 per task in both versions, while the improved system adds $0.15 per task for feedback processing, memory, evaluation, and related operations.
Every unsuccessful task requires eight minutes of employee review or rework. We use an illustrative fully loaded labor cost of $45 per hour, which makes the correction cost $6 per failed task.
These values are scenario inputs, not published industry averages.
| Input | Original system | Improved system |
|---|---|---|
| Monthly tasks | 10,000 | 10,000 |
| Successful tasks | 90% | 95% |
| Failed tasks | 1,000 | 500 |
| Base AI cost per task | $0.25 | $0.25 |
| Added improvement cost per task | $0.00 | $0.15 |
| Human cost per failed task | $6.00 | $6.00 |
Finding #1: Cutting Failures in Half Does Not Cut Total Costs in Half
Under our assumptions, the original agent produces 1,000 unsuccessful tasks per month.
At $6 per correction, those failures create $6,000 in monthly review and rework costs.
The improved agent produces 500 unsuccessful tasks, reducing those costs to $3,000.
However, the improvement system introduces an additional $1,500 in monthly operating costs.
Modeled monthly operating cost
10,000 tasks per month, excluding fixed implementation costs; USD.
■ Base AI cost
■ Learning and evaluation
■ Human rework
Original monthly cost
$8,500
Improved monthly cost
$7,000
Monthly savings
$1,500
Operating cost reduction
17.6%
In this modeled scenario, the company’s monthly operating costs fall from $8,500 to $7,000.
That represents savings of $1,500 per month, or approximately $18,000 per year if volumes and assumptions remain unchanged.
The important lesson is that a 50% reduction in failed tasks translates into a smaller reduction in overall operating cost because the improvement system itself costs money.
Finding #2: The Value Depends on the Cost of Fixing Mistakes
An agent that writes social media drafts may be inexpensive to correct.
An agent that creates faulty financial records or mishandles a customer account may require more expensive review and recovery.
To test how this changes the business case, we kept the 10,000-task volume, success rates, and AI expenses constant, then varied the human correction cost.
| Human correction cost per failure | Monthly savings from improved agent |
|---|---|
| $2 | −$500 |
| $4 | $500 |
| $6 | $1,500 |
| $10 | $3,500 |
| $20 | $8,500 |
Monthly value of reducing failures
SFTechScene scenario sensitivity analysis; negative values mean the improved system costs more.
The result points toward a clear business strategy.
Self-improving agents are likely to have a stronger economic case in workflows where errors are common enough to matter, expensive to repair, and measurable.
That does not mean startups should immediately target the highest-risk applications. It means they should identify work where better reliability produces enough value to justify the additional controls needed to achieve it.
Finding #3: Break-Even Improvement Can Be Calculated Before Building
For this model, the extra learning system costs $1,500 per month.
If each avoided failure saves $6, the company needs to prevent 250 failures per month to cover the extra operating expense.
With 10,000 tasks, that means increasing the task success rate by at least 2.5 percentage points.
The break-even target is therefore 92.5% success, compared with the original 90%.
This gives founders a much better question than asking whether they can build a self-improving agent.
They can ask whether they can achieve a reliable, measurable 2.5-point improvement at the assumed cost.
Interactive Model: Test the Business Case
The model below uses the same equations as the research scenario. Adjust the workload, accuracy, review cost, and improvement-system expense to see how the financial result changes.
Agent Improvement ROI Calculator
Illustrative modelMonthly tasks
10,000Original success rate
90%Improved success rate
95%Cost to correct a failed task
$6Added learning cost per task
$0.15
Original monthly cost
$8,500
Improved monthly cost
$7,000
Estimated monthly savings
$1,500
The assumed improvement pays for its added operating expense.
Assumes every failure requires one correction, the same task volume in both systems, and a $0.25 base AI cost per task. Excludes development, integration, fixed infrastructure, and compliance costs. No commercial outcomes are being claimed.
Where Self-Improving Agents Could Create the Most Value for San Francisco Startups
The economics point toward a few particularly promising markets.
The common feature is not simply that these industries use AI. It is that they generate enough repeated work to make improvement measurable.
AI Coding Agents: Learning From Tests and Failed Fixes
Software development is a natural environment for agent improvement because code can often be checked automatically.
An AI coding agent can write a change, run tests, inspect errors, and revise its approach. If a fix passes review, the system can preserve useful information about the solution.
The process works especially well when developers provide clear requirements, reliable tests, and controlled access to development environments.
However, passing tests does not prove that a change is secure, maintainable, or suitable for production.
A coding agent may accidentally learn to satisfy a narrow test while introducing a problem elsewhere. Startups need regression testing, code review, and checks for security-sensitive changes.
The opportunity extends beyond coding agents themselves. There is room for products that organize failed coding attempts, generate meaningful tests, and recommend proven repair methods across similar projects.
Customer Service Agents: Learning From Real Resolutions
Customer service creates large volumes of repeated questions.
Agents can encounter the same return policies, account problems, delivery issues, and billing disputes many times each week.
A useful learning system can compare AI recommendations with actual customer outcomes. It can identify cases where employees overruled the agent and investigate the reasons.
The best feedback is not simply whether the customer liked the answer.
A customer might be satisfied with an unauthorized refund, even though the business lost money. A reliable evaluation should check policy compliance, correct resolution, customer experience, and financial impact.
For companies building customer service agents in San Francisco, this creates a valuable product feature: the ability to show customers that verified resolution quality improves over time.
Sales Agents: Learning Which Work Leads to Better Outcomes
Sales teams create many opportunities for collecting feedback, but outcomes are harder to interpret.
An agent might draft an email, research a prospective customer, recommend an outreach time, or update a customer relationship management system.
Some actions are easy to verify. A company name is either correct or incorrect. A meeting time either matches the calendar or it does not.
Other outcomes are more complicated. An email receiving no reply does not necessarily mean it was poorly written. The buyer might have been unavailable or uninterested for reasons unrelated to the message.
Sales agents should therefore separate factual accuracy from commercial outcomes.
Startups that can identify repeated data-quality failures and improve research accuracy may provide more dependable value than systems that automatically change sales tactics after every unsuccessful campaign.
Finance and Accounting Agents: Learning Under Strict Controls
Finance operations have repeated workflows involving invoices, purchase orders, expense reports, and account records.
An agent can learn to recognize recurring document formats and common mismatches.
For example, a company may receive invoices from the same supplier every month. A learning system could preserve a verified method for extracting the relevant fields and matching them to purchase records.
But financial agents must operate within approval limits.
A correction to an invoice-reading rule should not automatically authorize a larger payment. The system can learn how to prepare information while human employees or approved controls retain responsibility for sensitive decisions.
This is a useful distinction between learning to complete work and gaining permission to take action.
Healthcare and Legal Operations: Improving Support Work First
San Francisco’s broader technology ecosystem includes companies developing AI for healthcare and legal services.
These sectors offer substantial opportunities for better document search, administrative work, information extraction, and workflow coordination.
They also carry higher risks when agents rely on incorrect information.
A legal research agent should not treat its own earlier answer as a verified legal source. A healthcare administration agent should not turn an unreviewed guess into a permanent patient record.
Startups should begin with supporting activities where results can be checked against authoritative documents and approved procedures.
For sensitive decisions, self-improvement should mean better evidence gathering, more accurate extraction, and fewer administrative errors—not uncontrolled changes to professional judgment.
Original SFTechScene Research #4: Which Types of AI Agent Work Should Start Learning First?
The previous calculations establish an important point: agent improvement has value when it prevents costly, repeated mistakes, but stronger controls may be necessary when those mistakes can harm people or businesses.
We used that finding to create a practical workflow selection framework for San Francisco founders.
This is an original editorial decision model, not a statistical ranking of existing AI companies.
Research Methodology
We evaluated six workflow categories using four criteria: how frequently the work repeats, how clearly success can be measured, how easily mistakes can be corrected, and how serious an incorrect action could become.
For each category, we applied a qualitative rating of high, medium, or low based on the nature of the task. These are analytical judgments, not observations from a representative business dataset.
The framework helps founders select pilots. It does not claim that a particular category delivers a higher measured return on investment.
| Workflow | Repetition | Ease of testing | Error risk | Pilot suitability |
|---|---|---|---|---|
| Code generation with tests | High | High | Medium | Strong |
| Customer support triage | High | High | Medium | Strong |
| Internal knowledge search | High | Medium–high | Low–medium | Strong |
| Sales research | High | Medium | Medium | Promising |
| Invoice preparation | High | High | High if payments execute | Promising with approval |
| Clinical or legal decisions | Variable | Difficult | High | Limited to supervised support |
Finding #1: The Best Early Workflows Have Clear Answers
Coding tests, document matching, customer account lookups, and internal information searches all have something useful in common.
They provide ways to check whether the agent’s result is correct.
An AI system that can compare its output against a reliable answer has a stronger foundation for improvement than one relying entirely on vague user reactions.
This makes narrowly defined operational tasks attractive starting points for founders.
Finding #2: Human Corrections Can Become a Valuable Source of Data
An employee correcting an AI mistake provides information that may be useful in future tasks.
However, that information becomes valuable only when the correction is recorded in a structured and privacy-conscious way.
The system should preserve what failed, what the correct outcome was, why the correction was accepted, and which similar cases should be tested.
A product that transforms routine corrections into high-quality evaluation cases could provide value even without creating an entirely new AI model.
Finding #3: The Most Valuable Learning May Happen Between Deployments
Many people imagine an agent that automatically becomes smarter during every customer conversation.
That is not necessarily the best commercial design.
For high-volume business software, a safer approach may be to collect feedback continuously but improve the production system through scheduled, tested updates.
An agent can operate with stable instructions while the development team evaluates proposed improvements separately.
Once those changes pass the required tests, a new version can be released.
This approach offers the benefit of learning from real work without allowing every customer interaction to rewrite production behavior.
How San Francisco Startups Can Build Their First Self-Improving AI Agent in 90 Days
A startup does not need a large research team to begin developing a system that learns from mistakes.
It needs a clearly defined business task, reliable feedback, a way to inspect what happened, and a controlled process for testing changes.
The following plan is designed for a small team building a first production-oriented pilot.
Days 1–15: Choose One Workflow and Establish the Baseline
Start with a task that occurs frequently and has a clear definition of success.
Good candidates include checking incoming support tickets, organizing sales research, extracting invoice details, or preparing internal reports.
Avoid beginning with a broad instruction such as managing an entire customer relationship. That makes it difficult to identify whether the agent actually completed its work correctly.
During the first two weeks, document the task from beginning to end.
Record the input, expected output, allowed tools, required approvals, and the conditions under which an employee must take over.
Create an initial test dataset containing representative examples. For a small pilot, 100–200 carefully reviewed cases may provide a useful starting point, although that number is not sufficient to prove every safety or reliability claim.
Separate common cases from rare and difficult cases so the evaluation does not reward the agent merely for handling easy work.
Days 16–30: Record Every Meaningful Action
The next step is visibility.
Developers should be able to inspect which information the agent accessed, which tools it called, whether those calls succeeded, and why the final result was accepted or rejected.
Use tracing and evaluation tools when practical rather than building an entire monitoring system from scratch.
OpenAI’s agent evaluation guidance and Arize Phoenix documentation both describe methods for collecting execution records and evaluating model behavior.
OpenAI API
+1
For each task, record the agent version, task type, final outcome, tool errors, review time, and any employee correction.
Sensitive customer information should be protected, and logs should contain only what is needed for legitimate evaluation and debugging.
By day thirty, the team should know its most common error categories.
Days 31–45: Build a Structured Feedback System
Do not ask the agent to remember every mistake.
Instead, build a process that converts selected failures into structured records.
A useful record includes the original task, the observed failure, the verified correct outcome, the likely cause, and the change proposed to prevent recurrence.
For example, if the agent repeatedly selects the wrong customer account, the improvement might involve a stricter account matching rule.
If an agent cannot answer a question because it lacks current information, the fix may involve updating its search tool rather than changing the prompt.
The aim is to identify causes rather than collect long explanations that cannot be tested.
Days 46–60: Test Memory and Improvement Methods
Introduce one improvement method at a time.
The team might begin with a verified library of common solutions. Another experiment might introduce written reflections after failed tasks. A third might improve the agent’s tool selection rules.
Run each candidate against the same held-out test cases.
Compare it against the original system using complete-task accuracy, error severity, review time, cost, and response speed.
Keep a separate test set that is not used to design the improvements. Otherwise, the agent may appear more capable simply because it has learned the test examples.
Only preserve lessons that demonstrate a measurable benefit.
Days 61–75: Run a Controlled Pilot
Once the improved agent performs well in tests, introduce it to a limited amount of real business work.
A useful approach is to operate the improved version in a supervised mode where it recommends actions but does not independently execute sensitive changes.
Compare its results against an existing workflow or a stable earlier version.
Make sure differences in customer type, task difficulty, or workload do not explain the apparent performance change.
For example, comparing a new system’s easy requests with an older system’s difficult requests would produce misleading results.
The team should also track negative outcomes, including incorrect tool calls, privacy issues, unexpected costs, and cases where the system fails to ask for help.
Days 76–90: Establish a Repeatable Release Process
The final stage is turning a pilot into a dependable improvement system.
Every proposed update should have a clear reason, a set of supporting examples, a comparison against the existing version, and a rollback procedure.
The team should define minimum standards for quality and safety.
An update that makes the agent faster but increases serious mistakes should not pass merely because average performance improves.
By day ninety, the startup’s main achievement should be a working cycle for collecting feedback, evaluating improvements, and releasing better versions with evidence.
That is more useful than simply claiming the product learns on its own.
How to Measure Whether an AI Agent Is Actually Improving
A common mistake in AI development is measuring activity instead of useful outcomes.
An agent may complete more model calls, generate longer explanations, or attempt more tasks without becoming more effective.
Businesses need metrics that show whether the system completes work correctly and economically.
Measure Complete-Task Success
The most important metric is the percentage of tasks completed correctly under the required conditions.
A support agent has not successfully resolved a request if it provides a friendly answer but updates the wrong customer record.
A coding agent has not completed its task if it writes code that runs but breaks another feature.
Success criteria should reflect the real business outcome, not simply whether the agent returned a response.
Track the Cost of a Correct Task
A self-improving system may use additional model calls, memory searches, testing steps, and monitoring tools.
Those expenses can make it more costly than the original agent.
One useful measure is:
\[ \text{Cost per correct task} = \frac{\text{Total operating cost}}{\text{Correctly completed tasks}} \]
This should include model usage, tool charges, review and rework, and recurring improvement infrastructure.
If the improved system becomes more accurate but its cost per correct task rises dramatically, the business should investigate whether the added reliability justifies the expense.
Measure Repeated Mistakes
A system that claims to learn should become less likely to repeat a previously identified failure under comparable conditions.
Developers can build a separate test collection containing known error cases and examine how frequently later versions repeat them.
This measure must be used carefully. If an agent simply memorizes the exact answer to every test, its score may improve without showing better performance on new cases.
Include fresh examples that test the same underlying problem in different forms.
Measure Regressions
A regression occurs when an update makes something that previously worked perform worse.
For example, an agent might become better at handling one customer policy but worse at interpreting another.
Every proposed improvement should therefore be checked against older test cases as well as the new failures it was designed to address.
Anthropic’s guidance on agent evaluations emphasizes the value of catching these problems before they reach real users.
Anthropic
A Practical Agent Improvement Scorecard
| Metric | What it reveals | Desired direction |
|---|---|---|
| Complete-task success rate | Overall reliability | Increase |
| Serious error rate | Business or safety risk | Decrease |
| Repeated-error rate | Effectiveness of corrections | Decrease |
| Cost per correct task | Operating efficiency | Decrease where feasible |
| Human review minutes | Employee workload | Decrease safely |
| Regression rate | Quality of updates | Decrease |
| Successful recovery rate | Ability to handle problems | Increase |
| Unsupported action rate | Control and compliance quality | Approach zero |
No single score can capture every important part of agent quality.
A company should examine the measures together and set clear minimum safety standards before allowing greater autonomy.
The Biggest Risks of Self-Improving AI Agents
Self-improving systems introduce risks that ordinary AI assistants do not face in quite the same way.
When software can reuse experiences and change future behavior, a single incorrect lesson may affect many later tasks.
Agents Can Learn the Wrong Lesson
Imagine a sales agent sends an inaccurate message and receives a reply.
If the system treats the reply as proof that the message was successful, it may reinforce misleading behavior.
The immediate result was positive, but the process was wrong.
This is why businesses should not use sales conversions, customer satisfaction, or task completion as their only feedback signals.
They must also evaluate whether the agent followed approved procedures and used accurate information.
Agents Can Overfit Their Tests
If a development team repeatedly improves an agent using the same small collection of examples, the software may become excellent at those examples without improving broadly.
This is similar to students memorizing answers before an exam.
To reduce the problem, teams should use separate development and final test sets, refresh evaluation examples, and test on unfamiliar situations.
Performance claims should describe which tasks were evaluated and how many examples were included.
Memory Can Become a Security Problem
Persistent agent memory may contain private customer information, employee details, business records, or sensitive operational knowledge.
A memory system should not allow one customer’s information to appear in another customer’s session.
It must also avoid treating malicious instructions found in documents, emails, or websites as trusted operating rules.
This risk is especially important for agents that retrieve external information before taking actions.
Companies need clear separation between approved instructions, retrieved content, and unverified agent-generated notes.
Self-Modification Can Make Software Difficult to Audit
If an agent can rewrite its own instructions or change its tools without recording the changes, developers may struggle to understand why its behavior changed.
This is a serious operational problem.
A business should know which version of an agent completed a task, which policies applied, what information it used, and whether its actions were authorized.
For this reason, production systems should maintain version histories and approval processes even when AI is used to generate proposed improvements.
More Autonomy Is Not Always Better
An agent that can correct a draft automatically may provide useful productivity gains with limited risk.
An agent that independently changes financial approval rules creates a very different situation.
Founders should increase autonomy according to demonstrated reliability, the impact of possible mistakes, and the availability of recovery procedures.
An agent should gain permission to do more only when the business has evidence that the additional actions can be managed safely.
What the Research Tells Us About the Future of Self-Improving AI
Several independent research directions suggest that AI agents are becoming more capable of carrying out complex work.
But progress in general AI capabilities should not be confused with proof that every deployed agent can learn safely from its mistakes.
The two developments are connected, but they need different measures.
Longer Tasks Are Becoming More Feasible
The research organization METR published work in March 2025 examining the length of software and research tasks that AI systems could complete.
Its analysis found that the task-length measure associated with 50% autonomous success had been doubling approximately every seven months over the study period.
The researchers also warned that translating this trend into real-world usefulness requires care. Their metric reflects the human time associated with a task’s difficulty, not the amount of time an agent can continuously operate without help.
METR
+1
This distinction is valuable for businesses.
An agent may become capable of tackling more difficult tasks because the underlying model improves. That does not prove that the agent has learned from its own earlier mistakes.
To demonstrate self-improvement, developers need controlled comparisons showing how feedback or retained experience changes later results.
Better Models Will Not Remove the Need for Feedback
As AI models improve, some tasks that require elaborate agents today may become easier.
However, business software must still understand company-specific rules, customer records, tool behavior, and changing procedures.
Even a highly capable general model may not know that a company revised its refund policy yesterday.
Feedback systems, trusted memory, and evaluations will remain useful because businesses need software that responds correctly to their actual operating environment.
The Next Competitive Advantage May Be Better Experience Data
Consider two startups using the same underlying AI model.
The first startup launches an agent and rarely examines its failures.
The second records verified outcomes, reviews difficult cases, maintains a library of tested solutions, and evaluates changes before deployment.
Over time, the second startup may develop a better understanding of the tasks its customers need completed.
That experience could help it build more reliable workflows, even if both companies continue using the same commercial models.
The advantage would not come from collecting the largest number of conversations. It would come from collecting the most useful, permissioned, and trustworthy examples of successful and unsuccessful work.
This suggests that high-quality feedback data could become an important competitive resource for specialized AI businesses.
The Startup Opportunities That Could Emerge From Self-Improving AI
The development of self-improving agents creates opportunities across several layers of the software market.
Some companies will build agents that complete tasks. Others will build systems that help those agents become more dependable.
Agent Memory Infrastructure
These products would help companies store, search, verify, update, and retire lessons learned during earlier tasks.
The strongest products would manage the quality and authority of memories, rather than simply increasing storage capacity.
Automated Evaluation Platforms
Evaluation businesses can help developers determine whether agents are completing tasks correctly.
A valuable platform could compare agent versions, detect changes in failure patterns, organize human reviews, and make release decisions easier to audit.
Agent Skill Libraries
Skill libraries preserve reliable sequences of actions for future reuse.
A company might create specialized libraries for accounting, software maintenance, customer support, or document processing.
The difficult part is making skills reusable while ensuring they remain compatible with changing tools and policies.
AI Failure Analysis
A failure analysis platform could examine thousands of unsuccessful agent runs and group them by cause.
Instead of forcing developers to review every case individually, it could identify repeated problems such as incorrect tool selection, missing information, or inconsistent policies.
This may become especially useful as businesses deploy agents across many workflows.
Continuous Agent Improvement Services
Some enterprises may prefer to purchase a managed improvement service rather than build the entire process internally.
Such a service could collect feedback, maintain evaluation datasets, recommend changes, and verify whether updates meet agreed standards.
The business model would need clear responsibilities for privacy, security, human approvals, and the accuracy of reported results.
Final Takeaway: The Next AI Winners May Be the Companies That Make Agents Better After Every Mistake
The rise of self-improving AI agents represents an important change in how software can be built and operated.
Traditional software follows rules established by developers. Modern AI agents can interpret goals, make decisions, and take actions. Self-improving systems add the ability to use feedback and past experience to refine how those actions are performed.
Published research provides real evidence that these methods can work in controlled environments. Reflexion demonstrated performance gains from feedback and reflection. Voyager demonstrated the value of building and reusing a library of learned skills.
Our original SFTechScene calculations highlight why these advances matter commercially.
The Reflexion figures imply a 55% relative reduction in failures under the study’s benchmark comparison. Our workflow model shows how small improvements in per-step reliability can produce larger changes in complete-task success. And our economic model demonstrates that reducing failures can create meaningful savings when the cost of correcting mistakes exceeds the expense of improving the system.
None of these findings proves that self-improving agents will automatically generate strong returns for every San Francisco business.
The evidence points to a more practical conclusion.
The most valuable self-improving systems will combine reliable feedback, controlled memory, repeatable evaluation, and clear business measurements. They will know when a lesson is useful, when an experiment has failed, and when a human must make the final decision.
For San Francisco founders, the strongest opportunity may not be building software that claims it can teach itself everything.
It may be building software that makes fewer costly mistakes as verified experience grows—and can prove that improvement to its customers.
That is the difference between an AI agent that changes and an AI agent that genuinely gets better.
Research Sources and Methodology Notes
The following primary research and technical publications form the evidence base for this article. The SFTechScene analyses are secondary calculations and transparent scenario models created for this article, not new field experiments or measurements of proprietary startup data.
| Source | Evidence used | Research role |
|---|---|---|
| Reflexion — NeurIPS 2023 | Reported HumanEval performance | Benchmark reanalysis |
| Voyager — TMLR 2024 | Reported skill-learning performance | Published experimental comparison |
| METR task-length research | Autonomous task capability trends | Research context |
| Anthropic: Building Effective Agents | Evaluator-optimizer architecture | Engineering methods |
| Anthropic: Demystifying Evals for AI Agents | Agent evaluation and reliability | Testing methodology |
| OpenAI: Evaluate Agent Workflows | Tracing, grading, and datasets | Engineering methods |
| Arize Phoenix documentation | Monitoring and evaluation | Infrastructure example |
Research limitations: The workflow reliability model assumes independent steps, identical per-step success probabilities, and no recovery. The economic model uses illustrative inputs and excludes fixed implementation costs. The workflow opportunity rankings are qualitative judgments. The empirical comparisons are limited to results reported by the original researchers and should not be interpreted as independently reproduced experiments or claims about commercial agent deployments.
Suggested SEO meta description: Discover how San Francisco startups are building self-improving AI agents that learn from mistakes, with original research, performance data, and practical business strategies.
