That is two problems: your docs fall behind every release, and assistants keep repeating what they learned before the removal. You control only part of the second.

Deleting the removed endpoint's page takes away a first-party page that could explain the removal. OpenAI keeps shut-down endpoints on its deprecations page, and Stripe marks its deprecated Sources API in both its guide and its llms.txt. Carry that to every surface an assistant reads: spec, reference, llms.txt, changelog, code samples, and help center. Update the docs in the same release as the removal, then ask the assistants again to see what changed. A retirement page does not guarantee an assistant stops recommending the endpoint. It gives a browsing assistant a current page that says the endpoint is gone.

TL;DR

• Two common sources of outdated API answers are training data you cannot edit and live pages retrieved while answering. Your docs can affect retrieved pages and the context developers load into their tools.
• ReadMe's OpenAPI sync deletes a removed operation's page, and Google drops indexed pages that return a 404 or 410 error. After that, the reference no longer says the endpoint is gone.
• Keep a retirement page at the old URL and mark the removal on every surface in the same release, with a named owner for each.
• After the release, ask the assistants the same questions and compare the pages they cite.

Where assistants get the old endpoint from

The first source is training data. A study of LLM code completion (arXiv 2406.09834v3, February 2025) evaluated seven models on 28,125 prompts covering 145 deprecated/replacement API mappings across eight Python libraries. Among completions that used either the target deprecated API or its mapped replacement, roughly 25% to 38% used the deprecated API. That denominator excludes other completions. This was a library-code benchmark, not a measurement of current ChatGPT use of retired HTTP endpoints. The authors discuss deprecated training examples and missing deprecation knowledge as possible causes.

Prompt context mattered in that benchmark. For prompts built from outdated functions, the same conditional deprecated-usage rate was roughly 70% to 90%; for prompts from updated functions, it was roughly 9% to 18%. These are rates among completions using one of the target API pair, not all completions.

The second source is the live web. Each of the four major providers documents how it reads pages at answer time. OpenAI and Anthropic keep those fetchers separate from their training crawlers, GPTBot and ClaudeBot.

All four providers document ways to retrieve live pages or a search index while answering. That gives current documentation a route into answers that use retrieval, though it does not guarantee that any particular page will be fetched or cited.

Why deleting the page makes it worse

ReadMe's OpenAPI sync treats removal as deletion: "The one thing it removes is the page for an operation that's no longer in the spec, and that page's body goes with it." The spec cannot help either. OpenAPI 3.1.1 has a deprecated field on the Operation Object, but no state for removed. Once the operation leaves the spec, a sync like ReadMe's removes its reference page too.

Google says "URLs that are already indexed and return a 4xx status code are removed from the index" for errors such as 404 and 410; it treats 429 differently and gives no fixed removal timeframe. That is the index Gemini grounds on. The deleted page drops out, while old tutorials, forum answers, and third-party pages that show the endpoint stay up. The training data still has it too, and the reference no longer contradicts any of them. A retirement page served with HTTP 200 avoids that particular 404/410 removal mechanism. It must also remain crawlable and indexable; HTTP 200 alone does not guarantee retrieval. No source we found measures whether that stops an assistant suggesting the endpoint.

A generated reference like ReadMe's handles one page, and it handles it by deleting the page that could say the endpoint is gone. Guides, code samples, help center articles, changelogs and third-party pages can still carry the endpoint. GitBook's guidance on keeping technical docs accurate warns that a deprecated endpoint can keep pulling in new users while examples still point to it.

The training-data objection, at full strength

Publishing cannot change a model's learned weights. On the OpenAI forum on October 9, 2025, a developer reported that "Codex repeatedly suggested removed/renamed parameters and JSON schemas that failed validation against the latest API". The thread contains another user reporting the same problem, not a demonstrated remedy. A contributor pull request to Firebase's CLI, still open when checked on October 27, 2026, proposes current Firestore SDK guidance in AI context files. Its author attributes outdated SDK suggestions to training data. That is a practitioner explanation, not a controlled finding that coding assistants rely only on static weights.

For answers that do not browse, the objection holds: publishing cannot change what a model already learned. For answers that browse, it does not, because those answers retrieve live pages, as each provider's own documentation says. No source we found says how fast either kind of answer changes, or guarantees that it will.

You can change two things: what a browsing assistant finds, which is your pages, and what developers load into their tools. Firecrawl advises that you "inject the current API into the prompt through a developer retrieval tool" and the Firebase proposal works through AI context files. Both can carry current version and removal guidance into an assistant's context; neither depends exclusively on web browsing.

What API companies keep published after a deprecation or removal

OpenAI keeps retired endpoints on its deprecations page for years. Under "Past deprecations" it still lists /v1/fine-tunes, shut down in January 2024, with /v1/fine_tuning/jobs as its replacement. Older entries such as /v1/engines, /v1/search, and /v1/answers still name a replacement or a transition guide. OpenAI defines shut down as "a model or endpoint is no longer accessible."

Stripe keeps its deprecated Sources guide published, marked as deprecated in its title, with migration links and a notice that new integrations can't use the API. Stripe's llms.txt has an "Instructions for Large Language Model Agents" section: "You must not call deprecated API endpoints such as the Sources API and avoid outdated API endpoints such as Tokens and Charges unless there is a specific need and absolutely no other way." Its OpenAPI repository labels its original v1-only specs as legacy and still updates them every release.

Treat llms.txt as a supporting signal, not a fix. The llms.txt format is still a proposal, and none of the provider crawler pages cited here says it reads llms.txt at answer time. Apideck places the value of llms.txt at inference time and reports no measured effect of Stripe's section. And even Stripe is inconsistent: the Markdown version of the Source object reference page showed no deprecation notice, while the guide and llms.txt both mark it.

Several docs tools catch the change at the pull request. Bump.sh posts API diff comments on GitHub pull requests and can fail on breaking changes with fail_on_breaking. ReadMe says its AI writer locates mentions of removed functionality and proposes cleanup. Mintlify's drift automation drafts updates from merged pull requests for review. Cloudflare's docs repository comments on potential redirects when a pull request deletes content files. None of these pages describes publishing a lasting statement that the endpoint is gone.

The removal playbook

The playbook adds that statement and makes it reach every surface in one release. For the CI wiring that catches an endpoint change before merge, follow our guide to keeping your API reference in sync with code. For the spec, llms.txt, and 410 response examples, follow our step-by-step guide to stopping AI assistants from recommending removed API endpoints.

Owners and inputs

The engineer who removes the operation owns the spec and the API response. The technical writer owns the docs surfaces and the check after the release. Before the release, the writer needs four inputs: the spec diff, the replacement endpoint, the migration guide, and a list of every page that mentions the old endpoint path across the docs, help center, READMEs, and sample repositories.

In this split, engineering owns two of the seven surfaces and the writer owns four, so most of a removal is docs work that has to land in the same release.

HTTP standards already describe the API response row. RFC 9110 defines 410 Gone as "access to the target resource is no longer available at the origin server." RFC 9745 (March 2025) defines a deprecation link relation that points to human-readable documentation about the deprecation. RFC 8594 defines the sunset link relation for retirement policy. Your retirement page is the documentation those links point to. Also check that robots.txt lets the answer-time fetchers in the first table read it. OpenAI says its search systems can take about 24 hours to adjust after a robots.txt update.

Cadence and checkpoints

1. At the deprecation announcement, mark the operation deprecated, add the notice to its reference page, add the llms.txt instruction, and add the changelog entry.
2. On the pull request that removes the operation, make the docs impact decision GitBook recommends for every pull request. Checkpoint: a linked docs pull request exists before the removal merges.
3. On release day, confirm the retirement page is live at the old URL and the sweep of guides, samples, and help center articles is merged.
4. After the release, ask the assistants again and compare the answers.

One worked cycle

Say your team is retiring /api/v1/users for a v2 endpoint. At announcement, the engineer sets deprecated on the operation, and the writer adds a reference notice, an llms.txt line naming the v2 replacement, and a changelog entry. When the removal pull request opens, the writer opens a linked docs pull request. It turns the reference page into a retirement page, replaces the v1 call in the quickstart, the guides, and the sample repository, and updates the help center article on listing users. The engineer makes the old route return 410 with a link to the retirement page. Both pull requests merge in the same release. A week later, the writer reruns the assistant questions and compares the answers with the ones recorded before the release.

How to check whether assistant answers changed

None of the guides we reviewed compares assistant answers from before and after a removal. Build the check yourself:

1. Before the release, write down the questions developers ask about the removed endpoint, in their words.
2. Ask each assistant every question with browsing on, more than once. Record whether the removed endpoint appears and which pages the answer cites.
3. After the release, ask the same questions in the same assistants and compare.

When an answer cites your retirement page, check the URL and whether the answer reflects the removal. An answer that cites a third-party tutorial still showing the old call gives you a page to ask its author to update. An answer without citations does not by itself show whether retrieval occurred; inspect the tool or search trace when available. Date every run, because no source we found says how long answers take to change.

Where EkLine fits

For the docs side of this fix, use EkLine Docs Agent. The sweep is where the playbook is hardest: the guides, READMEs, and knowledge-base articles a generated reference never touches. Docs Agent keeps READMEs, API references, guides, and knowledge-base pages current in three ways. When a non-draft pull request becomes ready for review in a monitored repository, it checks the diff against your docs and, on a high-confidence match, drafts the update as a documentation pull request. On a schedule you set, a drift audit finds customer-facing changes that merged without documentation and fixes the highest-priority gap. And it takes on-demand requests, like the example in its update docs: "We deprecated the /api/v1/users endpoint. Update the API reference to mark it as deprecated and point to the v2 replacement." In GitHub, mentioning @ekline-ai in a pull request comment starts a session that drafts the docs and can open a docs pull request.

Before: the removal merges, the reference page disappears, and the guides keep the old call until a developer reports it. After: on a high-confidence match, EkLine drafts a documentation pull request. Your writer checks that it covers the reference and related pages before approval, because "Every update arrives as a pull request on your documentation repository." At Socure, one technical writer merged 348 documentation updates in seven weeks with no new hires.

EkLine also runs the check after the release. It asks ChatGPT, Claude, Perplexity, and Gemini the questions your developers and buyers ask, shows which pages they cited, and lets you "Re-run the same questions and tasks after you ship and compare." EkLine cannot change what a model already learned. It keeps your pages current and shows whether the answers that browse have caught up. For drift beyond removed endpoints, see our guide to documentation drift.

Book a demo to see what AI assistants currently say about your API, or get started with EkLine.

FAQ

Why does ChatGPT recommend API endpoints that no longer exist?

Training data can preserve old API patterns, while retrieved pages and developer-provided context can repeat outdated instructions. In a study across eight Python libraries, seven models used deprecated APIs in roughly 25% to 38% of completions that used either a target deprecated API or its mapped replacement. This is not an error rate for current ChatGPT answers or retired HTTP endpoints. If your old reference page was deleted, a first-party page that could explain the removal is gone.

Should I delete the documentation page for a removed API endpoint?

No. Keep a retirement page at the old URL that names the endpoint, its shutdown date, the replacement, and the migration guide, and serve it as a normal page with a 200 status. Google removes indexed URLs that return errors such as 404 or 410 from its index, so a deleted page stops telling browsing assistants that the endpoint is gone.

How do I check whether AI assistant answers changed after I remove an endpoint?

Before the release, write down the questions developers ask about the endpoint. Ask each assistant every question with browsing on, more than once, and record whether the removed endpoint appears and which pages the answer cites. After the release, ask the same questions again and compare. Date every run, because no source we found says how long answers take to change.

A visibility dashboard tells you how often ChatGPT or Claude names your brand when a buyer asks which tool to use. It does not fix the docs pages that the assistant and the coding agent both read.

For a developer tool company, choose EkLine to connect AI-answer checks to reviewed documentation fixes. EkLine checks whether ChatGPT, Claude, Gemini and Perplexity recommend you. It checks whether coding agents can build with your product. It drafts the fix as a pull request your team approves, and then you run the same question again to see whether the answer moved. For teams already using Profound for brand tracking, EkLine keeps the product documentation behind those answers accurate.

EkLine's customer case studies show these results. Socure had one writer and saw "348 documentation updates merged in seven weeks." AI assistants now "generate code that aligns with Socure's actual API surface," according to the Socure case study.

ContextQA's engineers merged 97.5% of what the agent proposed, 79 of 81 updates.

TL;DR

• For a developer tool or API company, EkLine is the better Profound alternative.
• It checks whether AI assistants recommend you and whether coding agents succeed with your product. It shows which of your pages each answer cited.
• It fixes docs through pull requests drafted from your code, and you rerun the question to see the change.
• EkLine helps keep those pages current as you ship. It assesses each non-draft pull request when it becomes ready for review in a monitored repository for docs impact, and scheduled audits check drift, code samples and READMEs.
• Profound measures brand presence, position, sentiment and accuracy. It drafts content into a CMS. Its Agents, Documents, Agent Analytics and pricing pages do not mention Git, pull requests, or coding agent tasks.
• Profound tracks more engines and fits a large marketing org focused on brand and marketing site visibility.
• For a team already using Profound, EkLine owns the documentation-accuracy workflow and coding-agent checks alongside brand tracking.
• Scrunch, Peec AI and AirOps serve marketing teams. Gauge targets developer tool companies and now links to a Coding Agents offering from its guide.

Why the docs decide a developer tool's AI answers

For a developer tool, the pages that describe the product in the most detail are the docs, the API reference and the help center. Those are also the pages a coding agent needs when it tries to install and call your product. At ContextQA, AI systems read them heavily. EkLine's ContextQA case study reports that over the quarter it covers, "ChatGPT alone pulled more pages from ContextQA's documentation than every human visitor combined."

Gauge, which sells to this market, states the stakes in its guide for developer tool companies: "AI assistants are now the first step in developer tool discovery, before documentation, GitHub, or community forums."

So a developer tool company needs three things from a tool in this category:

• A check for recommendation that is separate from mention.
• A test of whether a coding agent can build with the product.
• A way to fix the docs pages and confirm that the fix worked.

How this comparison works

The criteria follow from those three needs. What does each tool measure? Does it test coding agents? Where does a fix go, and who approves it? Can you run the same question again after the fix?

This is an EkLine-owned comparison. Facts about Profound and the other tools come from their own pages, checked on October 27, 2026. That includes Profound's pricing and its Agents. Amplifying's results are an early-2026 snapshot from greenfield repositories, not a measure of developer consensus or tool quality. We cite roundups for how others group these tools, for Gauge's own positioning and for Profound's earlier self-serve prices.

Profound vs EkLine for a developer tool company

Profound covers more engines and publishes to a marketing CMS. EkLine checks recommendation, tests coding agents and shows which of your pages shaped the answer. It also fixes the surfaces in the table after your team reviews them. Docs in your repo change through a pull request. Help center articles publish after you approve them.

How EkLine measures, fixes and proves

Illustrative workflow. The diagram names Codex, but EkLine's current public homepage does not confirm Codex support. Confirm available coding agents in your demo.

Measure what buyers and coding agents see

EkLine checks buyer questions across ChatGPT, Claude, Gemini and Perplexity. It also checks whether coding agents can build with your product, the way a developer would after picking a tool.

EkLine separates recommendation from mention and checks whether agents answer correctly about your product. In your evaluation, also record whether your product appears in the first answer of a conversation and remains in the final answer.

For coding-agent tasks, ask to see the evidence behind each result: whether the commands succeeded, whether the documented steps worked in order, and whether pages gave conflicting values, commands or sequences. These checks help you judge whether a quickstart is usable.

EkLine shows which pages answers cite. Inspect those sources to see whether an answer used your current docs, an old page, or a competitor's roundup, then decide which page to fix first.

Fix the page as a pull request

Use the finding to start a Docs Agent session and describe the correction you need. Docs Agent connects to GitHub and GitLab. It takes context from Jira, Linear, Slack, Notion, Confluence and Google Docs, and opens a documentation pull request.

As the guide to keeping docs in sync with code puts it, "You review and merge it through your normal process before any change reaches your main branch." EkLine works with docs in Mintlify, ReadMe, GitBook, Docusaurus or Markdown in your repo, and with Confluence or Pylon knowledge bases. You keep your docs where they are. The EkLine homepage says "If your docs live in Git, setup takes about 15 minutes."

Run the question again

After the merge, run the same questions and tasks again and compare them with the baseline. Check whether the answer changed and whether it now reflects the updated documentation.

Docs Agent drafts the fix, so engineers review instead of write. At P0 Security, "In 6 months, engineers reclaimed 150+ hours of product development time by reviewing documentation instead of writing it."

What EkLine does that a visibility tracker does not

Fixes grounded in your code

Docs Agent reads your source code as well as the page. It can generate an API reference from your source code, reading "function signatures, types, and comments" to produce parameters, response schemas, example requests and error codes. It turns Jira or Linear tickets into release notes. And it drafts help center articles from the support thread that already resolved the question, then publishes them to Confluence or Pylon after your review.

Pages that stay current on every release

Docs fall behind code the moment a pull request merges without a docs update. EkLine reviews every ready-for-review pull request when it becomes ready for review in a monitored repository. The assessment runs once; later commits need a manual mention to trigger another review. When it finds clear docs impact, it starts a docs update. When the impact is less certain, it suggests one in a comment on the pull request. Scheduled Agents sweep for what slipped through. The Code Sample Review template "verifies that code samples in your docs are accurate and runnable against the APIs and SDKs they demonstrate." Other templates audit drift, README accuracy, UI labels and screenshots, and can deliver pull requests when that output is enabled.

Your voice and your terms

Docs Agent learns your style from your existing docs. Custom instructions add the conventions it can't infer, such as spelling, terminology and which versions you support. Repo changes land as pull requests and help center edits publish only when you approve them, so nothing ships without your team.

Measure, fix and rerun in one place

The buying question is how measurement connects to a reviewed documentation change. MentionX defines a "Recommendation Rate" and a "First-choice Rate." Mintlify's agent "creates pull requests with proposed changes to your documentation" on Pro or Enterprise. EkLine connects recommendation and coding-agent checks with product-grounded documentation updates and a post-release comparison.

The case for keeping Profound

Profound's Enterprise plan tracks "Up to 9 Answer Engines," including ChatGPT, Perplexity, Gemini, Microsoft Copilot, Anthropic Claude and Google AI Overviews. It includes SSO and SOC 2. Agent Analytics tracks AI traffic to your site. Agents "connect to your existing CMS and publish approved content," and Profound Documents names Webflow and Contentful as destinations. Profound says it serves "more than 700 enterprise brands and large agencies," including MongoDB and Plaid.

These features cover brand and marketing-site visibility. For a developer tool, the docs are the pages AI assistants and coding agents read about how your product works. Profound's Agents and Documents pages do not mention Git or pull requests. Its Agent Analytics page does not mention Claude Code, Codex or Cursor. Its help docs compute the Visibility score from "the number of responses that include your brand" and name no metric for whether a brand is recommended.

When to choose EkLine and when Profound fits

Choose EkLine if you sell a developer tool or API. Your buyers ask ChatGPT or Claude what to use, then build with Claude Code, Codex or Cursor. You need to know whether you are recommended and whether the agent completes the task. And you want the docs fixed through pull requests your team reviews. Marketing identifies the answer gap, and the docs owner reviews and merges the proposed correction.

Profound fits a large marketing org that tracks many engines and brands across marketing sites. That team wants agent traffic analytics and SSO, and publishes to a CMS. EkLine owns the documentation-accuracy workflow alongside that brand tracking. The EkLine homepage says the same about SEO tools: "Keep your SEO tool; EkLine works alongside it."

Other Profound alternatives and where they fit

The Frase and AirOps roundups sort these tools by how far they carry a visibility finding toward a published page. AirOps puts it plainly: "Dashboards show what's happening, but they don't update content." The roundups emphasize content workflows and CMS publishing.

Scrunch offers AI visibility monitoring, site audits and AI agent/bot traffic reporting. Its pricing page lists Core at $250/mo. Confirm billing terms directly with Scrunch. It fits a marketing team that wants monitoring at a published price.

Peec AI offers "AI search analytics for marketing teams" across ChatGPT, Perplexity and Gemini, with recommendations and an API. It also offers a Peec MCP server. It fits a marketing team that wants AI search data inside its own tools.

AirOps combines AI search insights with content workflows. AirOps Pro currently lists "Insights Across 10+ Answer Engines" and CMS integrations. Its pricing page and its Profound alternatives post do not describe Git pull-request delivery. It fits a content team that publishes marketing pages.

Gauge's own guide positions it for developer tool companies and lists Growth at "$599/mo." Its current navigation also links to Coding Agents and a guide to coding-agent visibility tools. Compare that offering directly if coding-agent evaluation is part of your buying criteria.

Docs platforms are a separate category. Mintlify's agent opens pull requests, and its agent page does not mention measuring AI visibility. If you are choosing a docs platform, our Mintlify alternatives guide covers that decision.

A checklist for evaluating any Profound alternative

• Does it report whether you are recommended, separately from whether you are mentioned, and does it say how it decides?
• Does it check where you stand in the first answer of a conversation and in the last?
• Which engines does it cover, and is Claude one of them?
• Does it give tasks to coding agents such as Claude Code or Codex?
• Does it judge a task by whether the commands actually succeeded?
• When it finds a problem, does the fix go to a CMS or to a pull request in the repo where your docs live?
• Does the fix start from your code and tickets?
• Does it show which of your pages each answer cited?
• Does it keep pages current on every release, or only react after an answer goes wrong?
• Who approves the change before it ships?
• Can you run the same question again after the fix and see the trend?
• Are its prices and security claims, such as SOC 2 Type II and SSO, on its own pages?

FAQ

What is the best Profound alternative for a developer tool company?

EkLine. For a developer tool company, the docs, API reference and help center are the pages AI assistants and coding agents read. EkLine checks whether ChatGPT, Claude, Gemini and Perplexity recommend you. It checks whether coding agents can complete tasks with your product. It shows which of your pages each answer cited, opens a pull request with the fix drafted from your code, and lets you run the question again. It also assesses non-draft pull requests when they become ready for review in monitored repositories, helping keep pages current with releases. For teams already using Profound, EkLine owns the documentation-accuracy workflow alongside brand tracking.

How can we track whether ChatGPT and Claude recommend our developer tool, not just mention it?

Define recommendation separately from mention and check it on every run. EkLine distinguishes recommendation from mention. For multi-turn evaluations, also record whether your product appears in the first answer and remains in the final answer. MentionX also defines a "Recommendation Rate." Profound's help docs describe Visibility score, Share of Voice, Average Position, Citations, Sentiment and FactCheck, and name no metric for recommendation.

Which tools fix the pages AI assistants read instead of only reporting mentions?

Profound and AirOps draft content and publish it to a CMS such as Webflow or Contentful. Mintlify's agent opens pull requests with proposed docs changes. Use an EkLine finding to request a product-grounded documentation pull request. Your team reviews and merges it, then you run the question again.

Does EkLine replace Profound?

If your AI visibility work is about docs and coding agents, yes. EkLine covers the recommendation checks, the coding agent tasks, the cited pages and the docs fixes in one loop. A large marketing org that also wants Profound's brand tracking across more engines and its AI traffic analytics can keep it alongside EkLine.

How much does Profound cost now?

Profound's pricing page lists a free Trial and an Enterprise plan priced as "Custom," with no dollar figure. Eclincher, which sells a competing tool, reports that in mid-September 2026 Profound "removed its self-serve Starter ($99 per month) and Growth ($399 per month) plans for brands."

Next step

If you sell a developer tool and want to know whether ChatGPT and Claude recommend it and whether Claude Code can build with it, book a demo with EkLine.

EkLine provides the accuracy workflow around that help center: scheduled agents draft missing or corrected articles from Pylon tickets, while a custom scheduled prompt can check the product changes you specify against relevant articles. Your team reviews the proposed changes and controls when they reach Pylon.

Keep the knowledge base under review from two directions. Tickets tell you what is missing or confusing. Releases tell you what may have become wrong. Start the release check from what is shipping and give each fix one named approver. Pylon's own guide recommends updates with every significant product release and a quarterly audit of the full knowledge base.

TL;DR

• Tickets show you which articles are missing or incomplete. Releases show you which articles became wrong. Each direction needs its own check.
• Pylon provides ticket-based gap signals. Add release context to find articles affected by changes before the next support issue.
• A release-time check starts from the list of user-facing changes, searches the help center for every old name and step, and gives each fix one approver before customers see the release.
• Run three cadences with one owner: a weekly ticket check, a check for every release, and a quarterly sweep.

Tickets show what is missing, releases show what became wrong

A knowledge base fails customers in two ways. In the first, the article was never written, and customers keep asking a question the help center does not answer. Your ticket queue shows this well, because repeat questions are the signal. EkLine has written about how your support tickets are a documentation backlog in disguise.

In the second, the article was right when you published it and a release changed the product underneath it. A support ticket may arrive later, when a customer follows the old steps and they fail. If that article is available to an AI agent, stale instructions can also enter its answers before your team notices.

In a Slite survey of 143 founders, operators, and engineers about internal docs, 41% of respondents who had seen an AI tool surface a stale document and produce a wrong answer said the wrong information had reached a customer. That is a reported experience within Slite's survey group, not an industry-wide failure rate.

Support teams have described release maintenance as manual work. In an Intercom community feature request, Audrey Ware asked for release notes to identify affected articles and suggest edits. Cassie Penatzer's April 16, 2026 reply described the time spent finding those articles. This is practitioner evidence of the task, not proof that the vendor still lacks a solution: Intercom now documents Fin Operator's product-update-to-article workflow, with proposed changes reviewed before they go live.

How EkLine runs the accuracy workflow

EkLine connects ticket evidence, specified product changes, and reviewable knowledge-base edits in one accuracy workflow. Use the ticket template for recurring gap discovery and configure the release check for the changes your team needs to validate.

For the ticket side, the Improve Pylon knowledge base scheduled agent reviews recent Pylon tickets for questions that the knowledge base should have answered, then drafts missing or corrected articles for review. This template delivers knowledge-base drafts in Slack by default rather than repository changes. Review its prompt and notification settings before enabling it.

For the release side, configure a custom scheduled prompt. Each run executes the prompt against your connected repository. Specify the release context, managed Pylon articles, and draft output you want; the knowledge-base management guide illustrates a related check by comparing a billing FAQ with the latest pricing page. Set a recurring cadence, or use Run now before a release. Treat this as a configured workflow to test, not an automatic release-event trigger or a claim that the ticket template checks code.

Keep publication approval with a named person. In the documented knowledge-base workflow, review the diff under View All Changes, then use Update KB to write the edits back. That action immediately changes a live article's content; for Pylon it does not change the article's published or unpublished status. Setup requires Docs Agent and knowledge-base management access, a Pylon API token, selected managed knowledge bases, editing permission, and a Default author for new articles.

Before: the support lead reads tickets individually, chases the ship list, and searches for old setting names by hand. After: EkLine runs the configured ticket and release checks, drafts the corrections, and brings the changes to a reviewer. The reviewer checks each fix in View All Changes, and the authorized publisher uses Update KB. Pylon remains the article destination throughout.

The ticket template is documented for ticket review; do not assume it also audits releases or code. A release check depends on the context and managed articles you make available, and it needs a test run before you rely on it. Scheduled agents require Docs Agent and a connected repository. Knowledge-base management is available on request.

To evaluate this accuracy workflow, follow the walkthrough for managing a Pylon knowledge base with EkLine, then request access at support@ekline.io with your organization name. Test one ticket batch and one release against a small set of managed articles before expanding the cadence.

Pylon signals and publishing controls

The table separates support-driven signals from the controls used to review and publish a correction. The release check connects the ship list to the articles that need to change.

The first five rows use support activity or article usage; the last two cover review and shared edits. Pylon's knowledge base page describes turning conversations across Slack, Microsoft Teams, email, and in-app chat into suggested articles and updates. Its AI resource access differs by feature: AI Assistants can use replies, internal notes, fields, KB articles, recordings, and training data; AI Agents use configured KB resources and selected training data, rather than historical issues. Pylon also supports configurable Skills with connected tools. The workflow here assigns recurring, repository-grounded accuracy checks and managed-KB corrections to EkLine; it does not assume Pylon is limited to the signals in this table.

Why ticket signals need release context

The ticket-based checks above can already produce drafts and suggested edits. Missing Content collects issues that agents flag, while Improvements uses the AI agent's escalation reason, topic, and frequency to suggest article work. Reviewing another routine costs time, so each release check should name the changed behavior and the articles it needs to verify.

Use ticket evidence to prioritize corrections and release context to find affected pages that have not yet produced a support issue.

Those issue-based queues need support activity to supply evidence, while search and feedback reports need reader activity. An Improvements suggestion can follow an escalation; a Missing Content entry can follow a flagged issue. Topic suggestions begin after 300 logged issues and require an eligible plan. A zero-result search can expose missing terminology, but it does not identify every existing article with old instructions. A feedback score also depends on someone rating the page. These signals help prioritize work; they do not replace checking a known product change against affected content.

Pylon's guide recommends article updates with significant releases. To find affected content before customers encounter it, start the check from the release itself.

How the release-time check works

The release-time check, step by step.

The release check starts from the product change and looks for articles it may have made wrong. Run it for every release a customer can see, with enough time for review before rollout. The steps below define the check whether the team is testing an EkLine custom prompt or verifying its draft output.

1. Get the list of user-facing changes before the release goes out, from the release notes, the changelog, or the product manager. In a September 11, 2024 community reply, Intercom's senior knowledge manager Beth-Ann described getting involved before shipping and typically receiving at least four weeks to prepare new content. That is a historical operating example, not a required lead time or a claim about Intercom's current process.
2. For each change, write down the old words customers would have read: setting names, button labels, menu paths, limits, and steps that moved or disappeared.
3. Search your help center for each old word and list every article that contains it. Check your Synced Blocks too, since one edit there updates every article that embeds the block.
4. Name one approver for each fix, usually the product manager or engineer who shipped the change, and add them as a reviewer under Review & Publish if using Pylon's review flow. The author drafts the edit and clicks Request Review. Require review before publishing where appropriate, and check publishing permissions: Pylon allows reviewers with publishing permission to publish without a separate approval.
5. Publish the approved edits when the release reaches customers. Knowmax's knowledge-ops guidance recommends reviewing content affected by launches, policy changes, pricing updates, or regulation within 24 to 48 hours after the change. Use publication with the release as the target for known changes, rather than treating that window as an acceptable delay for every update.

Teams have done this by hand for years. In an older HelpDocs roundup, Front's support manager met the product team weekly and kept a running list of articles to update, while Wistia's team watched GitHub pull requests. In the 2024 Intercom reply above, Beth-Ann reported setting aside time from 13 support agents for product-change audits. These are historical examples of coordination work, not current staffing requirements. EkLine makes the repeated checks and draft preparation a configured routine, while the team retains review and release timing.

Keep the old words from each release on a running list. The next weekly ticket check can then show whether any article slipped through.

Owners, cadence, and checkpoints

Assign one knowledge-base owner to coordinate the routine and its reviewers. Knowmax's guidance recommends a named person responsible for each article's accuracy. One knowledge-base owner coordinates the three checks below; subject-matter reviewers own individual fixes.

The release and quarterly rows follow Pylon's recommended cadence; the weekly row makes ticket review repeatable. The release row explicitly starts from a known change before rollout, while the quarterly sweep checks the wider inventory. Knowmax also recommends periodic review of evergreen content, giving 90 days as an example. These are operating targets, not a guarantee that every mismatch will be found.

One worked cycle

Take one illustrative week. On Monday, the support lead reviews EkLine's scheduled ticket drafts alongside Missing Content, Improvements, and searches with no results. They check issues that agents flagged for article updates, use Article Generation where needed, and accept or reject suggested edits. Each correction gets an owner before it is published.

On Wednesday, a release will rename a setting. Earlier in the week, the product manager shares the ship list. The support lead gives EkLine's custom release check the changes and relevant managed articles, then runs it before rollout. The reviewer checks the drafts against the product and searches for the old name, including synced blocks. The product manager approves the corrections through the team's chosen review flow, and the authorized publisher applies the edits when the release reaches customers.

The next Monday, the ticket check also tests the release check. If a ticket still quotes the old setting name, the support lead investigates a missed article, a saved copy, or another source before deciding what to fix. If an article was missed, they correct it and add the search term or context that would have found it to the next run.

FAQ

How do you know when a knowledge base article is outdated?

Check from tickets and releases. A support agent can flag an issue for an article update, Improvements can identify articles needing revision from AI-agent experience, and readers can rate content unhelpful. Separately, a release can change a setting, label, or step that an article still describes. Checking the release before rollout gives the reviewer a chance to correct that mismatch before it produces a support issue.

Can article view counts tell us which content needs updating?

View counts help prioritize review, but they do not show whether instructions match the product. Pylon separates human and non-human traffic, so use the relevant traffic view when estimating customer exposure. Give heavily used articles priority in the quarterly sweep, and use release context to identify content affected by a specific change.

How can AI improve my knowledge base?

Pylon's Article Generation drafts from issues, Improvements suggests edits from AI-agent experience, and Topics groups uncovered questions. EkLine schedules drafts from ticket evidence and configurable checks against the product or release context you supply. Review the proposed fixes against that evidence before publishing; an AI system can use stale instructions if they remain in its accessible sources.

Does EkLine replace Pylon?

Pylon remains the destination where your team manages and publishes knowledge-base articles. EkLine replaces the fragmented accuracy work around it: reading ticket evidence, checking the product changes you specify, drafting corrections, and bringing those changes to a reviewer. Your team keeps publication control.

Start with one release and one named reviewer. Use the EkLine setup walkthrough above to connect the evidence, inspect the proposed corrections, and verify the updated Pylon articles after publication. Expand the cadence only after that cycle produces changes your team can review and trust.

Write one auditable billable-event definition before you pick a rate. Choose an outcome unit only when completion can be observed, attributed, explained, and disputed under an agreed definition, at an operating cost you can accept. When it cannot, price an output your system observes, or a base fee with an allowance, and give the buyer spend controls.

The definition questions, record fields, and edge-case rules below are a proposed design. They are not a recognized standard, and they are not any vendor's policy or legal terms. Vendor rules and linked guidance were rechecked on October 21, 2026.

TL;DR

• Separate four layers: input cost, performed action, delivered output, and business outcome. Bill on a layer both sides can inspect.
• Define the unit before the rate: success predicate, boundary, evidence, window, uniqueness, exclusions, reversal, inspection rights, and who settles disagreements.
• Cost per attempt says little about cost per billable outcome. In the illustrative model below, $0.12 per attempt becomes about $2.89 per net billable outcome.
• Seats, metered usage, and hybrid plans remain sound choices. Zendesk draws its outcome units from an allowance, and Intercom sets a minimum commitment for standalone Fin.
• Before you set a price, replay recent task traces through the edge-case checklist.

Separate four layers before you pick a unit

Scott Woody, identified in Stripe's October 1, 2026 post as Metronome's CEO and founder, splits AI pricing into three metrics on Stripe's blog: inputs such as tokens, outputs that are objective and verifiable such as a generated image or a sent email, and outcomes, such as gains in revenue or pipeline and a reduction in customer churn. This post adds a fourth layer between input and output: the performed action.

An API call that succeeds, or a flag the agent sets on itself, shows that the agent acted. It does not show that the customer received value, so billing actions as outcomes is usage pricing with a misleading label. At the top layer, Woody identifies a measurement problem: agreeing on the result, detecting it, and separating your product's contribution from other causes.

Compare the five units on the same criteria

The alternatives a buyer already knows are seats, metered usage, and credits with an allowance. Intercom and Zendesk keep their outcome units next to those models. Zendesk draws resolutions from an allowance, and Intercom's pricing page sets a minimum monthly commitment for standalone Fin. Salesforce sells its agent product as Flex Credits, a per-conversation price, and per-user add-ons and licenses.

The ratings are this post's assessment. They show a trade: the closer a unit sits to the customer's result, the better it tracks value and the more each charge costs to observe, attribute, and defend.

Where each unit wins

PricingOS offers a decision path that fits the table. It reserves outcome pricing for cases where success fits "one unambiguous sentence" and you can absorb the failures. When usage varies strongly across customers, it points to credits with a published conversion or usage with a spend cap. Seats remain the simplest sale when usage is stable and the agent is one feature among many. When buyers demand predictability and you need cost coverage, it recommends a hybrid of a base fee plus an allowance, with overage priced transparently.

Add one step to that path: when the customer's result is out of reach but your system emits a clear output, price the output. BillingPlatform's comparison draws the same line: its working examples share "a clean success signal that the AI system itself generates and the customer can verify."

Define the unit before the rate

PricingOS frames outcome pricing as defining, in writing, the single event that triggers a charge, in a way a customer will not dispute on an invoice. A billable-event definition that holds up answers nine questions:

1. Success predicate: the one sentence that says the unit happened. Brocoders asks whether "done" fits in one sentence that a lawyer and an engineer would both sign off on.
2. Unit boundary: what starts and ends one unit, such as a conversation, a ticket, or a job.
3. Evidence source: the record that proves the predicate. Flexprice's glossary says each verification signal needs "a timestamp and an actor so the charge survives a dispute."
4. Observation window: how long you wait before an event becomes billable.
5. Uniqueness: one identifier per unit, so a duplicate report bills once.
6. Exclusions: the cases that never bill, written as a list.
7. Reversal and dispute policy: what cancels a billed unit, until when, and how the credit appears.
8. Inspection: which records behind each charge the buyer can see.
9. Resolution: who settles a disagreement, and by when.

A billable-event record you can adapt

The billable-event record stores those answers for each charge. The values below are invented placeholders and contain no customer data.

# Proposed billable-event record: an illustrative design, not a standard
event_id: evt_9a21_example        # unique; reused if the same event is reported again
task_id: task_7f3c
account_id: acct_example          # customer or account scope
outcome_type: ticket_resolved
definition_version: "2026-10-01"  # definition wording this event was judged against
success_evidence:
  method: customer_confirmation   # or system_check, no_follow_up_in_window
  ref: evidence/task_7f3c/final_turn
eligibility: eligible             # eligible | excluded
exclusion_reason: null            # for example greeting_only, default_escalation
occurred_at: "2026-10-14T09:12:05Z"   # UTC
observed_at: "2026-10-14T11:12:05Z"   # UTC, when the window closed
observation_window: PT2H          # ISO 8601 duration
status: billable                  # pending | billable | excluded | reversed
reverses_event_id: null           # on a reversal record, the event it cancels
credit_ref: null                  # link to the credit issued, if any
actor:
  completed_by: agent             # agent | human | agent_after_human
  human_touched: false

occurred_at and observed_at separate when the work happened from when it became billable. definition_version ties each charge to the definition wording in force, which matters when you add outcome types later. reverses_event_id makes a reversal its own record, so the original stays intact for audit.

An example lifecycle: attempts, evidence, an observation window, then billing, with exclusion and reversal as the two exits.

Test the definition against seven edge cases

The example rules are this post's design choices. The last column describes a vendor's documented rule where one exists.

The Stripe late-event rule has a contract consequence. If your billing system can accept an event without changing either invoice, your billable-event definition needs its own rule for late evidence.

What vendor definitions teach

Intercom defines a resolution as counted when, after Fin's last answer, the customer either confirms the answer was satisfactory or "exits the conversation without requesting further assistance." Zendesk hands the judgment to an LLM. If you also use these definitions to judge support quality, see this comparison of vendor resolution definitions as quality metrics.

Definitions also change. Stripe's account says Intercom's Fin first charged for each support conversation it resolved. When Fin expanded into work such as lead qualification, a support-only unit no longer described all the work customers bought, so Fin added more billable outcomes. The definition_version field is there for that day.

Give the buyer a control surface

In the same Stripe post, James Brown names the buyer's side of usage-based billing: customers "need to understand what drives usage, anticipate spend, set spend limits, and connect usage to value." Design spending caps, alerts, cost and usage visibility, spend estimates, audit records, and a way to contest an event.

Know whether each control warns or stops. In Stripe Billing, usage alerts notify you when a customer crosses a meter threshold, or trigger an invoice at a billing threshold; the alerts page does not describe either one as stopping usage. For accounts on Zendesk's resolution-allowance model, "Don't allow overage" pauses capabilities that require paid resolutions at the allowance limit. The setting cannot be changed for Forethought AI agents by Zendesk plans. Under the default, "Allow overage," agents keep serving customers and overage is billed monthly.

The cited cap and alert pages do not specify a complete billing rule for work already in progress at the limit. Pick one answer (finish and bill, finish and waive, or stop and exclude) and write it into the definition. Credit pools are another control: Stripe reports that Intercom's Fin lets customers commit to a pool of credits and draw it down across outcomes without forecasting each one.

Model cost per billable outcome

This model is illustrative: every input is an assumption chosen to show the method. The boundary is one task type for one month.

At $0.12 per attempt, the model still costs about $2.89 per net billable outcome, and disputes alone cost more than all the inference. It sets no price and claims no margin. Replace each input with numbers from your own traces.

When you cannot observe the customer's business result

You may not be able to observe the customer's business result, or prove your agent caused it. You can still bill a system-observable event that both sides agree, in advance, to inspect.

Woody's sales-agent example shows why. Charging on the revenue an agent generates, he writes, invites an argument about which part came from the software and which from the sales team. Intercom bills a resolution under a published rule, and revenue never enters it. Your job is to prove the evidence rule was met, under a rule the buyer saw before it signed.

The case for keeping it simple

Seats are easy to budget, and a long billable-event definition adds friction to every sales conversation. PricingOS still calls seats fine, and simplest to sell, when usage is stable and the agent is one feature among many.

Woody goes further, as one CEO's opinion: "For everyone else, outcome-based pricing is, for all intents and purposes, a myth." He ties the exceptions to monopoly-like power or a ticket price high enough to justify the cost of measuring outcomes.

A procurement-minded commenter, u/highfivesalute, described the budgeting objection on October 15, 2026: "Procurement answer: boring old predictable pricing." The commenter preferred an annual amount finance could plan around. This is one practitioner report, not evidence of what all buyers prefer.

If agent work per seat is stable, seats may be the right unit, and you do not need a billable-event definition at all. Where you do write one, the buyer needs to see only the predicate sentence, the exclusions, and the way to contest a charge.

When outcome pricing is a poor fit

• Your system cannot observe the business result.
• Attribution is ambiguous because people and other tools contribute to the same result.
• The feedback loop is long, so the result appears weeks after the work and the invoice.
• Dispute cost is out of proportion to the value of each unit. In the model above, disputes cost $450.00 against $213.60 of inference.

In those cases, price a system-observed output, or a hybrid of a base fee plus an allowance or credits, with spend controls.

Publish the definition where buyers read it

A billable-event definition settles disputes only if the buyer can find the current version. Intercom's billing rule, for example, is a public help article. Put your success predicate, exclusions, and reversal window on the pricing page and in a help article, and change both in the same release that changes definition_version. A stale pricing page reopens every dispute the definition was meant to close, and AI assistants that answer "how does this product bill?" can draw on those same public pages.

Keeping those pages current is a documentation problem. EkLine's Docs Agent comments on pull requests when it detects documentation impact and can open a pull request with the documentation fix for your team to review.

Replay your traces before you set a price

1. Choose one task type with recent volume.
2. Write its billable-event definition, the predicate sentence and the nine answers, and map it to the record fields.
3. Pull recent task traces you already have access to under your current agreements.
4. Replay each trace through the edge-case table and count how many land in each row: clean billable, retry, duplicate, partial, handoff, reopened, delayed, disputed, and unclassifiable.
5. Decide what must change before you bill on this unit: a missing evidence field, a window that is too short, an unwritten exclusion, or a unit that should be an output.
6. Publish the definition on your pricing page and in a help article, and update both whenever definition_version changes.

If many traces land in the unclassifiable row, fix the definition before you choose the model and the price.

FAQ

How do you verify an AI outcome?

Check it against a written success predicate and a named evidence source inside an agreed observation window. Intercom counts a resolution when the customer confirms the answer or leaves without asking for more help. Zendesk has an LLM evaluate the conversation text. Record a timestamp, the actor, and an evidence link for each charge.

Is outcome-based pricing better than token-based pricing for AI agents?

Neither wins in general. Token pricing tracks your cost and leaves the buyer carrying retries and failed attempts. Outcome pricing moves that risk to you and works only when the outcome can be observed, attributed, and disputed under an agreed definition.

Who gets credit when a human finishes the conversation?

Decide in the contract and record the actor on every event. Intercom documents one split: a handoff that a customer-configured Procedure calls for is billable, and default escalations are not billed.

What's the hardest part of implementing outcome-based pricing?

Defining the event so it survives a dispute, then paying to run that definition. In this post's illustrative model, inference is about 14% of total cost, so model verification and disputes before you set a price.

Does outcome-based pricing always work better than usage-based or per-seat pricing?

No. PricingOS still calls seats fine, and simplest to sell, when usage is stable and the agent is one feature among many, and Metronome's Scott Woody argues outcome pricing works only in a very limited set of circumstances. Outcome units fit when completion is discrete and system-observable.

Here is the short answer. By our arithmetic, a 4-bit build needs about 14 to 17 GB for the weights plus the conversation cache, at 32K to 131K tokens of context. That is before runtime overhead, the operating system and your agent harness. A machine with 24 GB available to the model has room for it. On a 16 GB machine, almost nothing is left for the rest.

Only a test on your own hardware shows whether it runs your agent. Today's evidence is vendor-run benchmarks and launch-week anecdotes, and we found no independent reproduction. So the sensible verdict for now is to test it. Base the decision on four checks on the target machine: memory at your real context length and concurrency, runtime support, tool-call reliability, and completed tasks.

TL;DR

• Memory follows all 24.0B weights plus a cache that grows with context length and parallel sessions. The 2.45B active count does not shrink it.
• Our calculation: a 4-bit file plus a 16-bit cache comes to 14.27 to 17.18 GB for one session at 32,768 to 131,072 tokens of context, before overhead.
• Runtime builds matter. MLX, Ollama and llama.cpp each have a version or flag requirement for this model family.
• Tokens per second on a short prompt says little about finished tasks. A worksheet of real tool-using tasks on the target machine, with failure traces, tells you much more.
• Local inference keeps the model on the device. Retrieval, web search, MCP servers, other tools and harness telemetry are separate data paths.

What the model is

Liquid calls it "the largest and final model in the LFM2.5 family." The model card lists 24.0B total parameters and 2.45B active. The launch post rounds the active count to 2B. The context window is 131,072 tokens.

It is a mixture-of-experts model. Each MoE block holds 64 experts, and each token routes through 4 of them. The configuration also declares two dense layers. That keeps compute per token small, and Liquid credits the model's speed to that small active count. Thirty of its 40 layers are short convolution blocks and only 10 are attention layers. That split matters for memory, as the next section shows.

The active count describes compute, and memory is a separate bill. The next token can route to different experts, so runtimes normally keep every expert loaded. Plan memory for all 24B weights.

How much memory it needs (our calculation)

Memory has three parts, and each moves for a different reason. We calculated the figures below with Python from Liquid's published files and config. We did not measure them on hardware.

Weights and quantization

Quantization stores each weight in fewer bits, and it is the biggest lever you have. Liquid's official GGUF files are the format llama.cpp and Ollama load. They range from 13.6 GB at 4 bits to 48 GB at BF16, the full 16-bit precision. The MLX 4-bit files, for Apple's MLX runtime, are about the same size as the 4-bit GGUF. File size sets the floor for runtime memory. Everything else comes on top.

Cache, context length and parallel sessions

The KV cache is the memory that holds the conversation so far. It grows with every token in the context and with every session that runs in parallel. Only the 10 attention layers keep one, which is why this model's cache stays small. If all 40 layers were attention layers, the cache at full context would be four times as large.

Four parallel sessions at 32,768 tokens need the same cache as one session at 131,072 tokens. Count sessions as carefully as context length.

For implementers: the config.json gives 8 KV heads, a hidden size of 2048 and 32 attention heads. It lists no head dimension, so we assumed 64 (2048 divided by 32). At 16 bits per value, that is 2 (K and V) × 10 layers × 8 heads × 64 × 2 bytes, or 20,480 bytes per token.

Runtime overhead

We did not estimate runtime overhead or compute buffers. We also left out the operating system, other apps, the agent harness, the small fixed convolution state, and any embedding or tool server. Measure those on the target machine.

Assumptions for both tables: head dimension 64 (derived), decimal GB, one session, and about 1 byte per value for the 8-bit cache. They exclude runtime overhead, the operating system and the harness. We calculated these figures and did not measure them.

Read the tables this way: quantization moves the total by tens of gigabytes, and context length moves it by about 2 GB at 4 bits.

A third-party calculator on llmrun.dev adds an overhead term and lands close to our figures, at 17.4 GB for Q4_K_M at full context. Neither the launch post, the model card nor the docs page gives a minimum RAM or VRAM figure. Liquid's agent-harness guide suggests that memory-constrained machines "serve a smaller window such as 32K tokens, which is usually plenty for a single agent task." Its worked example uses a smaller LFM2.5 model.

Runtime support and version gates

Liquid's docs page for the model covers Transformers, vLLM, SGLang, llama.cpp and MLX. The model card also lists LM Studio. Neither names Ollama, but Liquid has a separate Ollama page with a version note for its mixture-of-experts models.

The Ollama page says "To run any LFM MoE model you specifically need v0.17.1-rc0 or later." The models it names as affected are older LFM2 MoE models, and the page is undated. The gate covers any LFM MoE model, and this model's config declares lfm2_moe. Plan for the gate, and confirm it on the exact build your customers will install.

A runtime can also break the agent loop by itself. One Hacker News commenter ran Ollama on an AMD Ryzen 7 8745HS and found it was not caching the prompt, so every turn reprocessed the whole prompt. They concluded it would not work in an agentic loop in that state. Record the runtime build as carefully as the model file.

How tool calls work and what your parser must handle

The model card says that by default "LFM2.5 writes Pythonic function calls (a Python list between <|tool_call_start|> and <|tool_call_end|> special tokens)." You can ask for JSON function calls in the system prompt instead. The chat template places tool definitions in the system prompt and opens every assistant turn with a <think> block. The card says the model always reasons before it answers. Raw generated output includes a reasoning block; a serving layer may parse or hide it before the harness receives the response.

Liquid's llama.cpp function-calling guide says --jinja is mandatory, because it renders tool definitions through the chat template and turns on tool-call parsing. Its examples use a smaller LFM2.5 model. From the same guide, your harness needs to handle:

• several tool calls in one turn, because "LFM2.5 may request several calls at once"
• unknown tool names and malformed arguments, returned to the model as an error in the tool result instead of crashing
• a limit on the number of steps
• a short tool list, since every tool adds prompt tokens on every turn. The guide prefers 3 to 8 well-described tools over 30 vague ones.

Six things "good enough" can mean

"Good enough for agents" bundles six separate properties. Launch-week reports mostly measure token throughput, which is the property least tied to finished tasks.

The case for adopting now, and its limits

The case for adopting has real evidence behind it. Liquid reports the model "decodes 184 tokens/s on an Apple M5 Max and 114 tokens/s on an AMD Ryzen AI Max+ 395."

On agentic benchmarks that Liquid ran itself, it scores 27.69 on AutomationBench, against 8.72 for Qwen3.6-35B-A3B. The card recommends it for agents that reason, call tools and run on the device. Hacker News commenters also report high local decode speeds, such as 200 to 320 tokens per second on a 5070 Ti in llama.cpp.

Each piece has a limit. Liquid's model card discloses the laptop test settings: Pipette benchmarking infrastructure, llama.cpp, Q4_K_M quantization and a 4K-token input context. Its separate GPU throughput test used 1,024 input tokens and up to 256 output tokens, far shorter than an agent's accumulated context. The benchmarks are vendor-run, and we found no independent reproduction. The same post says larger models keep an edge on knowledge-heavy and coding benchmarks. The card calls the model "not recommended for knowledge-heavy tasks on its own" and suggests pairing it with retrieval or web search. User reports split between speed and reliability. This evidence puts the model on your test list. Shipping it needs results from your own tasks.

The evaluation worksheet

Run each task several times on the target machine and fill in every column. Hold these constant across comparisons: tasks, prompts, tool definitions, runtime build, quantization, context setting, sampling settings and machine. If one of them changes between runs, you are comparing setups, not models.

The rows below are illustrative examples to adapt to your product. They are not results. The measured columns stay empty until you run the tasks.

Record peak memory with your harness and tool servers running, because that whole stack has to fit. Keep failure traces whole. They show whether the model, the parser or the runtime broke.

Local inference does not make the agent local

Running the weights on a laptop or a customer server keeps inference there, and only inference. The card suggests pairing the model with retrieval or web search, and each of those is a data path. So is every MCP server and tool the harness calls, and any telemetry the harness sends. We did not review any harness's telemetry, so check each harness's own documentation.

If the agent calls hosted services, it still needs credentials for them, and the questions in OAuth for AI agents still apply. One Hugging Face user reported the model insisting it was a cloud model while it used local tools, so do not rely on the model to tell you where your data goes.

License: read the revenue threshold

The card lists the license as lfm1.0, the LFM Open License v1.0. It defines "Threshold" as "annual revenue of 10 million United States dollars ($10,000,000) or more." Commercial use by a legal entity above that threshold is not licensed under the agreement. The page shows no effective date.

One Hacker News commenter called it "a questionably-enforceable non-commercial use clause," which does not match the revenue-threshold wording. For a customer-hosted option, ask who the licensee is and whose revenue counts. Have counsel read the license before you ship. This is not legal advice.

What early reports cannot establish

Launch-week reports show which questions to test. They cannot show how often a problem occurs or how the model performs on your workload.

• Speed reports come from single users who did not state the quantization. One reported 94.74 tokens per second on an M4 Pro Mac mini with 64 GB, from a single prompt.
• Reliability reports mix runtimes, and none states the quantization. One user on a 32 GB M5 MacBook Pro reported repetitive loops around 32,000 tokens while using about 16 GB of RAM. The runtime and quantization were not stated.
• Some reports use hosted endpoints. The stuck tool calls report ran through OpenRouter in OpenCode, so it says nothing about a local quantization.
• The laptop speed figures use disclosed settings: llama.cpp, Q4_K_M and a 4K-token input context. They do not measure completed agent tasks, and we found no independent reproduction of the benchmarks.

For another model launch read with the same caution, see Gemini 4 Argon: what it does and where it falls short. Treat each early report as a row to add to your worksheet.

Adopt, test or defer

Set your pass rate and memory headroom before you run the worksheet. Then apply these criteria.

If your local option runs agent loops with a short tool list on a machine with 24 GB or more available to the model, put the model in your test queue now. If it mainly needs coding or broad knowledge without retrieval, keep the model you run today until independent results appear.

FAQ

What hardware do you need to run LFM2.5-24B-A2B locally?

Neither the launch post, the model card nor the docs page gives a minimum. By our calculation, a 4-bit GGUF file plus a 16-bit cache needs 14.27 to 17.18 GB for one session at 32,768 to 131,072 tokens of context. That excludes runtime overhead, the operating system and your harness. It points to a machine with about 24 GB available to the model at 4 bits. The BF16 weights alone are 48 GB.

Is LFM2.5-24B-A2B worth running locally for agent workloads?

It is worth testing. Liquid's speed figures and vendor-run agentic benchmarks are promising. We found no independent reproduction, and early user reports split between speed and reliability. Run a representative tool-using task suite on your target machine before you decide.

Why does LFM2.5-24B-A2B say it is a cloud model?

One Hugging Face user reported the model insisting it was a cloud model while it used local tools. The same user saw repetitive loops around 32,000 tokens. It is a single report, with runtime and quantization unstated. What the model says about itself does not tell you where your data goes.

How does LFM2.5-24B-A2B tool calling work?

By default the model writes Pythonic function calls as a Python list between special tokens. You can ask for JSON in the system prompt instead. In llama.cpp, Liquid says the --jinja flag is mandatory. Your harness should handle several calls in one turn and return an error for malformed arguments.

Your next step

Before you choose a deployment, write five or six representative tool-using tasks from your product. Run them on the target machine with a pinned runtime build, and record the failure traces. Liquid's guide to running local agents with LFMs shows how to point a harness at a local endpoint.