GUIDE TO DESIGNING A KNOWLEDGE BASE

KB DESIGN SPEC — Guide to Designing a Knowledge Base for an AI Agent

Version: 1.0
Date: 2026-09-21
Intended for: AI or people who design, evaluate and improve the KB for new or operating agents.

0. Instructions for using this document

When you are given this document together with a request to build an agent, use it as a guide for designing the KB. Do not copy this entire spec into the target agent's KB. Only convert the principles and structures that fit into a concrete design.

The goal is to create a KB that helps the agent solve tasks better than its base capability under the same conditions, while remaining adaptable, verifiable and modifiable. Do not optimize for length, number of rules, rigidity or resemblance to a program.

Before designing, determine the task, data, tools and operating environment. If information is insufficient, state your assumptions and continue with the parts that do not depend on it; only ask about gaps that could significantly change the architecture or correctness. Do not invent professional thresholds or policies and present them as fact.

Keywords in this spec:

• MANDATORY: a design requirement that must be met, or must be explicitly recorded as unmet along with its consequences.
• SHOULD: the default recommendation; may be adjusted when there is a reason that fits the task.
• MAY: an option depending on context.

These keywords express the level of requirement. Writing them in a text does not by itself create a technical enforcement mechanism.

1. Goals and boundaries of the KB

A KB is a reusable body of knowledge that supports understanding, deciding and carrying out tasks in a domain. A KB may contain facts, conventions, professional models, policies, heuristics, protocols, examples and evaluation criteria.

Distinguish the components by responsibility:

Component	Main responsibility
System/developer instructions	Role, goals, authority, general requirements and how to use the KB
KB	Domain knowledge, conventions, standards, policies and how to solve classes of tasks
Tool layer	Tool contracts, data access, performing actions, returning results/errors
Case input	The request and data specific to the current task
Operational state/memory	What is known, what has been done, what is missing, status and intermediate results
Runtime/orchestrator, if any	Orchestrates flow, stores state, validates schemas, enforces permissions and blocks actions

This is a division of responsibility; it does not require each row to be a separate service or file. The KB can also be updated frequently; each run needs to know the version or effective date of important content.

MANDATORY: specify how the KB is brought into context and the authority of each type of content. Retrieved documents or input text do not become authoritative instructions just because they contain imperative sentences. Separate policy content approved by the agent's owner from external documents used as evidence.

2. Design outcomes to achieve

A good KB design should make clear:

1. Which task the agent does and what success looks like.
2. Where the KB adds capability or reliability.
3. Which knowledge is activated in which situation.
4. What is mandatory and what is only guidance.
5. What the model is free to decide.
6. How results or actions are checked.
7. How to handle missing data, contradictions or tool failures.
8. How changes to the KB will be evaluated and traced.

Do not claim the KB is better than the baseline without evaluation results. If it was only just designed, state that the benefits are hypotheses that need testing.

3. Choosing content: keep what creates value

Prioritize putting into the KB:

• Knowledge specific to the organization, product, market or process.
• Standard definitions, units, conventions and authoritative sources that the task needs to be consistent.
• Right/wrong standards, selection criteria and specific trade-offs.
• Points the model commonly misunderstands, misses or coordinates wrongly.
• Business constraints, authority and completion conditions.
• Professional heuristics with a clear scope of application and limits.
• Examples that distinguish cases that look alike on the surface but require different decisions.

Limit:

• Background-knowledge lectures that change neither behavior nor quality.
• General advice such as “analyze deeply”, “be careful”, “be professional” without stating the expected manifestation.
• Many phrasings repeating one requirement but lacking an official place of definition.
• Processes that anticipate every case before the real task is understood.

For each important piece of content, answer: Which decision does it help the agent make better, which error does it avoid, or which obligation does it meet?

Do not remove content just because the model may already know it. The model knowing a concept does not mean it will use the convention, source or version the task requires.

Do not delete a rule just because a few cases don't trigger it. Before deleting, check test coverage and the severity of the situation the rule protects against.

4. Content layers and how to represent them

Layer	Content	Usually suitable representation
Domain knowledge	Concepts, relationships, facts, conventions	Short text, tables, schemas, sources
Decision criteria	When to choose which option	Condition tables, diagnostic questions, decision graphs
Policy	What is mandatory, limits, authority	Rules with scope, conditions and consequences
Heuristic	Approaches that are often useful	Principles, reasons, exceptions, counterexamples
Protocol	A chain of work with conditions and feedback	Steps, state, state transitions, recovery
Examples	How to apply and distinguish situations	Correct cases, wrong cases, edge cases
Evaluation	What an acceptable output looks like	Rubrics, checks, validators, test cases

Layers may sit in the same module if they are closely related. The goal is to distinguish meaning, not to split into as many files as possible.

Use text when nuance and reasons matter; use tables/schemas when fields repeat; use code when you need computation or enforcement that a machine can perform.

Do not build a DSL or mini-language by default. Only consider it when there is a stable repeating structure, clear semantics and a concrete benefit in parsing, checking, reuse or execution.

5. Choose a control level that fits each task

The types below are architectural choices, not a quality ranking scale:

Type	Suitable when	Limits to be aware of
Reference KB	You need to find the right fact, definition or source	Correct retrieval does not guarantee correct use
Reasoning framework	Open tasks with a lot of judgment	May miss criteria without checks
Conditional protocol	Work with repeated steps and decisions	May be rigid or wrong outside its designed scope
Executable spec with checks	You need clear state, data contracts and step-transition conditions	The level of assurance depends on the actual validator/runtime

An agent often combines several types: looking up facts, using a framework to analyze, using a protocol to fetch data, using a runtime to control actions.

Do not assume “the tighter the better”. When a protocol lowers quality, it can be fixed, narrowed or dropped. Do not assume you must keep tightening to get past a “valley”.

6. Balancing constraints and reasoning ability

Classify requirements:

• Mandatory requirement: a condition that must be met for an output or action to be accepted.
• Constraint: a limit on valid options; can be quantitative or qualitative.
• Heuristic: an approach that is often useful and may be changed when the situation doesn't fit.
• Preference: a priority among options that are all valid.

A gate is a decision point that allows/blocks a step based on a requirement. MANDATORY: state clearly whether the gate is self-checked by the model or checked by an external system.

Prefer specifying:

• Which variables need to be considered and why.
• What evidence is sufficient for which kind of conclusion.
• The criteria for choosing or eliminating options.
• The invariants to maintain throughout the task.
• The conditions to stop, change direction or ask for more.

Let the model choose strategy, synthesize, explain and adapt within the valid scope. Only fix the order when there is a dependency, a business requirement or evidence that the order is useful.

Do not force the model to write out its entire chain of thought. Ask for useful intermediate products: facts, assumptions, calculations, choices, brief reasons and check results.

Edge cases are not automatically handed over entirely to free reasoning. Determine the severity of consequences to decide whether it needs a rule, a check, asking the user, or letting the model handle it.

7. Designing rules that can be operated

An operable rule needs a clear scope of application, trigger condition and consequence. Not every rule has to contain a number or a prohibited behavior.

Minimal template:

ID:
Purpose:
Scope / when it applies:
Trigger condition:
Required behavior or state change:
How to check:
Exceptions / when it does not apply:
Source or reason for existing:

For important rules, add:

Inputs and dependencies:
Level: mandatory / heuristic / preference
Priority and conflict handling:
Enforcement level: model self-check / external check / runtime block
If the condition cannot be determined:
If execution fails:
Related test cases:
Version / effective date:

Weak example:

Check the data carefully and do not conclude without basis.

More concrete example:

Before comparing two metrics, confirm that their definitions, measurement periods and units are compatible. If they are not compatible, normalize when there is enough information; if they cannot yet be normalized, state the incompatibility and do not present the difference as an equivalent comparison.

The example above still needs an external check if the system wants to firmly prevent an invalid comparison from being output.

Every professional threshold must have a basis, scope and unit. If it is a proposed parameter that has not been approved or verified, label it clearly and do not use it as official policy.

8. Modules, decision authority and conflicts

Each module needs to define its responsibility, inputs, outputs and dependencies. Do not let a module quietly change variables outside its responsibility.

For each important decision variable, specify where the responsibility lies for finalizing the value. Multiple modules may provide evidence or proposals, but the way they are merged must be clear.

When multiple rules apply at once:

1. Determine whether they really conflict or are acting on different aspects.
2. Apply a priority order by authority, scope or effective date that has been defined.
3. If aggregation is needed, define an aggregation operation that fits the meaning of the variable.
4. If it cannot be resolved validly, keep the conflict state and handle it per policy.

Do not use min, max, worst_of or “strictest” as the default for every variable. These operations are only valid when the value domain and business meaning allow them.

Avoid double counting: multiple rules may reflect the same underlying risk cause. Before accumulating consequences, check for dependence and overlap.

9. State, dependencies and execution loops

Only design a state machine when the task genuinely needs state tracking. Do not add state just to make the document look rigorous.

If state is needed, define:

• The name and meaning of each field.
• The source that creates it, where it is stored and where it may be updated.
• The valid values; with an indeterminate or not-applicable state when needed.
• The conditions for state transitions and the corresponding actions.
• The conditions for completion, failure and waiting for information.

A closed enum suits control states with a finite value domain. Do not force every professional judgment into a closed label. If you need UNKNOWN, CONFLICT or NOT_APPLICABLE, define the accompanying behavior instead of forcing PASS/FAIL.

When facts change, mark and re-evaluate the dependent results. Only update the affected parts. Re-evaluation may give the same result; for important conclusions, keep a brief basis.

If there are loops, define the stop condition, the resource budget and the behavior when there is no progress. If a tool has side effects, handle the possibility that the action partially succeeded before retrying to avoid duplicate execution.

10. Evidence, assumptions and uncertainty

Minimum distinctions:

Type	How to use
Fact from a source	Record the source, time and limits; a source can still be wrong
Derived or calculated result	Tie it to the inputs, formula/method and assumptions
Estimate	State the basis and appropriate uncertainty; use ranges/scenarios when needed
Assumption	Used for conditional analysis; must not be turned into a verified fact
Unverified	Only used within the scope policy allows; not quietly promoted to firm evidence

There is no need to label every sentence. Prioritize claims that affect decisions, figures and key conclusions.

Distinguish the causes of missing data: not yet searched, searched but not found, the source doesn't publish it, inaccessible, extraction failed, or not applicable. Each state may require a different action.

Not finding something only proves the search result within the scope checked. Only conclude that it does not exist when there is a basis for the completeness of the scope and sources.

When sources contradict, consider authority, definition, timing, scope and quality. Do not assume the newer source is always more correct.

When a result deviates far from a reference point, check the data, units, assumptions, method and the reference point itself. Do not force the result toward consensus, and do not assume you are right just because you followed the protocol.

Do not use a confidence score as a probability of being correct without calibration. Uncertainty can be described through evidence quality, degree of missing data, sensitivity and open assumptions.

11. Retrieval and context budget

SHOULD organize as a short core plus modules that are loaded only when relevant:

• Core: scope, general principles, essential terminology, orchestration and module references.
• Task modules: the knowledge, rules, examples and check criteria of one class of task.
• Reference sources: long documents, data and evidence to look up.

Each module has a “use when” description, scope and dependencies. When retrieving a rule, bring along its essential conditions and exceptions; do not cut them off from the rule.

Avoid retrieving by keyword similarity alone. Retrieval needs to consider the current task, the decision to be made and the work state.

Prefer clear meaning before compressing syntax. Symbols are only useful when the model understands them consistently and maintainers can read them. Do not rely on placing a rule at the start/end of a document as a guarantee of compliance.

Keep one official place of definition for each policy. Other places refer to it; if you restate it to increase usability, avoid creating copies that drift out of version.

Test both finding the right module and applying the content correctly. These two kinds of errors need two different fixes.

12. Distinguishing guidance, checking and enforcement

MANDATORY: describe the actual capability accurately:

Level	Mechanism	What can be claimed
Guidance	The model reads and follows on its own	The requested behavior; compliance rate needs to be measured
Checking	A validator or evaluator checks the product	Some violations can be detected within the scope of the check
Enforcement	The runtime controls step transitions or the performing of actions	Violations of conditions that have been checked can be blocked at that point

Schema validation checks shape and some data constraints; it does not verify the truth of the content by itself. A model self-reporting PASS is not equivalent to independent evidence.

For each key requirement, record:

Requirement → required evidence → check → where it is executed → handling when not met

If the environment only allows attaching a KB file to the chat, design a clear protocol, intermediate products and suitable self-check steps; be open about the limits of enforcement. Do not promise that a text file has become a deterministic engine.

If there is a runtime, prefer to hand deterministic computation, validation, tool permissions and durable state to components that can be machine-checked. The model handles the parts that need semantic understanding and judgment, with checks suited to the severity of consequences.

Useful logs include the KB version, relevant inputs, sources, tool results, decisions, rules applied, state and check results. Explanations generated by the model are only part of the log, not proof that every step actually happened.

13. Process for designing a KB for a new agent

Step 1 — Determine the task and environment

Record the main task, users, input/output types, tools, state-storage capability, how the KB is loaded, constraints and success criteria. Determine which parts must be absolutely correct per regulation and which accept judgment.

Step 2 — Establish a baseline

Run the agent with good task instructions and equivalent tool/data conditions, without the KB part to be evaluated. Zero-shot can be an additional baseline, not the only yardstick.

Choose common cases, hard cases and high-consequence cases. Save the model configuration, prompt, tools, data and output for a fair comparison.

Step 3 — Find the gaps and causes

Classify errors: missing knowledge, wrong retrieval, wrong/missing data, wrong application, rule conflict, lost state, wrong calculation, tool error or unclear acceptance criteria.

Do not assume every error needs a new rule. Some errors need a fix to the source, tool, schema, routing or input requirements.

Step 4 — Design the minimum addition

Choose the content and form suited to each gap. Combine design from goals/risks with learning from real errors. One generalized principle can handle many failure cases.

State clearly the areas where the model decides on its own, the mandatory policy and the limits of application. Do not formalize the entire domain before there is evidence it is needed.

Step 5 — Design the checks at the same time

For each important rule, create a triggering case, a case that should not trigger, a case with insufficient data to decide, and a conflict case if relevant. Define expectations with observable criteria.

Step 6 — Test and check for regression

Compare with the baseline and the previous version. Evaluate both the benefits and the new errors. Use cases not used for writing the rule, to avoid optimizing only for familiar examples.

Step 7 — Package and operate

Set an ID/version, record sources, dependencies and verification status. Provide guidance for loading, updating, observing errors and rollback. Keep the limitations that remain unresolved.

14. Evaluation and acceptance

Choose metrics according to the task; do not use one checklist to award yourself a “level”.

Groups of metrics may include:

• Correctness and usefulness of results.
• Task completion rate.
• Rate of serious errors or policy violations.
• Rate of unnecessary refusals/follow-up questions.
• Consistency on important criteria; identical wording is not required.
• Ability to handle missing data, contradictions and tool incidents.
• Token cost, number of tool calls, time and maintenance effort.

SHOULD use a combination of:

• Cases representative of real work.
• Edge, adversarial and conflict cases.
• Trying different phrasings of the input or adding irrelevant information.
• Repeated runs when you need to assess variability.
• Turning off each module to check its contribution.
• Evaluating on new cases not used to fix the KB.

Do not treat model self-grading as the only source of evaluation for key requirements. Use deterministic checks, reference data or qualified people where appropriate.

Keep the old baseline for historical comparison, but rerun it when the model, tools or environment change. Do not treat baseline output as the correct answer.

If tests cannot yet be run, hand over the test plan and the status NOT YET VERIFIED; do not invent evaluation figures.

15. Mistakes to actively avoid

Mistake	How to fix
Stuffing in background knowledge	Keep the parts with a concrete effect or that need standardizing
Deleting every definition because the model already knows it	Keep the conventions and standard meanings of the task
Using an output template in place of analysis quality	Define content criteria and evidence
Prescribing every reasoning step	Keep the decision points and necessary constraints, leave the strategy open
A framework that is too generic	Add distinguishing criteria, conditions of application and edge examples
The more errors, the more rules	Find the root cause and merge rules with the same purpose
Treating MUST or YAML as enforcement	State where the actual check and blocking happen
Specific thresholds with no basis	Attach a source, scope or a proposed-parameter label
Multiple modules each editing the same result	Designate where it is finalized and how it is merged
Always picking the strictest result	Use logic that fits the business meaning
Forcing every case to a full conclusion	Allow partial, uncertain or insufficient-data results
Treating self-reported checks as passed	Require a check product and observable evidence
A more complex protocol that is not better	Measure against the baseline, narrow or remove it if needed
Patching exactly the old case	Add new cases and check generalization

16. Handover contract for the AI designing the KB

When given a design task under this spec, hand over the following parts with a level of detail proportional to the scale:

1. Design summary: task, environment, assumptions, success criteria and KB scope.
2. Responsibility map: which content belongs to the KB, system instructions, tools, state or runtime; avoid asking a component to do what it cannot do.
3. KB architecture: modules, purpose, when loaded and dependency relationships. Explain the important choices.
4. Usable KB content: write the concrete modules, not just section names or a generic outline. If professional sources are missing, mark the parts to be supplemented instead of inventing them.
5. Short integration guide: how the agent selects and uses modules, handles missing information, checks results and records state when needed.
6. Table of key requirements: how to check, the actual enforcement level and handling when not met.
7. Acceptance case set: the input or case description, expected behavior, errors to detect and pass criteria. A fixed sample answer is not needed for open tasks.
8. Status report: what has been tested, what has not, limitations and next steps.

For a simple agent, the parts above can be combined in a single file. For a large agent, split by modules with clear responsibility; do not create many files just to satisfy appearances.

Do not claim the design has become an execution engine because there are enums, state or gates in the text. Only describe what the system actually supports.

17. Starter template for the person assigning the task

You can attach this file together with the request below:

Please design the KB for the agent per KB_DESIGN_SPEC.md.

What the agent needs to do:
Users and context:
Typical input:
Desired output:
Tools and data available:
Deployment: [prompt + KB file only / with retrieval / with runtime / not yet clear]
Constraints or unacceptable errors:
Existing professional documents and examples:

Please determine a suitable architecture, write a usable KB and an acceptance case set.
If information is missing, state assumptions and ask about the points that change the design.
Do not set professional policies or thresholds as fact on your own.
Do not add complexity without showing what problem it solves.
Clearly distinguish content that guides the model from actual checking/enforcement.

18. Closing principles

A mature KB is one that contributes clearly to the work, is used at the right time, retains the important requirements and allows errors to be detected and fixed. Increase maturity through operational evidence and maintainability; do not assume it means more content, more steps or more constraints.