Preface

Artificial intelligence has spent years trying to become capable enough to be useful. That effort has paid off. A model can now write working software, translate an unfamiliar document, examine an image, explain a technical argument and help a researcher find an unexpected connection. It can also be wrong in ways that are difficult to spot. Both facts matter.

When a technology crosses a threshold of usefulness, its surrounding arrangements become visible. A competent engineer needs a repository, tests, permissions, colleagues and a release process. A competent doctor needs records, instruments, other specialists and the authority to order some interventions but not others. We do not mistake either person for the whole system. Yet discussions of artificial intelligence still often treat the model as if it were the entire unit of analysis.

That was understandable when the model could do so little. Improving it was the obvious task. It remains a serious task. But a model that can do many things presents a different problem: deciding which of those things to do, with what information, under whose authority, and with what means of correction. A system can fail even when its central model is excellent. It can spend an hour reasoning about a trivial request. It can use a reliable source for the wrong question. It can cite a document it never checked. It can execute a valid instruction from someone who had no right to give it. It can quietly become the place where an organisation’s policies are interpreted without that organisation noticing.

The architecture around the model determines whether these are isolated mistakes or normal operating conditions. Architecture here means more than software components. It includes the allocation of computation, the representation of evidence, the design of permissions, the placement of human judgement, and the institution’s ability to inspect and change the system. It is the arrangement through which capability becomes useful intelligence.

We have written this book because that arrangement is becoming a research subject in its own right. Some parts already have strong experimental foundations. Other parts remain proposals, and we will say so. The aim is to make the problem precise enough to investigate, build and contest. We are interested in systems that can reason well, but equally in systems that can recognise a doubtful premise, stop before an unauthorised action and leave behind a record that someone else can examine.

Threadneedle Labs is one place where these questions are being pursued. The book does not depend on any particular product or unpublished result. It publishes the intellectual territory and leaves the machinery of specific experiments to the work that follows.

The argument travels from model capability to intelligence systems, from systems to knowledge, from knowledge to agency, and from agency to the human and institutional structures in which it operates. That sequence matters. Authority cannot be designed well if we have not distinguished an answer from an action. Governance cannot be designed well if evidence disappears before the decision is made. An institution cannot retain control through a declaration of principle alone; control must exist in the architecture it uses.

The book also argues against a convenient form of technological fatalism. We are sometimes told that increasingly capable systems will decide how they are used, as though the design of permissions, evidence and institutions were an afterthought. These are choices. Some can be made well, some badly, and many can be tested. The difficult work is to make them explicit before fluent output obscures them.

The reader will find strong propositions here, but not a claim that one architecture will fit every domain. A scientific assistant, a warehouse controller and a public agency operate under different evidence and authority conditions. The framework is meant to help us ask better design and research questions in each setting. Where the evidence is thin, the book names the uncertainty rather than borrowing confidence from the speed of the technology.

The work is interdisciplinary because the failures are. Engineers can build a permission boundary, but someone must define the authority it enforces. Researchers can measure a gain, but someone must decide whether the measured outcome represents the real task. Leaders can announce a principle, but a system must make it operative. The book is addressed to readers willing to work across those lines.

There will be better models. We hope so. Their arrival will make the questions in these pages more pressing, not less.