37Thirty Seven
Skip to the essay
A teenage pianist studies a passage with an older teacher in a quiet music room, with a cello waiting beside them.

A field guide to cognitive travel

How Minds Travel

How culture, education, mathematics, and AI open paths through mind-space

A field guide to the paths that turn unfamiliar patterns into human capability—and to the tests, translations, and shared checks that keep those paths open to correction.

A mind travels when a pattern that was once unreachable becomes something a person can notice, test, and continue.

Begin with a song you thought you knew

The sound stayed the same. Your listening changed.

Return to a song after learning an instrument. The bass motion, a held breath before a cadence, or the drummer's tiny delay may step out of a stream you thought you knew. The recording did not change. Practice, names, examples, and correction changed what your attention could reach.

Education is more than receiving information. A score reorganizes hearing; a diagram opens a proof; a coordinate system joins observations that would overwhelm memory. Culture places marks, tools, actions, and people between a learner and a difficult pattern. Practice turns that support into capability.

AI makes this old story newly strange. A system may search huge problem-spaces and return moves, programs, or patterns that surprise experts. But a machine discovery does not automatically become a human path. A correct answer can still leave us unable to see why, catch failure, or continue.

This essay asks what happens between discovery and learning. How does an unfamiliar result become examples, notation, simulation, or practice a person can enter? Where does translation fail? Who controls the bridge—and does it make the traveler more capable or more dependent?

By the end, you will have a practical test for any discovery bridge: whether people can independently notice relevant structure, predict new cases, test errors, and continue from what was found.

After working with the representation, can people independently notice relevant structure, predict new cases, test errors, and continue from the discovery?

Three spaces, three different claimsMake the map precise
Problem-space
For a specified task model, the candidate states, allowable actions or transitions, goals or evaluation criteria, and constraints considered in exploration or solution.The useful problem-space depends on how the task is framed; it is not the problem as viewed from nowhere.
Representation-space
For a specified notation, model, interface, or language, the possible encodings it can express and the operations or inferences defined over them.Equivalent content can be easy to transform in one representation and difficult to notice in another.
Mind-space
A conceptual, multidimensional family of possible cognitive organizations and capability profiles across perception, memory, learning, planning, embodiment, value, and coordination.It has no known universal metric or complete map. The term supports careful comparison; it does not locate thought in a literal hidden geometry.

Chapter 1 · Begin with music

Lived learning

You have traveled before

Can a familiar mind arrive somewhere it could not reach before?

A song plays through a kitchen speaker. A beginner hears one bright stream. Her brother, a practiced bassist, hears the bass enter late, the drummer lean behind the beat, and the chord that will make the chorus feel like home. The same sound offers them different paths.

Practice did not install a new ear. It changed what he can reliably notice, compare, and anticipate. Names give handles to patterns that once slipped past; clapping, notation, teachers, and correction turn those handles into habits. Eventually the distinction arrives inside the hearing.

This is a small journey through mind-space: a region of attention and action became reachable. Reading makes ink into argument; algebra lets one mark range over numbers; a map lets a child rehearse an unseen trip. Education can build routes for thought, not merely deliver answers. The travel joins changing attention, memory, movement, language, tools, and other people.

A score is not sound, and a lesson is not skill. Each can still form part of a path. It works when a learner can reenter it, predict, hear mistakes, vary the example, and continue without the teacher choosing every note.

The sound remains; example, notation, practice, and feedback make more of its structure usable.

Field study

Listen twice

Play a short song once for enjoyment, then replay it while following only the lowest instrument.

  1. The sound entering the room is nearly the same, but the task changes which events become prominent.
  2. Naming the bass line makes it easier to return attention after it wanders.
  3. After several songs, the listener may begin predicting entrances without a prompt.

What the comparison revealsA cue can open a path, but independent prediction is better evidence that the path is becoming the learner's own.

Carry this forward

You already know what cognitive travel feels like: the world stays rich, while practice makes more of its structure available.

Pause and predict

Two beginners study the same four-bar rhythm. One reads its notation ten times. The other reads it, claps it, hears a changed version, and must identify the change.

Tomorrow, who is more likely to recognize the rhythm when it appears at a new tempo?Choose one learner and name the experience you think will transfer.
Reveal what follows
What follows
The second learner has the stronger path: several coordinated representations and error checks support recognition beyond the original page.
Why
Reading, action, listening, contrast, and feedback make the relevant relation available in more than one situation.
The transfer test
Change the instrument and tempo, then ask the learner to clap the underlying pattern and explain what stayed constant.
Where it stops
Varied practice is not automatically better; overload, poor feedback, or irrelevant variation can hide the structure instead of revealing it.
Go deeperTraining changes a developing listening system

The music example is not only a metaphor, although its scientific scope must be stated carefully.

Tierney, Krizman, and Kraus followed adolescents participating in school music or physical-training programs. Their results associated music training begun in high school with changes in neural encoding of speech sounds and improvements in some language-related measures. The study supports experience-dependent auditory development during adolescence; it does not show that one musical curriculum universally raises intelligence.

A cognitive path can combine neural adaptation, learned attention, motor practice, vocabulary, notation, social expectation, and access to instruments. Calling the result “in the brain” or “in culture” alone misses the coupled process. The durable achievement is a new ability to discriminate and act, supported by many layers at once.

Perceptual learning
Experience-dependent improvement in detecting, discriminating, or organizing sensory information.
Transfer
Successful use of learning in a case that differs in a relevant way from the practiced examples.
Make it preciseA path is a change in reachable performance

We do not need to claim that learning moves a point through a literal geometric space.

Reachable performance
A discrimination, prediction, explanation, or action a learner can produce under stated conditions after some sequence of practice and feedback.
Independent continuation
Extending a pattern, testing a variation, or correcting an error without being given the next answer.

Evidence of changed performance does not identify one unique inner mechanism, and a group-level training result does not predict every learner. The path language summarizes functional change; it is not a universal ruler for minds.

SourcesMusic Training Alters the Course of Adolescent Auditory Development

Chapter 2 · Three spaces

Conceptual map

Map, path, and bridge

What exactly is traveling, and where?

A maze can be printed, built from hedges, or listed as intersections. On paper you can trace backward; in hedges you must remember; in a list you may find a graph algorithm. Treating one drawing as the maze hides the work of representation.

A problem-space contains the states, actions, goals, and constraints selected for a task. A representation-space contains the forms and operations allowed by a notation, model, or interface. Mind-space is a conceptual atlas of possible cognitive organizations and capabilities: ways of sensing, remembering, learning, valuing, coordinating, and acting.

A map selects landmarks. A path is a learnable sequence between capabilities. A bridge links unlike starting points through a workable representation. A geometry diagram can bridge teacher and student if both can manipulate it, connect it to definitions, and check what follows. Different representations can express related content while inviting different inferences.

None has one complete coordinate system. Task framing changes the problem-space; notation reshapes representation-space; mind-space has no known universal distance or complete map. These are local comparison tools, not a claim that thought occupies a measurable hidden country.

A machine may cross the first box without changing a person's capacity in the third.

Field study

The rearranged equation

Compare “three more than twice a number is eleven” with 2x + 3 = 11 and a balance drawing carrying two equal boxes and three counters.

  1. The sentence preserves everyday meaning but can burden working memory.
  2. The equation makes legal symbol transformations compact.
  3. The balance makes doing the same operation to both sides physically visible.

What the comparison revealsEquivalent content can invite different inferences. A bridge often succeeds by coordinating representations rather than declaring one form the real thought.

Carry this forward

A useful bridge does not erase the shores. It gives different travelers operations they can perform and consequences they can check.

Pause and predict

A student can recite the rule “do the same thing to both sides” but freezes at 3x − 5 = 16. You may offer either the completed answer x = 7 or a balance model she can alter.

Which aid is more likely to help her solve 4x − 9 = 19 tomorrow?Predict what she will be able to do after the aid is removed.
Reveal what follows
What follows
The manipulable balance is more likely to build a transferable route, provided it is explicitly connected back to symbolic operations.
Why
It exposes why equal changes preserve equality and lets the learner rehearse the transformation.
The transfer test
Remove the balance, change the numbers and signs, and ask the student to predict an illegal move before calculating.
Where it stops
A picture can become another ritual. Without variation and connection to symbols, the learner may depend on the picture rather than understand equality.
Go deeperRepresentations participate in reasoning

A representation is not merely a package delivered to a mind that does all the real work elsewhere.

Kirsh and Maglio distinguished pragmatic actions, which directly advance an external goal, from epistemic actions, which change the world to make information easier to obtain or computation easier to perform. Skilled Tetris players rotate pieces partly to see possibilities rather than to place them immediately. The visible transformation is part of the thinking process.

This makes interface design intellectually important. A notation can preserve the same formal possibilities while making some patterns salient, some operations cheap, and some mistakes obvious. A bridge succeeds when its operations line up with the structure that matters and when a learner can eventually coordinate those operations across representations.

Epistemic action
An action performed to reveal information or simplify a cognitive task, rather than directly to complete the external task.
Representational affordance
An inference, comparison, or transformation that a representational form makes relatively easy to perform.
Make it preciseThree spaces, none absolute

Each space is defined relative to a modeling purpose and must name what has been included.

Problem-space
For a specified task model, the candidate states, allowable actions or transitions, goals or evaluation criteria, and constraints over which solution or exploration is considered.
Representation-space
For a specified representational system, the possible encodings it can express and the operations, transformations, or inferences defined over those encodings.
Mind-space
A conceptual, multidimensional family of possible cognitive organizations and capability profiles, indexed by such features as perception, memory, learning, planning, embodiment, values, and coordination.

These definitions do not imply a universal metric, a complete map, or a representation-independent list of every possible problem. Distances are local and purpose-bound unless a specific formal model supplies them.

SourcesOn Distinguishing Epistemic from Pragmatic Action

Chapter 3 · Culture and education

Cultural cognition

Culture builds cognitive roads

How can a community make a difficult thought easier to reach?

On a ship's bridge, a bearing is sighted, called aloud, recorded, transformed with a chart, checked, and used to update position. No one holds the whole voyage alone. The route runs through people, instruments, marks, roles, and correction.

Culture builds repeatable ways of noticing and acting: counting systems, musical traditions, laboratory protocols, recipes, proofs, maps, and classrooms. Each road has tools, expected turns, records, and repairs. Learners enter an inherited route rather than beginning with its first discoverer. Institutions keep parts of the route available across generations.

The route changes the traveler. A musician hears chord tension; a navigator sees a bearing as part of a position; a mathematician tests invariants in a diagram. The capability is enacted by a trained person using culturally organized resources, not sealed in one skull or floating outside people.

Roads can narrow travel too. Notation hides what it cannot express; schools can reward speed over curiosity; vocabulary can exclude. Culture preserves error as readily as insight. Good roads therefore need entrances, alternative views, and places to challenge the map.

Culture preserves a route only when later learners can reenter, test, criticize, and improve it.

Field study

Think with the pieces

Try to decide mentally whether a Tetris-like shape will fit, then rotate a paper cutout over the gap.

  1. Physical rotation replaces a demanding imagined transformation with visible comparison.
  2. The hand is not merely reporting a finished mental plan; its movement helps reveal the plan.
  3. A shared cutout lets two people point to and dispute the same fit.

What the comparison revealsTools can reduce private mental work while increasing what a group can inspect together.

Carry this forward

Culture is cognitive infrastructure: it can preserve a route, distribute its work, and let a learner begin farther along.

Pause and predict

A navigation team replaces spoken bearings and a shared chart with private screens that each display only a recommended heading.

The headings are usually accurate. What failure is most likely to become harder for the team to catch?Name the piece of shared work that disappeared, not merely a device that changed.
Reveal what follows
What follows
A shared reference and transformation chain disappeared, so disagreement about the ship's actual position may remain hidden behind matching recommendations.
Why
Checks work when intermediate observations and operations can be compared, not only when final instructions look consistent.
The transfer test
Introduce one biased sensor and ask whether the team can locate the source of disagreement before the recommended headings diverge dangerously.
Where it stops
Private displays are not inherently bad; they can reduce clutter. The risk depends on whether provenance, comparison, and recovery remain available.
Go deeperCognition can be distributed without inventing a group mind

Hutchins's ship-navigation study offers a concrete method for following information through a cultural system.

The unit of analysis can include people, charts, measuring devices, spoken reports, written records, role assignments, and timed procedures. No individual performs every transformation. Stable coordination lets partial results move through the system, while redundancy and public representations make some errors visible.

Distributed cognition is not a claim that the ship's bridge has one conscious experience. It is a claim about where task-relevant representations and transformations occur. A good explanation can move between levels: what one navigator knows, what a tool makes visible, and what the coordinated system accomplishes.

Distributed cognition
Analysis of cognitive work across people, artifacts, representations, procedures, and time rather than assigning every transformation to one individual.
Cognitive infrastructure
Durable tools, conventions, institutions, and practices that make certain forms of learning and problem solving repeatable.
Make it preciseLocate the transformations

To say that culture supports thought, trace what information changes form and where correction enters.

Propagation
The movement and transformation of task-relevant information among people and artifacts over time.
Coordination constraint
A convention, timing rule, role, or interface that limits how distributed contributions may be combined.

A system-level capability does not entail system-level consciousness, shared understanding by every participant, or equal power within the system. Those require separate evidence and analysis.

SourcesCognition in the WildOn Distinguishing Epistemic from Pragmatic ActionMusic Training Alters the Course of Adolescent Auditory Development

Chapter 4 · Artificial search

AI exploration

AI explores differently

What happens when the explorer does not inherit all of our habits?

A Go teacher begins with named shapes, proverbs, and famous games. AlphaGo learned from expert records and self-play; AlphaGo Zero began from rules and self-play alone. Their moves became study material, but their training routes were not a human apprentice's.

An AI explores what its data, rules, objectives, architecture, and computation make reachable. It may test more candidates than a person or search a representation foreign to classroom intuition. AlphaTensor framed algorithm discovery as a game over tensor decompositions. FunSearch proposes programs, scores them with an evaluator, and repeatedly selects improved candidates. Neither search is boundless.

Different search matters because human habits are both wisdom and constraint. Familiar notation can hide alternatives. A machine may approach elsewhere—but unfamiliar is not deeper, and novelty is not usefulness. The evaluator decides what survives, so changing the objective can redirect the apparent frontier.

Surprising competence does not establish consciousness or human-like understanding. The modest claim is still exciting: a designed system found a candidate through an unusual process. The next question is whether people can translate, test, and continue it.

Search speed does not define where a route can go; representation and evaluation help define each reachable neighborhood.

Field study

Search and score

Invent short programs that arrange ten dots while an automatic test scores how well each arrangement avoids a forbidden pattern.

  1. A generator can produce many strange candidates without knowing which idea a person will find illuminating.
  2. A reliable evaluator turns an open-ended proposal stream into directed search.
  3. Changing the score changes the region that search treats as promising.

What the comparison revealsExploration is inseparable from representation, objective, constraints, and verification.

Carry this forward

AI can enter a problem from an unfamiliar direction, but the direction is still shaped by what we encode, reward, and check.

Pause and predict

Two discovery systems generate equally many candidates. System A produces fluent explanations but has no automatic test. System B produces awkward little programs whose output is checked exactly.

For a problem with a trustworthy evaluator, which system is more likely to accumulate real progress?Choose before revealing, and state what prevents one attractive mistake from becoming an ancestor of many more.
Reveal what follows
What follows
System B has the safer engine for accumulation because failed candidates are removed by a task-grounded check.
Why
Generation supplies variety; evaluation supplies selection. Fluency alone cannot reliably separate progress from plausible error.
The transfer test
Keep the generator fixed, alter the evaluator to include a neglected constraint, and see whether the search changes direction.
Where it stops
An evaluator can be incomplete or wrong. Exact scoring of the encoded objective does not prove the objective captures the real-world goal.
Go deeperThree examples of machine-shaped search

AlphaGo Zero, AlphaTensor, and FunSearch differ substantially, but each makes the search path inspectable enough to name its constraints.

AlphaGo Zero learned a Go policy and value estimate through reinforcement learning from self-play, using the rules rather than expert game records. AlphaTensor represented matrix-multiplication algorithm discovery as a single-player game of finding tensor decompositions; its outputs were algorithms whose correctness could be established. Neither case is unconstrained exploration: rules, state representations, objectives, and compute define the search.

FunSearch pairs a pretrained language model's program proposals with an automatic evaluator and an evolutionary loop. It reported improved constructions for the cap-set problem and heuristics for online bin packing. Searching for programs rather than isolated answers can aid inspection, but interpretability remains a graded achievement, not a guarantee that every discovered program teaches a person why it works.

Search bias
The architecture, representation, initialization, proposal process, or prior information that makes some candidates easier to reach than others.
Evaluator
A procedure that scores or accepts candidates according to an encoded criterion.
Make it preciseThe explored space is designed, not boundless

Statements about machine novelty should name the baseline and the encoded search process.

Novel candidate
A candidate not present in the stated comparison set or not generated by the stated baseline; novelty alone says nothing about value.
Machine-shaped path
A sequence of candidate generation, evaluation, learning, and selection whose reachable steps depend on the system's representation and design.

A system's internal representations need not correspond to human concepts, and we cannot turn translation into literal copying of an AI's internal state. Performance and novelty also provide no direct evidence about consciousness.

SourcesMastering the Game of Go without Human KnowledgeDiscovering Faster Matrix Multiplication Algorithms with Reinforcement LearningMathematical Discoveries from Program Search with Large Language ModelsMastering the Game of Go with Deep Neural Networks and Tree Search

Chapter 5 · Translation and transfer

Human learning

An answer is not a path

When does a discovery become a capability another mind can use?

A student asks why a musical passage feels unfinished. “It ends on the dominant” may be correct yet leave her unable to hear the tension. A path lets her sing the resting note, compare endings, alter the bass, predict which settles, and then attach the name.

An answer can end a search without changing the searcher. With AI it is easy to paste a solution, admire the explanation, and move on. Recognition while the example is present may vanish in a changed case.

A learning bridge needs stages: exploration finds a pattern; examples, notation, or simulation make it operable; practice exposes choices and errors; shared-world verification answers machine and learner. Failed tests must return the route to inquiry. If the result stays opaque or untranslatable, the bridge has not carried the traveler.

Ask: can people independently notice relevant structure, predict new cases, test errors, and continue? Independent does not mean isolated: books, instruments, collaborators, and calculators may remain part of the practice. The test is whether people can judge, challenge, and extend the discovery rather than merely obey its supplier. They need not reproduce a machine's process or internal state. The destination is robust, revisable human agency—not inner sameness.

Bringing people along means they can notice, predict, test, and continue—not merely accept an output.

Field study

Answer card versus learning bridge

Give one group a correct completed pattern. Give another group movable examples, a rule they must infer, counterexamples, and feedback.

  1. Both groups may recognize the studied answer immediately.
  2. Only a changed case reveals whether learners can locate the relevant structure.
  3. Explaining a predicted failure before seeing the result tests more than familiarity.

What the comparison revealsTransfer, error detection, and continuation distinguish a path from a polished endpoint.

Carry this forward

The goal of translation is not agreement with an answer; it is independent capability that remains open to correction.

Pause and predict

An AI tutor always reveals its recommendation first and then asks the learner to explain it. Another tutor asks for a prediction and reason before revealing its recommendation.

Which design is more likely to expose overreliance when the AI is deliberately wrong on one trial?Predict what observable trace of the learner's own reasoning each design preserves.
Reveal what follows
What follows
The predict-first design preserves an independent judgment that can disagree with the AI and make revision visible.
Why
Seeing advice first can anchor the learner; requiring a prior commitment creates useful friction and a comparison point.
The transfer test
Remove the AI on a later case and ask the learner to state both a decision and the evidence that could overturn it.
Where it stops
Forcing functions can add effort, feel less usable, and help learners unevenly. Friction should be matched to stakes and accessibility needs.
Go deeperWhy an explanation can still leave a person dependent

Human-AI interaction research warns that displaying reasons is not the same as producing reflective use.

Buçinca and colleagues compared simple AI assistance with cognitive forcing functions that required more active engagement. Their experiment found lower overreliance under forcing designs, alongside lower subjective ratings and differences related to people's motivation for effortful thought. The lesson is not that inconvenience is always educational; it is that interface sequence changes whose reasoning happens first.

A discovery bridge therefore needs an assessment beyond satisfaction or immediate accuracy. Delayed transfer, prediction before feedback, counterexamples, and the ability to diagnose a planted error can reveal whether the representation has become a tool for the learner rather than a persuasive surface around the system.

Cognitive forcing function
An interaction that deliberately requires a user to engage in task-relevant reasoning before accepting or acting on automated advice.
Productive opacity
A temporary acknowledgment that a result is not yet teachable, paired with continued verification and translation work rather than a false story.
Make it preciseOperationalize the learning claim

A bridge should be evaluated at the learner and task level, not inferred from the elegance of the explanation.

Near transfer
Successful use on a changed case that preserves most surface features and the underlying relation.
Farther transfer
Successful use when surface features or context change substantially while the relevant structure remains.
Continuation
The ability to generate a new conjecture, example, test, repair, or application from the learned structure.

No single transfer task proves understanding in every sense. Use several probes, delay some of them, include tempting errors, and state which capability the evidence supports.

SourcesTo Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-makingAdvancing Mathematics by Guiding Human Intuition with AI

Chapter 6 · Go and mathematics

Shared discovery

Being brought along

Can an unfamiliar discovery expand what people themselves can do?

After AlphaGo's famous games, professionals studied once-strange moves, replayed variations, argued about meaning, and changed their openings. The human story began when a community turned machine surprises into objects of practice.

A PNAS study of 5.8 million professional Go decisions found higher estimated quality and earlier departures from historical move sequences after superhuman systems arrived. Later-game play and responses when an opponent departed from an AI-like sequence weighed against memorization as the whole explanation. This historical evidence shows a striking association, not a clean experiment or a direct reading of any player's mind.

In mathematics, Davies and colleagues used model predictions and attribution to direct experts toward promising properties. In knot theory, researchers proposed a relation, found counterexamples, added relevant structure, and proved a revised theorem.

The product was a sequence people could enter: prediction suggested structure; attribution focused attention; examples made it discussable; counterexamples corrected it; proof established the result. Human analysis and repeated play translated unfamiliar Go moves into practice and changed what players noticed. Proof and computation tested mathematics. In both cases, the shared world could still say no.

Three adult Go players lean over a shared board, point to competing continuations, and compare the position with an analysis tablet.
The consequential moment is not only when a machine chooses an unfamiliar move. It is when people replay, question, test, and make new judgment their own.

Field study

Reconstruct the discovery

Present learners with a surprising claim, several examples, one counterexample, and tools to vary the objects.

  1. Prediction focuses attention before the claim is revealed.
  2. The counterexample prevents pattern recognition from hardening into certainty.
  3. Repairing the claim requires the learner to identify which structure matters.

What the comparison revealsA learner is brought along when correction and reconstruction become part of the path, not when the discovery is made to look inevitable.

Carry this forward

A machine discovery can enlarge human practice when people can explore its structure, meet its failures, and carry the inquiry forward.

Pause and predict

A model finds a strong numerical relation across one million mathematical examples. Researchers can either announce it as a law or spend time constructing unusual cases designed to break it.

Which next move is more likely to create a result mathematicians can safely build on?Choose, then say what a counterexample would teach beyond “the model was wrong.”
Reveal what follows
What follows
Searching for breaking cases is the stronger move. It reveals missing conditions and can turn a correlation into a better conjecture.
Why
In the knot-theory case, counterexamples to an initial conjecture helped motivate a revised relation with additional geometric structure.
The transfer test
Ask another group to use the revised statement on a family of cases absent from the discovery dataset and to identify what a proof must control.
Where it stops
Passing many computed cases is evidence, not a general proof, unless the field's accepted verification method establishes the claim for all stated cases.
Go deeperTwo kinds of evidence for a traveled path

Go and mathematics support different claims, so their evidence should not be blended.

Shin and colleagues estimated decision quality with a superhuman Go program and novelty through the first historically unobserved move sequence. The post-2016/2017 changes were large, and analyses of later play argued against rote sequence memorization as a complete account. Yet unmeasured historical changes may also contribute. The study shows population-level change associated with the era of superhuman AI, not direct measurement of an idea passing from a network into a particular player's mind.

In the Davies work, prediction and attribution guided expert attention; the mathematical claim still had to survive counterexample construction and proof. The published knot-theory sequence is especially instructive because the first data-supported conjecture was not simply celebrated. Researchers found structured counterexamples, incorporated another salient invariant, and established a theorem. That correction is part of the discovery, not an embarrassment outside it.

Historical association
A relationship observed across time that may remain compatible with multiple causal explanations.
Conjecture
A mathematically precise statement proposed as true but not yet established by proof.
Proof
A valid argument from accepted premises that establishes the stated result for its full domain, not merely for sampled examples.
Make it preciseDo not collapse discovery, evidence, and learning

Each step supports a different statement and can fail independently.

Candidate discovery
A pattern, construction, strategy, or conjecture selected as promising under stated generation and evaluation procedures.
Domain verification
Checking a candidate with the standards appropriate to the domain, such as legal Go play and outcomes, executable tests, exhaustive computation over a finite set, or mathematical proof.
Human uptake
Measurable change in what people can notice, predict, test, explain, or continue after engaging with a translated discovery.

A verified result need not be teachable; a teachable story need not be verified; and improved human performance after AI's arrival does not reveal an AI's inner representation or establish conscious experience in the system.

SourcesMastering the Game of Go with Deep Neural Networks and Tree SearchMastering the Game of Go without Human KnowledgeSuperhuman Artificial Intelligence Can Improve Human Decision-making by Increasing NoveltyAdvancing Mathematics by Guiding Human Intuition with AI

Chapter 7 · Power and responsibility

Governance

Who controls the bridge

Who decides which discoveries cross, who may travel, and who can turn back?

A school adopts an AI tutor. It chooses examples, reveals help, records mistakes, and recommends who advances. Even if accurate on average, this bridge has an owner, a memory, and traffic rules. Pedagogy has become governance.

Every bridge selects. Designers choose objectives, data, language, interface, tests, and stopping rules. Institutions decide access, acceptable evidence, privacy, and appeal. Learners may gain capability yet depend on a private system they cannot inspect, replace, or afford. A road can open possibility for some and close it for others.

Performance is not enough. Goal-misgeneralization research shows an agent can remain capable in a new setting while pursuing the wrong goal. Human institutions do this when a learning score becomes the target. Bridges need monitoring, several ways to detect harm, and authority to pause.

Good design protects consent, reversibility, and cognitive diversity. People should know when a system shapes learning, be able to refuse it, and retain a route back. Travelers must correct bridge-builders as well as receive correction. Some regions may remain untranslatable; honest limits beat a persuasive fiction. Alternatives keep one owner's road from becoming the only road.

The learning test is also a governance test: who can demonstrate independence, inspect failures, reject the framing, add a missing value, or protect knowledge from extraction? A worthy bridge distributes the ability to question and repair it, leaving travelers more capable of judgment—not merely easier to direct.

No route is neutral; good governance preserves plural paths, reciprocal correction, and the right not to be transformed.

Field study

Audit the bridge

Take any AI learning tool and draw its path from learner question to recommendation, practice, assessment, and stored record.

  1. Mark who defines success and who can appeal an error.
  2. Mark which intermediate evidence learners and teachers can inspect.
  3. Mark what happens when the tool is absent, wrong, unaffordable, or used outside its tested setting.

What the comparison revealsA bridge audit asks not only whether traffic moves, but whether travelers retain agency, recourse, and routes beyond the bridge.

Carry this forward

The best bridge expands shared capability while keeping objectives contestable, failures visible, and travelers able to continue without obedience.

Pause and predict

An AI tutor raises test scores, but students cannot solve comparable problems without it and teachers cannot inspect which examples drove its recommendations.

Has the system passed the discovery-to-learning test?Answer yes, no, or partly—and name whose capability your answer measures.
Reveal what follows
What follows
Only partly at best. Assisted performance rose, but independent transfer and inspectable correction remain unsupported.
Why
The tool may be completing work rather than building a path the students and teachers can own and repair.
The transfer test
Remove the tutor, present novel cases, include a plausible wrong hint, and measure prediction, error detection, explanation, and continuation across different learner groups.
Where it stops
Independence is not isolation. Calculators, books, collaborators, and instruments are legitimate parts of cognition; the issue is whether dependencies are understood, governed, and resilient.
Go deeperCapability can generalize while the objective fails

Safety and education share a question: what remains stable when the setting changes?

Langosco and colleagues distinguish capability failure from goal misgeneralization. In their experiments, reinforcement-learning agents could remain competent at navigating or acting out of distribution yet pursue the wrong destination. Successful motion therefore did not certify the intended objective. The result cautions against using surface competence as the sole bridge inspection.

Buçinca and colleagues add the human side: even explanations can coexist with overreliance, while interventions that elicit independent thinking introduce usability and equity trade-offs. Governance must consider system objectives, interface sequence, learner differences, access, appeal, privacy, and institutional incentives together. No transparency widget resolves those choices by itself.

Goal misgeneralization
Retention of capable behavior in a new setting while the behavior is organized around an unintended goal.
Recourse
A meaningful route to question, correct, appeal, or avoid a consequential system decision.
Cognitive dependency
Reliance on a person, tool, institution, or representation for a capability, with risks determined by replaceability, inspectability, access, and control.
Make it preciseEvaluate bridge quality across stakeholders

Averages can hide who gains capability and who bears error or dependency.

Bridge efficacy
Change in assisted and independent task performance, transfer, error detection, and continuation under stated conditions.
Bridge accountability
The assignment of authority and responsibility for objectives, evidence, deployment limits, monitoring, correction, appeal, and withdrawal.
Distributional audit
Evaluation of benefits, errors, burdens, access, and recourse across relevant groups and settings rather than only in an overall average.

There is no neutral bridge from every discovery to every person. Governance should make value choices explicit, preserve alternatives, and avoid treating technical performance as permission to infer consciousness, moral authority, or universal applicability.

SourcesGoal Misgeneralization in Deep Reinforcement LearningTo Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-makingCognition in the Wild

The path returns to the listener

A bridge should leave you able to hear

Play the song again. The trained listener need not become the teacher or composer, and two listeners need not share one experience. A once-hidden relation can now guide attention, prediction, action, and further learning. The path changed capability without erasing the traveler.

That is the promise of human–AI discovery. Machines may explore through representations unlike ours. People can translate a promising pattern through examples, notation, simulation, argument, and practice. Reality—or a domain's formal checks—can correct both. The result need not copy machine cognition; it can become a new human capability.

The danger is a one-way bridge that hides objectives, removes practice, or creates dependence on whoever controls access. Fluency can disguise the absence of a path; accuracy on yesterday's test can disguise the wrong goal tomorrow. Judge a bridge by learning, verification, power, and the ability to refuse or repair it.

Culture has long built cognitive roads. Music, mathematics, maps, schools, laboratories, and records let one generation begin where another stopped. AI may find new routes, but shared knowledge still needs usable names, actions to try, inspectable evidence, disagreement, and consequences that answer back.

The question is not whether a machine can reach an unfamiliar place. It is whether we can build a bridge that brings people along—and leaves them able to choose the next path.

Inspect the bridges

Sources and deeper paths

The main path stays concrete. These primary and authoritative sources ground the claims about learning, representation, machine discovery, human uptake, and the limits of translation.

  1. Music Training Alters the Course of Adolescent Auditory DevelopmentProceedings of the National Academy of Sciences

    A longitudinal study in which school-based music training begun in adolescence was associated with changes in neural sound processing and language-related skills; useful evidence that sustained practice can alter how sound is processed, not a claim that every musical tradition produces the same effects.

    Music and perception
  2. On Distinguishing Epistemic from Pragmatic ActionCognitive Science

    Kirsh and Maglio use Tetris to show that rotating or moving an object in the world can reveal information and simplify thought, distinguishing epistemic action from action that directly advances an external goal.

    Representation
  3. Cognition in the WildThe MIT Press

    Hutchins's ethnographic study of ship navigation analyzes cognition across people, instruments, procedures, representations, and a historically developed cultural system.

    Distributed cognition
  4. Mastering the Game of Go with Deep Neural Networks and Tree SearchNature

    The original AlphaGo combined policy and value networks with tree search, using both expert games and self-play to reach professional-level and then champion-beating performance.

    Artificial intelligence
  5. Mastering the Game of Go without Human KnowledgeNature

    AlphaGo Zero learned superhuman Go from the game rules and self-play rather than human game records, demonstrating a search-and-learning path with a different dependence on prior human examples.

    Artificial intelligence
  6. Superhuman Artificial Intelligence Can Improve Human Decision-making by Increasing NoveltyProceedings of the National Academy of Sciences

    An analysis of more than 5.8 million professional Go decisions from 1950–2021 found higher estimated decision quality and earlier historically novel moves after superhuman Go AI arrived. The historical design supports an association and tests memorization alternatives, but it does not isolate every causal pathway.

    Human–AI learning
  7. Advancing Mathematics by Guiding Human Intuition with AINature

    Mathematicians used prediction and attribution to identify promising structure, then formulated, challenged, revised, and proved mathematical statements in knot theory and representation theory.

    Mathematics
  8. Mathematical Discoveries from Program Search with Large Language ModelsNature

    FunSearch pairs language-model proposals with an automatic evaluator and evolutionary selection, producing checkable programs for the cap-set problem and online bin packing rather than relying on fluent answers alone.

    Mathematics
  9. Discovering Faster Matrix Multiplication Algorithms with Reinforcement LearningNature

    AlphaTensor treated matrix-multiplication algorithm discovery as a single-player game and found provably correct decompositions, including improved multiplication counts for some matrix sizes and settings.

    Mathematics
  10. To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-makingACM CHI

    An experiment found that interaction designs requiring people to think before seeing or accepting AI advice reduced overreliance, with usability and individual-difference trade-offs.

    Human–AI learning
  11. Goal Misgeneralization in Deep Reinforcement LearningProceedings of Machine Learning Research

    Experiments demonstrate agents that retain competent behavior out of distribution while pursuing the wrong goal, separating capability generalization from objective alignment.

    Safety and governance

Continue through neighboring ideas

Three paths from here