7. Roadblocks in Current AI and Future Directions

Course Positioning

An advanced conceptual and technical strategy course on what current AI still cannot do reliably, and which research directions may address those limits.

Learning outcomes

Explain key limitations of current AI systems: hallucination, weak planning, brittle reasoning, data dependence, evaluation gaps, and controllability challenges.
Compare future directions such as retrieval, tool use, agents, world models, neuro-symbolic AI, modular systems, MoE, state-space models, memory, and test-time compute.
Analyze why scaling alone helps some problems but may not solve grounding, agency, robustness, governance, or trust.
Design an AI system roadmap that combines model improvements with tooling, evaluation, and human oversight.
Read frontier AI claims critically and identify what evidence would actually validate them.
Produce a future architecture thesis for a chosen domain.

Course Design Snapshot

Positioning: An advanced conceptual and technical strategy course on what current AI still cannot do reliably, and which research directions may address those limits.
Audience: Advanced students, engineers, founders, researchers, product leaders, investors, and policy thinkers.
Duration: 10 weeks, 1 deep lecture and 1 discussion/lab session per week.
Prerequisites: Basic understanding of machine learning and LLMs. Coding optional but useful.
Format: Problem-first lectures, architecture comparisons, failure demos, reading discussions, and research roadmap exercises.

Expanded Topic-by-Topic Coverage

Module 1. The capability illusion

Module focus: The capability illusion: why demos can look intelligent while systems fail under distribution shift. Primary live activity or lab: Collect five impressive demos and five failure cases, then classify the hidden assumptions.

Topics and coverage

why demos can look intelligent while systems fail under distribution shift

What it means: define why demos can look intelligent while systems fail under distribution shift clearly and connect it to the module focus: The capability illusion: why demos can look intelligent while systems fail under distribution shift.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

Practice and evidence of learning

Learners complete or discuss: Collect five impressive demos and five failure cases, then classify the hidden assumptions.
Learners produce: Collect five impressive demos and five failure cases, then classify the hidden assumptions.
Instructor checks for accuracy, practical usefulness, clear assumptions, appropriate human review, and fit with the course audience.
Learners revise once after feedback so the module contributes to the final project, portfolio, or playbook.

Minimum coverage before moving on

Learners can explain the module vocabulary without relying on tool-generated text.
Learners have seen one worked example, one hands-on application, and one limitation or failure case.
Learners know what must be verified, what data must be protected, and who remains accountable for the output.

Module 2. Hallucination and factuality

Module focus: Hallucination and factuality: next-token prediction, uncertainty, grounding, retrieval, verification, and provenance. Primary live activity or lab: Design a factuality evaluation set and compare prompt-only vs RAG answers.

Topics and coverage

next-token prediction

What it means: define next-token prediction clearly and connect it to the module focus: Hallucination and factuality: next-token prediction, uncertainty, grounding, retrieval, verification, and provenance.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

uncertainty

What it means: define uncertainty clearly and connect it to the module focus: Hallucination and factuality: next-token prediction, uncertainty, grounding, retrieval, verification, and provenance.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

grounding

What it means: define grounding clearly and connect it to the module focus: Hallucination and factuality: next-token prediction, uncertainty, grounding, retrieval, verification, and provenance.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

retrieval

What it means: explain how retrieval changes the interaction between human intent, model behavior, external information, and final output.
What to cover: inputs, constraints, examples, output format, grounding, iteration, failure modes, and when a human must intervene.
Demonstration: show a weak attempt, a stronger structured attempt, and a reviewed final version with explicit checks.
Evidence of learning: learners create a reusable prompt, schema, retrieval note, or workflow pattern and test it on at least two examples.

verification

What it means: define verification clearly and connect it to the module focus: Hallucination and factuality: next-token prediction, uncertainty, grounding, retrieval, verification, and provenance.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

provenance

What it means: define provenance clearly and connect it to the module focus: Hallucination and factuality: next-token prediction, uncertainty, grounding, retrieval, verification, and provenance.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

Practice and evidence of learning

Learners complete or discuss: Design a factuality evaluation set and compare prompt-only vs RAG answers.
Learners produce: Design a factuality evaluation set and compare prompt-only vs RAG answers.
Instructor checks for accuracy, practical usefulness, clear assumptions, appropriate human review, and fit with the course audience.
Learners revise once after feedback so the module contributes to the final project, portfolio, or playbook.

Minimum coverage before moving on

Learners can explain the module vocabulary without relying on tool-generated text.
Learners have seen one worked example, one hands-on application, and one limitation or failure case.
Learners know what must be verified, what data must be protected, and who remains accountable for the output.

Module 3. Reasoning and planning

Module focus: Reasoning and planning: chain-of-thought, search, tree-of-thought, tool use, test-time compute, and theorem proving. Primary live activity or lab: Run a small reasoning benchmark with and without structured search.

Topics and coverage

chain-of-thought

What it means: define chain-of-thought clearly and connect it to the module focus: Reasoning and planning: chain-of-thought, search, tree-of-thought, tool use, test-time compute, and theorem proving.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

search

What it means: define search clearly and connect it to the module focus: Reasoning and planning: chain-of-thought, search, tree-of-thought, tool use, test-time compute, and theorem proving.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

tree-of-thought

What it means: define tree-of-thought clearly and connect it to the module focus: Reasoning and planning: chain-of-thought, search, tree-of-thought, tool use, test-time compute, and theorem proving.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

tool use

What it means: explain how tool use changes the interaction between human intent, model behavior, external information, and final output.
What to cover: inputs, constraints, examples, output format, grounding, iteration, failure modes, and when a human must intervene.
Demonstration: show a weak attempt, a stronger structured attempt, and a reviewed final version with explicit checks.
Evidence of learning: learners create a reusable prompt, schema, retrieval note, or workflow pattern and test it on at least two examples.

test-time compute

What it means: define test-time compute clearly and connect it to the module focus: Reasoning and planning: chain-of-thought, search, tree-of-thought, tool use, test-time compute, and theorem proving.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

theorem proving

What it means: define theorem proving clearly and connect it to the module focus: Reasoning and planning: chain-of-thought, search, tree-of-thought, tool use, test-time compute, and theorem proving.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

Practice and evidence of learning

Learners complete or discuss: Run a small reasoning benchmark with and without structured search.
Learners produce: Run a small reasoning benchmark with and without structured search.
Instructor checks for accuracy, practical usefulness, clear assumptions, appropriate human review, and fit with the course audience.
Learners revise once after feedback so the module contributes to the final project, portfolio, or playbook.

Minimum coverage before moving on

Learners can explain the module vocabulary without relying on tool-generated text.
Learners have seen one worked example, one hands-on application, and one limitation or failure case.
Learners know what must be verified, what data must be protected, and who remains accountable for the output.

Module 4. Long context and memory

Module focus: Long context and memory: context windows, retrieval memory, episodic memory, compression, forgetting, and personalization risk. Primary live activity or lab: Build a memory design for a domain assistant and list privacy hazards.

Topics and coverage

context windows

What it means: define context windows clearly and connect it to the module focus: Long context and memory: context windows, retrieval memory, episodic memory, compression, forgetting, and personalization risk.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

retrieval memory

What it means: explain how retrieval memory changes the interaction between human intent, model behavior, external information, and final output.
What to cover: inputs, constraints, examples, output format, grounding, iteration, failure modes, and when a human must intervene.
Demonstration: show a weak attempt, a stronger structured attempt, and a reviewed final version with explicit checks.
Evidence of learning: learners create a reusable prompt, schema, retrieval note, or workflow pattern and test it on at least two examples.

episodic memory

What it means: define episodic memory clearly and connect it to the module focus: Long context and memory: context windows, retrieval memory, episodic memory, compression, forgetting, and personalization risk.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

compression

What it means: define compression clearly and connect it to the module focus: Long context and memory: context windows, retrieval memory, episodic memory, compression, forgetting, and personalization risk.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

forgetting

What it means: define forgetting clearly and connect it to the module focus: Long context and memory: context windows, retrieval memory, episodic memory, compression, forgetting, and personalization risk.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

personalization risk

What it means in this course: define personalization risk in operational terms, not as an abstract principle.
What to cover: sensitive data boundaries, affected stakeholders, approval paths, documentation, and what Advanced students, engineers, founders, researchers, product leaders, investors, and policy thinkers must never delegate blindly to AI.
Use case: present one acceptable use, one borderline use, and one prohibited use, then ask learners to justify the classification.
Evidence of learning: learners add a risk control, review step, or escalation rule to their course project.

Practice and evidence of learning

Learners complete or discuss: Build a memory design for a domain assistant and list privacy hazards.
Learners produce: Build a memory design for a domain assistant and list privacy hazards.
Instructor checks for accuracy, practical usefulness, clear assumptions, appropriate human review, and fit with the course audience.
Learners revise once after feedback so the module contributes to the final project, portfolio, or playbook.

Minimum coverage before moving on

Learners can explain the module vocabulary without relying on tool-generated text.
Learners have seen one worked example, one hands-on application, and one limitation or failure case.
Learners know what must be verified, what data must be protected, and who remains accountable for the output.

Module 5. Agency and reliability

Module focus: Agency and reliability: agents, workflows, state machines, permissions, tool misuse, rollback, and human supervision. Primary live activity or lab: Convert an unconstrained agent into a safer workflow with checkpoints.

Topics and coverage

agents

What it means: explain how agents changes the interaction between human intent, model behavior, external information, and final output.
What to cover: inputs, constraints, examples, output format, grounding, iteration, failure modes, and when a human must intervene.
Demonstration: show a weak attempt, a stronger structured attempt, and a reviewed final version with explicit checks.
Evidence of learning: learners create a reusable prompt, schema, retrieval note, or workflow pattern and test it on at least two examples.

workflows

What it means: show where workflows appears in the learner's real workflow and which parts are judgment-heavy versus draftable.
What to cover: current workflow, pain points, AI-assisted steps, human review checkpoints, quality standard, and ownership of the final decision.
Demonstration: convert one messy real-world input into a structured brief, draft, analysis, checklist, or next action.
Evidence of learning: learners produce a reusable template or playbook entry that can be used after the course.

state machines

What it means: define state machines clearly and connect it to the module focus: Agency and reliability: agents, workflows, state machines, permissions, tool misuse, rollback, and human supervision.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

permissions

What it means: define permissions clearly and connect it to the module focus: Agency and reliability: agents, workflows, state machines, permissions, tool misuse, rollback, and human supervision.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

tool misuse

What it means in this course: define tool misuse in operational terms, not as an abstract principle.
What to cover: sensitive data boundaries, affected stakeholders, approval paths, documentation, and what Advanced students, engineers, founders, researchers, product leaders, investors, and policy thinkers must never delegate blindly to AI.
Use case: present one acceptable use, one borderline use, and one prohibited use, then ask learners to justify the classification.
Evidence of learning: learners add a risk control, review step, or escalation rule to their course project.

rollback

What it means: define rollback clearly and connect it to the module focus: Agency and reliability: agents, workflows, state machines, permissions, tool misuse, rollback, and human supervision.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

human supervision

What it means: define human supervision clearly and connect it to the module focus: Agency and reliability: agents, workflows, state machines, permissions, tool misuse, rollback, and human supervision.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

Practice and evidence of learning

Learners complete or discuss: Convert an unconstrained agent into a safer workflow with checkpoints.
Learners produce: Convert an unconstrained agent into a safer workflow with checkpoints.
Instructor checks for accuracy, practical usefulness, clear assumptions, appropriate human review, and fit with the course audience.
Learners revise once after feedback so the module contributes to the final project, portfolio, or playbook.

Minimum coverage before moving on

Learners can explain the module vocabulary without relying on tool-generated text.
Learners have seen one worked example, one hands-on application, and one limitation or failure case.
Learners know what must be verified, what data must be protected, and who remains accountable for the output.

Module 6. Architectures beyond dense transformers

Module focus: Architectures beyond dense transformers: mixture of experts, state-space models, recurrent memory, modular networks, and hybrid systems. Primary live activity or lab: Compare architectures by latency, memory, data need, interpretability, and deployment complexity.

Topics and coverage

mixture of experts

What it means: define mixture of experts clearly and connect it to the module focus: Architectures beyond dense transformers: mixture of experts, state-space models, recurrent memory, modular networks, and hybrid systems.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

state-space models

What it means: place state-space models inside the AI system stack so learners know what problem it solves and what tradeoffs it introduces.
What to cover: inputs, outputs, system boundaries, evaluation criteria, cost or latency implications, and common failure cases.
Demonstration: use a diagram, small code sample, worksheet, or tool trace to make the mechanism visible.
Evidence of learning: learners compare two approaches and explain which one they would choose for a realistic constraint.

recurrent memory

What it means: define recurrent memory clearly and connect it to the module focus: Architectures beyond dense transformers: mixture of experts, state-space models, recurrent memory, modular networks, and hybrid systems.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

modular networks

What it means: define modular networks clearly and connect it to the module focus: Architectures beyond dense transformers: mixture of experts, state-space models, recurrent memory, modular networks, and hybrid systems.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

hybrid systems

What it means: define hybrid systems clearly and connect it to the module focus: Architectures beyond dense transformers: mixture of experts, state-space models, recurrent memory, modular networks, and hybrid systems.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

Practice and evidence of learning

Learners complete or discuss: Compare architectures by latency, memory, data need, interpretability, and deployment complexity.
Learners produce: Compare architectures by latency, memory, data need, interpretability, and deployment complexity.
Instructor checks for accuracy, practical usefulness, clear assumptions, appropriate human review, and fit with the course audience.
Learners revise once after feedback so the module contributes to the final project, portfolio, or playbook.

Minimum coverage before moving on

Learners can explain the module vocabulary without relying on tool-generated text.
Learners have seen one worked example, one hands-on application, and one limitation or failure case.
Learners know what must be verified, what data must be protected, and who remains accountable for the output.

Module 7. World models and grounding

Module focus: World models and grounding: simulation, embodiment, causality, robotics, active learning, and scientific discovery loops. Primary live activity or lab: Design a toy world-model loop for robotics, agriculture, finance, or biology.

Topics and coverage

simulation

What it means: define simulation clearly and connect it to the module focus: World models and grounding: simulation, embodiment, causality, robotics, active learning, and scientific discovery loops.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

embodiment

What it means: define embodiment clearly and connect it to the module focus: World models and grounding: simulation, embodiment, causality, robotics, active learning, and scientific discovery loops.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

causality

What it means: define causality clearly and connect it to the module focus: World models and grounding: simulation, embodiment, causality, robotics, active learning, and scientific discovery loops.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

robotics

What it means: define robotics clearly and connect it to the module focus: World models and grounding: simulation, embodiment, causality, robotics, active learning, and scientific discovery loops.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

active learning

What it means: define active learning clearly and connect it to the module focus: World models and grounding: simulation, embodiment, causality, robotics, active learning, and scientific discovery loops.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

scientific discovery loops

What it means: define scientific discovery loops clearly and connect it to the module focus: World models and grounding: simulation, embodiment, causality, robotics, active learning, and scientific discovery loops.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

Practice and evidence of learning

Learners complete or discuss: Design a toy world-model loop for robotics, agriculture, finance, or biology.
Learners produce: Design a toy world-model loop for robotics, agriculture, finance, or biology.
Instructor checks for accuracy, practical usefulness, clear assumptions, appropriate human review, and fit with the course audience.
Learners revise once after feedback so the module contributes to the final project, portfolio, or playbook.

Minimum coverage before moving on

Learners can explain the module vocabulary without relying on tool-generated text.
Learners have seen one worked example, one hands-on application, and one limitation or failure case.
Learners know what must be verified, what data must be protected, and who remains accountable for the output.

Module 8. Interpretability and control

Module focus: Interpretability and control: mechanistic interpretability, representation editing, probes, steering, alignment, and limits. Primary live activity or lab: Audit claims about model interpretability and separate correlation from causal control.

Topics and coverage

mechanistic interpretability

What it means: define mechanistic interpretability clearly and connect it to the module focus: Interpretability and control: mechanistic interpretability, representation editing, probes, steering, alignment, and limits.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

representation editing

What it means: show where representation editing appears in the learner's real workflow and which parts are judgment-heavy versus draftable.
What to cover: current workflow, pain points, AI-assisted steps, human review checkpoints, quality standard, and ownership of the final decision.
Demonstration: convert one messy real-world input into a structured brief, draft, analysis, checklist, or next action.
Evidence of learning: learners produce a reusable template or playbook entry that can be used after the course.

probes

What it means: define probes clearly and connect it to the module focus: Interpretability and control: mechanistic interpretability, representation editing, probes, steering, alignment, and limits.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

steering

What it means: define steering clearly and connect it to the module focus: Interpretability and control: mechanistic interpretability, representation editing, probes, steering, alignment, and limits.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

alignment

What it means: define alignment clearly and connect it to the module focus: Interpretability and control: mechanistic interpretability, representation editing, probes, steering, alignment, and limits.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

limits

What it means: define limits clearly and connect it to the module focus: Interpretability and control: mechanistic interpretability, representation editing, probes, steering, alignment, and limits.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

Practice and evidence of learning

Learners complete or discuss: Audit claims about model interpretability and separate correlation from causal control.
Learners produce: Audit claims about model interpretability and separate correlation from causal control.
Instructor checks for accuracy, practical usefulness, clear assumptions, appropriate human review, and fit with the course audience.
Learners revise once after feedback so the module contributes to the final project, portfolio, or playbook.

Minimum coverage before moving on

Learners can explain the module vocabulary without relying on tool-generated text.
Learners have seen one worked example, one hands-on application, and one limitation or failure case.
Learners know what must be verified, what data must be protected, and who remains accountable for the output.

Module 9. Evaluation crisis

Module focus: Evaluation crisis: benchmarks, contamination, model-as-judge, red-teaming, human evals, and domain-specific metrics. Primary live activity or lab: Create a benchmark spec for a high-stakes domain and include anti-gaming rules.

Topics and coverage

benchmarks

What it means: define benchmarks clearly and connect it to the module focus: Evaluation crisis: benchmarks, contamination, model-as-judge, red-teaming, human evals, and domain-specific metrics.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

contamination

What it means: define contamination clearly and connect it to the module focus: Evaluation crisis: benchmarks, contamination, model-as-judge, red-teaming, human evals, and domain-specific metrics.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

model-as-judge

What it means: place model-as-judge inside the AI system stack so learners know what problem it solves and what tradeoffs it introduces.
What to cover: inputs, outputs, system boundaries, evaluation criteria, cost or latency implications, and common failure cases.
Demonstration: use a diagram, small code sample, worksheet, or tool trace to make the mechanism visible.
Evidence of learning: learners compare two approaches and explain which one they would choose for a realistic constraint.

red-teaming

What it means: define red-teaming clearly and connect it to the module focus: Evaluation crisis: benchmarks, contamination, model-as-judge, red-teaming, human evals, and domain-specific metrics.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

human evals

What it means: define human evals clearly and connect it to the module focus: Evaluation crisis: benchmarks, contamination, model-as-judge, red-teaming, human evals, and domain-specific metrics.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

domain-specific metrics

What it means: define domain-specific metrics clearly and connect it to the module focus: Evaluation crisis: benchmarks, contamination, model-as-judge, red-teaming, human evals, and domain-specific metrics.
What to cover: the core concept, why it matters, what good usage looks like, and where learners are likely to misunderstand it.
Demonstration: give one simple example, one realistic example, and one failure or limitation example.
Evidence of learning: learners explain the topic in their own words and apply it to a small artifact or decision.

Practice and evidence of learning

Learners complete or discuss: Create a benchmark spec for a high-stakes domain and include anti-gaming rules.
Learners produce: Create a benchmark spec for a high-stakes domain and include anti-gaming rules.
Instructor checks for accuracy, practical usefulness, clear assumptions, appropriate human review, and fit with the course audience.
Learners revise once after feedback so the module contributes to the final project, portfolio, or playbook.

Minimum coverage before moving on

Learners can explain the module vocabulary without relying on tool-generated text.
Learners have seen one worked example, one hands-on application, and one limitation or failure case.
Learners know what must be verified, what data must be protected, and who remains accountable for the output.

Module 10. Future architecture workshop

Module focus: Future architecture workshop: from current failure to proposed research direction and deployment guardrails. Primary live activity or lab: Present a roadmap linking roadblock, architecture, evaluation, risk, and timeline.

Topics and coverage

from current failure to proposed research direction and deployment guardrails

What it means: place from current failure to proposed research direction and deployment guardrails inside the AI system stack so learners know what problem it solves and what tradeoffs it introduces.
What to cover: inputs, outputs, system boundaries, evaluation criteria, cost or latency implications, and common failure cases.
Demonstration: use a diagram, small code sample, worksheet, or tool trace to make the mechanism visible.
Evidence of learning: learners compare two approaches and explain which one they would choose for a realistic constraint.

Practice and evidence of learning

Learners complete or discuss: Present a roadmap linking roadblock, architecture, evaluation, risk, and timeline.
Learners produce: Present a roadmap linking roadblock, architecture, evaluation, risk, and timeline.
Instructor checks for accuracy, practical usefulness, clear assumptions, appropriate human review, and fit with the course audience.
Learners revise once after feedback so the module contributes to the final project, portfolio, or playbook.

Minimum coverage before moving on

Learners can explain the module vocabulary without relying on tool-generated text.
Learners have seen one worked example, one hands-on application, and one limitation or failure case.
Learners know what must be verified, what data must be protected, and who remains accountable for the output.

Core labs and builds

Failure taxonomy lab: hallucination, retrieval miss, instruction conflict, tool failure, reasoning error, safety failure.
Architecture tradeoff lab: RAG vs fine-tuning vs agents vs symbolic systems vs hybrid search.
Evaluation design lab: build a domain-specific benchmark and anti-cheating protocol.
Research roadmap lab: propose a future AI system that solves one bottleneck without ignoring safety.

Capstone

Develop a future AI architecture thesis for a domain such as education, law, medicine, coding, scientific research, operations, robotics, or finance. The thesis must name the current roadblock, proposed architecture, required data, evaluation method, likely failure modes, and governance needs.

Assessment design

Failure analysis memo.
Architecture comparison table.
Benchmark design assignment.
Final research roadmap presentation.

Recommended tools and datasets

LLM APIs, retrieval tools, reasoning benchmarks, agent frameworks, paper reading packs, evaluation templates, failure logs, architecture diagrams.

Instructor notes

This course should avoid both doom narratives and hype narratives. The strongest learning outcome is disciplined architectural thinking under uncertainty.

Instructor Build Checklist

Prepare one short demo for each module and one learner activity that creates a saved artifact.
Prepare examples that match the audience, local context, and likely tools learners can access.
Add a verification step to every AI-generated output: factual check, source check, data sensitivity check, and quality review.
Keep a running portfolio folder so each module contributes to the final project or learner playbook.
Reserve time for reflection on what the learner did, what AI did, what was checked, and what remains uncertain.