AI-401-paul.wilmoth-2026-08-20-01-13-31
Title of Course: Multimodal AI and Foundation Models
Prefix and Number AI 401
Semester and Year [tbd]
Number of Credit Hours: 3
Prerequisites: AI 240 Foundations of Artificial Intelligence; AI/CPTS 437 Introduction to Machine Learning; AI/CPTS 434 Neural Network Design and Application
Course Details
Day and Time: [tbd]
Meeting Location: [tbd]
Instructor Contact Information
Instructor Name: [tbd]
Instructor Contact Information: [office location, phone, email] [tbd]
Instructor Office Hours: [click here for best practices] [tbd]
TA Name: [tbd]
TA Contact Information: [office location, phone, email]: [tbd]
TA Office Hours: [click here for best practices] [tbd]
Course Description
AI 401, Multimodal AI and Foundation Models, examines how pretrained foundation models represent, align, integrate, and generate information across text, images, audio, video, and sensor data. Students implement and compare multimodal alignment and fusion, contrastive learning, cross-modal projection, vision-language modeling, retrieval, and model orchestration approaches. Through programming assignments and guided experiments, students prepare multimodal data and select, adapt, and integrate foundation models for specific applications. They evaluate systems using performance, robustness, efficiency, and human-centered measures while examining hallucinations, modality conflicts, bias, privacy, accessibility, safety, and societal implications. In a culminating project, students develop and evaluate an application involving at least two modalities, document its design decisions, results, risks, and limitations, and communicate their findings through a technical report and presentation.
Course Materials
Books: Lectures will be provided based on selected research papers and technical reports which are publicly available (e.g., on arXiv) and instructor-prepared lecture notes. All lectures are self-contained. No required textbook.
Other Materials: Selected research papers and supplementary materials will be provided through the course website
Fees: 0$
|
Course Learning Outcomes (students will be able to:) |
Activities Supporting the Learning Outcomes | Assessment of the Learning Outcomes |
|---|---|---|
|
1. Explain how multimodal and foundation models represent, align, integrate, and generate information across different modalities (6a, 7g). |
Lectures, guided readings, architecture discussions, and model demonstrations |
Quizzes and architecture-analysis assignment |
|
2. Prepare and represent data from multiple modalities, including text, images, audio, video, and sensors (1c, 1e, 6a). |
Data-preparation exercises and programming laboratories |
Multimodal data-processing laboratory |
|
3. Implement and compare multimodal alignment, fusion, contrastive learning, and cross-modal projection methods (2e, 2g, 6a, 6b, 6c). |
Guided experiments, model-comparison activities, and programming laboratories |
Multimodal modeling assignment |
|
4. Select, adapt, and integrate foundation models, incorporating multimodal retrieval and model orchestration where appropriate (1d, 2a, 2c, 2g, 6a, 6f). |
Model-selection exercises, adaptation laboratories, retrieval experiments, design workshops, and project checkpoints |
Model-adaptation assignment and final project implementation |
|
5. Evaluate multimodal AI systems using appropriate performance, robustness, efficiency, and human-centered measures (6b, 6c, 6d). |
Benchmark design, evaluation laboratories, error analysis, and case studies |
System-evaluation assignment and final project evaluation report |
|
6. Analyze the reliability, risks, and responsible-deployment implications of multimodal AI systems, including hallucinations, modality conflicts, bias, privacy, accessibility, safety, and societal impact (4a, 4f, 6e, 6f). |
Failure analysis, responsible-AI case studies, risk analysis, and model-card development |
Reliability and responsible-AI analysis |
|
7. Communicate clearly a multimodal AI system’s design, results, limitations, risks, and implications through technical documentation and presentation (3a, 3b, 3c, 3e, 6c, 6e). |
Technical writing, design reviews, peer feedback, and presentations |
Final technical report and project presentation |
Course Schedule
| Dates | Lesson Topic | Assignment or Activity | Assessment |
|---|---|---|---|
| Week 1[dates] |
Introduction to Multimodal AI and Foundation Models Multimodal systems, modalities, foundation-model lifecycle, representative applications, capabilities, limitations, and course-development workflow |
Course orientation, computing-environment setup, model demonstrations, and multimodal-system mapping | |
| Week 2[dates] |
Transformers and Language Pretraining Targeted review of tokenization, embeddings, attention, transformer architecture, pretraining, and transfer learning |
Guided analysis of pretrained language representations and transformer outputs | Formative knowledge check |
| Week 3[dates] |
Vision Transformers and Visual Representation Learning Image tokenization, patches, visual encoders, vision transformers, and pretrained representations |
Work with pretrained text, vision, and selected audio or sensor encoders; extract and visualize embeddings | Assignment 1 begins: Understanding Pretrained Multimodal Representations |
| Week 4[dates] |
Multimodal Alignment Cross-modal projection, paired data, similarity measures, shared representations, and alignment objectives |
Compare representations from at least two modalities and analyze their similarity and alignment structures | Assignment 1 due (10%); preliminary project idea |
| Week 5[dates] |
Contrastive Representation Learning Positive and negative pairs, contrastive objectives, temperature, sampling, and representation quality |
Conduct guided contrastive-learning experiments using a small paired dataset | Assignment 2 begins: Contrastive Alignment and Mini-CLIP |
| Week 6[dates] |
CLIP: Contrastive Multimodal Pretraining CLIP architecture, image–text retrieval, zero-shot classification, capabilities, limitations, and failure cases |
Perform retrieval and zero-shot classification experiments using a pretrained CLIP model | Assignment 2 progress review |
| Week 7[dates] |
Working with a Shared Multimodal Representation Space Projection heads, lightweight encoders, shared embedding spaces, alignment quality, and retrieval evaluation |
Implement a mini-CLIP model on a small paired dataset; evaluate retrieval and zero-shot performance | Assignment 2 due (10%) |
| Week 8[dates] |
Multimodal Fusion Early, intermediate, late, and tensor-based fusion; comparison of alignment and fusion approaches |
Compare fusion methods and synthesize concepts from the first half of the course | Exam 1 (15%) |
| Week 9[dates] |
Attention-Based Multimodal Fusion Cross-attention, co-attention, encoder–decoder architectures, connectors, and multimodal fusion design |
Architecture-design workshop, model comparison, project planning, and peer feedback | Final project proposal and design review |
| Week 10[dates] |
From Vision-Language Models to Multimodal Foundation Models Vision-language architectures, multimodal language models, instruction tuning, retrieval, and model orchestration |
Analyze and compare representative multimodal foundation-model architectures and workflows | Assignment 3 begins: Adaptation and Integration of a Pretrained Multimodal Foundation Model |
| Week 11[dates] |
Adapting Multimodal Foundation Models Frozen pretrained components, projectors, prompting, parameter-efficient fine-tuning, LoRA, and resource-aware adaptation |
Connect pretrained components or adapt an existing multimodal model using a small captioning, retrieval, or visual-question-answering dataset | Assignment 3 implementation checkpoint |
| Week 12[dates] |
Multimodal Applications Multimodal retrieval, visual question answering, captioning, multimodal search, retrieval-augmented systems, model orchestration, and application integration |
Complete and demonstrate the adapted multimodal application; evaluate initial results | Assignment 3 due (10%) and final project implementation checkpoint |
| Week 13[dates] |
Evaluation and Failure Modes Task-specific metrics, retrieval metrics, grounding, faithfulness, hallucinations, modality ablation, modality conflict, robustness, efficiency, and human-centered evaluation |
Design an evaluation protocol and conduct controlled error and failure analysis | Assignment 4 begins: Multimodal Reliability, Evaluation, and Responsible-Deployment Audit |
| Week 14[dates] |
Beyond Image and Text and Responsible Deployment Audio, video, and sensor modalities; bias, privacy, accessibility, safety, efficiency, resource use, and societal implications |
Evaluate a multimodal system under modality-ablation, conflict, hallucination, and responsible-deployment tests; prepare a model card | Assignment 4 due (10%) and final project readiness review |
| Week 15[dates] | Project Presentations and Course Synthesis | Demonstrate and defend a multimodal application involving at least two modalities; participate in peer review | Final project presentation, demonstration and submission (30% project category) |
| Final Examination Period[dates] | Exam 2 (15%) |
Expectations for Student Effort
Students should expect to spend approximately nine hours per week on this three-credit course, including three hours of scheduled class time and approximately six hours of work outside class. Outside work will include assigned readings, programming exercises, guided experiments, preparation for examinations, and development of the multimodal application project. The time required may vary by week, with additional effort expected near major assignment and project deadlines.
Grading
| Assessment | Percentage |
|---|---|
| Assignment 1: Understanding Pretrained Multimodal Representations | 10% |
| Assignment 2: Contrastive Alignment and Mini-CLIP | 10% |
| Assignment 3: Adaptation and Integration of a Pretrained Multimodal Foundation Model | 10% |
| Assignment 4: Multimodal Reliability, Evaluation, and Responsible-Deployment Audit | 10% |
| Exam 1 | 15% |
| Exam 2 | 15% |
| Final Multimodal Application Project | 30% |
| Total | 100% |
Grading Schema
| Grade | Percentage | Grade | Percentage |
|---|---|---|---|
| A | 93–100% | C | 70–76% |
| A− | 90–92% | C− | 67–69% |
| B+ | 87–89% | D | 60–66% |
| B | 83–86% | F | Below 60% |
| B− | 80–82% | ||
| C+ | 77–79% |
The final course percentage will be calculated using full precision and rounded once to the nearest whole number. A fractional portion of 0.50 or higher will be rounded up, while a fractional portion below 0.50 will be rounded down. For example, 89.50% will be rounded to 90% and will earn an A−, while 89.49% will be rounded to 89% and will earn a B+.
Attendance and Make-Up Policy
Students should make all reasonable efforts to attend all class meetings. However, in the event a student is unable to attend a class, it is the responsibility of the student to inform the instructor as soon as possible, explain the reason for the absence (and provide documentation, if appropriate), and make up class work missed within a reasonable amount of time, if allowed. Missing class meetings too often may result in reducing the overall grade in the class.
Academic Integrity Statement
You are responsible for reading WSU's Academic Integrity Policy, which is based on Washington State law. If you cheat in your work in this class you will:
-Have zero credit
-Be reported to the Center for Community Standards
-Have the right to appeal my decision
-Not be able to drop the course of withdraw from the course until the appeals process is finished
If you have any questions about what you can and cannot do in this course, ask me.
If you want to ask for a change in my decision about academic integrity, use the form at the Center for Community Standards website. You must submit this request within 21 calendar days of the decision.