Below is a syllabus template that includes WSU's required syllabus elements. Please complete all items highlighted in yellow. 

 

Title of Course [Reinforcement Learning]

Prefix and Number [AI / CPT_S 474 / 574]

Semester and Year [tbd]

Number of Credit Hours [3]

Prerequisites [For CPT S 474/AI474—CPT S 437/AI 437 with a C or better; MATH 220 or 225 with a C or better; STAT 360 with a C or better. For CPT S 574-- CPTS 570

Course Details

Day and Time: [tbd]

Meeting Location: [tbd]

 

Instructor Contact Information

Instructor Name: [tbd]

Instructor Contact Information: [office location, phone, email] [tbd]

Instructor Office Hours: [click here for best practices] [tbd]

 

TA Name: [tbd]

TA Contact Information: [office location, phone, email]: [tbd]

TA Office Hours: [click here for best practices] [tbd]

 

Course Description

This course provides a comprehensive introduction to reinforcement learning and sequential decision making.
Students study how an agent learns to make decisions through interaction with an environment and how
reward, uncertainty, exploration, function approximation, and constraints affect learning. Topics include Markov
decision processes, dynamic programming, value iteration and policy iteration, Q learning, SARSA, temporal
difference learning, linear and nonlinear function approximation, deep Q learning, policy gradient methods,
deterministic policy gradient methods, multi armed bandits, optimistic learning, constrained and safe
reinforcement learning, AlphaZero, online and offline reinforcement learning, imitation learning, inverse
reinforcement learning, and reinforcement learning with human feedback. The course combines mathematical
reasoning, algorithm implementation, empirical evaluation, and a final project.

Rationale for the Conjoint Offering

Reinforcement learning provides a common technical foundation for advanced undergraduate and graduate study in artificial intelligence, machine learning, optimization, and sequential decision making. The core lectures, demonstrations, and implementation activities are appropriate for both advanced undergraduate and graduate students and benefit from a shared classroom environment. The conjoint offering allows CPT_S 474 / AI 474 and CPT_S 574 students to engage with the same reinforcement learning foundations while maintaining clear academic differentiation. Graduate students complete additional research oriented analysis and are held to higher expectations for mathematical depth, independent reasoning, engagement with current literature, experimental rigor, and research quality project work.

Differences between CPT_S 474 and CPT_S 574

Dimension	CPT_S 474	CPT_S 574
Learning expectations	Explain, derive, implement, compare, and evaluate established reinforcement learning methods and apply them to well defined problems.	In addition to the 474 expectations, analyze more advanced formulations and recent methods, justify assumptions and technical choices, and reason about limitations, failure modes, and research questions.
Assignments	Complete the common mathematical, implementation, and empirical components of four homework assignments.	Complete the common components plus designated graduate extensions requiring additional derivation, theoretical analysis, or deeper empirical evaluation. Graduate students also complete required research reading tasks.
Course project	Complete an applied or comparative reinforcement learning project, typically in a team of up to two students.	Complete an individual research oriented project with stronger literature engagement, technical contribution, controlled evaluation, and explicit limitations analysis.
Assessment	Evaluated on correctness, understanding of core theory, implementation quality, analysis, and technical communication.	Evaluated on the same dimensions with additional emphasis on mathematical depth, independence, research reasoning, and experimental rigor.
Class interaction	Participate in common lectures, technical discussions, and project presentations.	Participate in the same shared activities and contribute graduate level perspectives when discussing research papers, assumptions, and open problems.

 

Course Materials 

Books: 

No textbook is required. Recommended references include:

• Richard Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• Dimitri Bertsekas, Reinforcement Learning and Optimal Control, Athena Scientific, 2019.

Other Materials: Students will use Python, PyTorch, Google Colab, instructor provided notebooks, and selected research papers. Course materials and assignment instructions will be distributed through Canvas.

Fees: No additional course specific fees are proposed.

Student Learning Outcomes (SLOs) [add more lines if necessary]

Course Learning Outcomes (students will be able to:)	Activities Supporting the Learning Outcomes	Assessment of the Learning Outcomes
1. Formulate sequential decision problems using Markov decision processes, Bellman equations, value functions, and policies (1a, 1b, 1c, 6a).	Lectures, guided derivations, worked examples	HW1, Midterm 1
2. Derive, implement, and compare dynamic programming, Q learning, SARSA, temporal difference, and value based reinforcement learning methods (1d, 1e, 2e, 2g, 6a, 6b, 6c).	Programming exercises, algorithm analysis, controlled experiments	HW1, HW2, Midterm 1
3. Analyze function approximation, deep Q learning, policy gradient, and deterministic policy gradient methods (1d, 1e, 6a, 6b, 6c).	Implementation exercises, mathematical analysis, empirical comparison	HW2, HW3, Midterm 2
4. Explain exploration, multi armed bandits, optimistic learning, and constrained or safe reinforcement learning (1c, 6a, 6e).	Lectures, case studies, algorithm comparison	HW3, Midterm 2
5. Explain modern reinforcement learning settings including AlphaZero, online learning, offline learning, imitation learning, inverse reinforcement learning, and reinforcement learning with human feedback (6a, 7g).	Lectures, paper discussion, project work	HW4, research reading for 574, project
6. Design and evaluate a reinforcement learning study using appropriate baselines, metrics, analysis, and technical communication (2a, 2c, 2e, 3a, eb, 3c, 3d, 3e, 6b, 6c, 6d).	Project planning, experiments, presentation	Final project report and presentation

---

Course Schedule

[Please note that a WSU semester is 15 weeks + Thanksgiving/Spring Break. The schedule below does not include the break.]

Dates	Lesson Topic	Assignment	Assessment
Week 1 [dates]	Course overview; sequential decision making; Markov chains; states, actions, rewards, trajectories, and return		Project overview and example project directions
Week 2 [dates]	Markov decision processes; policies and value functions; Bellman expectation and optimality equations	HW1 assigned: MDPs and dynamic programming
Week 3 [dates]	Dynamic programming; policy evaluation; policy iteration; value iteration; principle of optimality
Week 4 [dates]	Linear programming view of MDPs; model free learning; Q learning and SARSA	HW1 due
Week 5 [dates]	Temporal difference learning; bootstrapping; on policy and off policy updates; convergence intuition	HW2 assigned: model free learning and function approximation	Project topic and short proposal due
Week 6 [dates]	Linear function approximation; feature representation; nonlinear approximation; stability and generalization considerations
Week 7 [dates]	Deep Q learning; experience replay; target networks; practical training issues	HW2 due	Midterm 1
Week 8 [dates]	Policy gradient methods; policy gradient theorem; REINFORCE; variance reduction and baselines	HW3 assigned: policy optimization and deep RL
Week 9 [dates]	Deterministic policy gradient methods; continuous action problems; comparison of value based and policy based methods		Project progress check
Week 10 [dates]	Multi armed bandits; exploration versus exploitation; optimism; optimistic Q learning	HW3 due
Week 11 [dates]	Constrained reinforcement learning; safe RL; reward versus constraint tradeoffs; evaluation of constraint satisfaction	HW4 assigned: modern and constrained RL
Week 12 [dates]	AlphaZero; planning and learning; value and policy prediction; search guided decision making		Midterm 2
Week 13 [dates]	Online reinforcement learning and offline reinforcement learning; data collection, distribution shift, and policy evaluation	HW4 due	Draft results and evaluation plan
Week 14 [dates]	Imitation learning and inverse reinforcement learning; learning from demonstrations and inferred objectives		Project work and presentation preparation
Week 15 [dates]	Reinforcement learning with human feedback; course synthesis; current directions in reinforcement learning		Final project report and presentation

Homework and Programming Assignments

The course will include four homework assignments. Each homework combines mathematical questions, algorithmic reasoning, implementation, and empirical analysis. The four assignments are planned to cover: (1) Markov decision processes and dynamic programming, (2) model free value based learning and function approximation, (3) deep reinforcement learning and policy optimization, and (4) exploration, constrained reinforcement learning, and selected modern topics. Programming will use Python, PyTorch, and Google Colab.

CPT_S 574 students complete the same core tasks plus designated graduate extensions. These extensions may require additional derivations, proof based reasoning, analysis of assumptions, more extensive experimental comparison, or interpretation of recent research results. The graduate extensions are part of the regular homework grade rather than optional bonus work.

 

Research Reading for CPT_S 574

Graduate students complete required reading and short analysis tasks based on selected recent reinforcement learning papers. Students are expected to identify the problem setting, assumptions, main method, key result, evaluation design, and limitations. Undergraduate students may use the same readings as optional enrichment or as background for their projects, but the reading component is formally assessed only in CPT_S 574.

Midterm Assessments

The course includes two midterm examinations. Both levels complete common questions that assess understanding of the mathematical foundations, core algorithms, and reasoning about reinforcement learning behavior. CPT_S 574 students also complete designated graduate questions requiring additional derivation, proof based reasoning, or analysis of less standard reinforcement learning settings.

Course Project

The final project integrates the course material and gives students an opportunity to investigate a reinforcement learning problem in greater depth. A short project proposal is submitted early in the semester, followed by a progress check and the final report and presentation.

CPT_S 474 / AI 474: The project may be completed by a team of up to two students. Appropriate projects include applying an established reinforcement learning method to a meaningful task, comparing several algorithms, or conducting an empirical study of an existing method. The final report is expected to be approximately six pages excluding references, together with code and a class presentation.

CPT_S 574: The project is completed individually and should be research oriented. Appropriate directions include a new theoretical result, a new algorithm or modification of an existing method, or a new application or dataset study with a clear technical question. The report is expected to be approximately ten pages excluding references and should include related work, technical approach, controlled evaluation, interpretation of results, limitations, and a clear statement of contribution. The final presentation includes technical discussion and questions.

 

Expectations for Student Effort 

[Describe how much time students should expect to invest in the course each week. Graduate courses should state: "For each hour of lecture equivalent, students should expect to have a minimum of two hours of work outside of class." Note that Global campus courses will automatically include credit hour equivalents in the syllabus.] [

For each hour of lecture equivalent, students should expect a minimum of two hours of work outside of class, including reading, homework, programming, project development, and exam preparation.

 

Grading [add more lines if necessary]

Assignment Breakdown

Type of Assignment (tests, papers, etc)	AI/CPTS 474	AI/CPTS 574
Four homework assignments	50%	35%
Research reading and analysis	Not separately weighted	10%
Two midterm exams	30%	20%
Course project	20%	35%
Total	100%	100%

For CPT_S 574, the lower homework and exam weights do not indicate lower expectations. Graduate students complete additional homework extensions and research reading, and the individual project carries a larger share of the course grade because it is used to assess independent graduate level analysis and research readiness.

Grading Schema

Grade	Percent	Grade	Percent
A	93 to 100	C	67 to less than 71
A-	90 to less than 93	C-	63 to less than 67
B+	85 to less than 90	D+	59 to less than 63
B	80 to less than 85	D	55 to less than 59
B-	75 to less than 80	F	less than 55
C+	71 to less than 75

[Provide information about how grades will be rounded (eg, if 89% earns a B+ and 90% earns an A-, what grade is given to a student with an 89.5?]

---

Late Submission and Regrade Policy

Unless an assignment states otherwise, work submitted within 24 hours after the deadline may receive up to 50 percent credit, and work more than 24 hours late receives no credit. Each student has one late homework pass that permits submission within 48 hours of the original deadline. Regrade requests must be submitted within three days after the solution or grade is posted and must explain why additional credit is warranted based on the correctness of the submitted work.

Attendance and Communication

Lectures are expected to be in person. Regular attendance is expected. Course materials, announcements, assignment information, and grades will be distributed through Canvas. Students should communicate questions about course content or logistics through the course communication channels or email. Office hours are available as announced by the instructor.

 

 

---

Academic Integrity Statement

You are responsible for reading WSU's Academic Integrity Policy, which is based on Washington State law. If you cheat in your work in this class you will:

Students are responsible for following the Washington State University Academic Integrity Policy. All submitted work must reflect the student’s own understanding and effort except where collaboration is explicitly permitted. Questions about acceptable collaboration should be directed to the instructor before submission.

Graduate Student Rights and Responsibilities

Students enrolled in CPT_S 574 are expected to be familiar with and follow the Graduate School policies on academic integrity, student rights, and responsibilities. Applicable policies are available through the WSU Graduate School.

Generative AI Statement

Generative AI tools may be used for general concept clarification or to search broad questions while studying, but they may not be used to write code or produce solutions for graded homework, programming assignments, or examinations unless the instructor explicitly permits their use for a specific task. Generative AI tools may not be used during exams. Students remain responsible for the correctness, originality, and explanation of all submitted work.