Instructor: Wu-chang Feng
Office hours: here
Required textbook


Week Topic Slides In-class Lab Homework
1 Introduction
Basic Analysis
Chapter 1: Basic Static Techniques
Chapter 2: Malware Analysis in VMs
Chapter 3: Basic Dynamic Analysis
1-1, 1-2
Install course VM
2 Advanced Static Analysis
Chapter 4: x86 Assembly
Chapter 5: IDA Pro
Ch01, Ch03
Chapter 6: C code in Assembly
Chapter 7: Malicious Windows Programs

6-1, 6-2

4 Advanced Dynamic Analysis
Chapter 8: Debugging
Chapter 9: OllyDbg
Malware Functionality
Chapter 11: Malware Behavior
Chapter 12: Covert Launching
Chapter 13: Data Encoding

12-1, 12-3

Chapter 14: Network Signatures
Chapter 15: Anti-Disassembly

15-1, 15-2
Ch11, Ch12, Ch13
Final project: MetaCTF walkthrough
Chapter 16: Anti-Debugging
Chapter 17: BluePill talk #1
BluePill talk #2
Anti-VM Techniques
talk #1

talk #2

Advanced Software Armoring
Chapter 18: Packers and Unpacking
Special Topics
Chapter 19: Shellcode Analysis


19-2, 19-3
Chapter 20: C++ Analysis
Final project work
talk 20-1 Ch18
10 (Tuesday) Final project work

Final project due @11:59pm
10 (Thursday) Final CTF

Finals week (Thursday)

Lab notebook, homeworks due at noon


Lab notebook
You will pair up with a lab partner to perform the in-class lab. As part of the lab work, you and your partner will maintain a shared lab notebook (a single Google Doc, Microsoft Office, or LibreOffice file) that will contain your write-ups of each lab. The write-up should include answers to questions asked and screenshots of the completed work (via gnome-screenshot, gimp, Print Screen, etc.). The notebook will be graded based upon thoroughness and clarity of the write-ups. You may also choose to work on labs individually.

Homework (MetaCTF)
For homework, we will be applying the concepts learned in the labs to Linux binaries. Assignments are to be done individually at the following site: Binaries are unique to each student and no collaboration is allowed. The binaries implement a set of capture-the-flag challenges that require you to reverse engineer a set of binary executables. Each binary asks for a password that will unlock it and print "Good Job". While you will be running binaries on your own machine, answers to each should be submitted at the above site. The homework is intended to give you practice for the final exam CTF.

Final project (Option #1: MetaCTF level)
"See one, Do one, Teach one". The goal of this project will be to develop your own metamorphic challenge that can be used to help someone learn a topic covered in this course that is not currently addressed in your homework assignment. Source code for several of the early MetaCTF challenges will be given as a template. We will solve each others' projects after the final exam. At a minimum, levels should:
  • Be tied to a single topic or technique. Levels that span multiple techniques are not recommended.
  • Be generated metamorphically using methods equivalent to the current MetaCTF challenges.
  • Follow the same format as the current MetaCTF challenges with each binary producing the flag "Good Job." upon successful completion.
The rubric can be found here. Projects are to be submitted as a zipfile via the corresponding D2L dropbox folder.

Final project (Option #2: Malware RCE)
The goal of this final project will be for you to reverse-engineer a piece of malware of your choice using everything you have learned in this course. After reverse-engineering the malware, you are to record a screencast walkthrough that steps through your analysis. Tools for doing the screencast can be found at PSU's Media Space. To submit a screencast, simply add it to the "Malware" channel on Media Space.

Final CTF
A final CTF will be run on the last day of class consisting of two parts. The first, graded part will be the exam CTF consisting of several CTF levels that are similar to the homework CTF. This is to ensure that you have mastered the knowledge and skills the course is attempting to provide. When students complete the first part, they will go on to the second part, in which students will attempt to solve the final project CTF levels submitted by others.

Course objectives

  • Understand the underlying mechanisms used by viruses, worms, trojans, backdoors, and rootkits.
  • Understand vulnerabilities that malware exploits to compromise end hosts.
  • Understand techniques malware uses to circumvent and evade detection
  • Understand counter-measures that prevent the spread of malware
  • Understand counter-measures that detect malware


Attendance 10%
Homeworks 30%
Lab notebook 30%
Final project 15%
Final CTF 15%
Academic misconduct
  • Includes allowing another student to copy your work unless specifically allowed by the instructor.
  • Results in a grade of 0 for the assignment or exam.
  • Results in the initiation of disciplinary action at the university level.