Teaching As Research

CMU’s Eberly Center for Teaching Excellence has outstanding resources  to support the faculty in their education research endeavors. They advocate an approach called Teaching as Research (TAR) that combines real-time teaching with on-the-fly research in education, for example to evaluate the effectiveness of a new teaching strategy while applying the strategy in a classroom setting.

TAR Workshops

Eberly Center’s interactive TAR workshops helps educators identify new teaching and learning strategies to introduce or existing teaching strategies to evaluate in their courses, pinpoint potential data sources, determine proper outcome measures, design classroom studies, and navigate ethical concerns and the the Institutional Review Board (IRB) approval process. Their approach builds on seven parts, each part addressing central questions:

  1. Identify a teaching or learning strategy that has the potential to impact student outcomes. What pedagogical problem is the said strategy trying to solve?
  2. What is the research question regarding the effect of the strategy considered on student outcomes? Or what do you want to know about it?
  3. What teaching intervention is associated with the strategy that will be implemented in the course as part of the study design? How will the intervention incorporate existing or new instructional techniques?
  4. What sources of data (i.e., direct measures) on student learning, engagement, and attitudes will the instructors leverage to answer the research question?
  5. What study design will the instructors use to investigate the research question?  For example, will collecting data at multiple times (e.g., pre- and post-intervention) or from multiple groups (e.g., treatment and control) help address the research question?
  6. Which IRB protocols are most suitable for the study? For example, different protocols are available depending on whether the study relies on data sources embedded in normally required course work, whether student consent is required for activities not part of the required course work, and whether any personal information, such as student registrar data, is needed.
  7. What are the actionable outcomes of the study? How will the results affect future instructional approaches or interventions?

After reviewing relevant research methods, literature, and case studies in small groups to illustrate how the above points can be addressed, each participants identifies a TAR project. The participants have a few months to refine and rethink the project, after which the center folks follow up to come up with a concrete plan in collaboration with the faculty member.

Idea

I teach a graduate-level flipped-classroom course with colleague Cécile Péraire on Foundations of Software Engineering.  We have been thinking about how to better incentivize the students to take assigned videos and other self-study study materials more seriously before attending live sessions. We wanted them to be better prepared for live session activities and also improve their uptake of the theory throughout the course. We had little idea about how effective the self-study videos and reading materials were. Once suggestion from the center folks was to use low stakes assessments with multiple components, which seemed like a good idea (and a lot of work). Cécile and I set out to implement this idea in the next offering, but we wanted to also measure and assess its impact.

Our TAR project

Based on the above idea, our TAR project, in terms of the seven questions, are summarized below.

  • Learning strategy: Multi-part, short low-stakes assessments composed of an online pre-quiz taken by student just before reviewing a self-study component, a matching online post-quiz completed by student right after reviewing the self-study component, and an online in-class quiz on the same topic taken at the beginning of the next live session. The in-class quiz is immediately followed by a plenary session to review and discuss the answers.  The assessments are low-stakes in that a student’s actual quiz performance (as measured by quiz scores)  do not count towards the final grade, but taking the quizzes are mandatory and each quiz completed counts towards a student’s participation grade.
  • Research question: Our research question is also multi-part. Are the self-study materials effective in conveying the targeted information? Do the low-stakes assessments help students retain the information given in self-study materials?
  • Intervention: The new intervention here are the pre- and post-quizzes. The in-class quiz simply replaces and formalizes an alternate technique based on online polls and ensuing discussion used in previous offerings.
  • Data sources: Low-stakes quiz scores, exam performance on matching topics, and basic demographic and background information collected through a project-team formation survey (already part of the course).
  • Study design: We used a repeated-measures, multi-object design that introduces the the intervention (pre- and post-quizzes) to pseudo-randomly determined rotating subset of students. The students are divided into two groups each week: the intervention group A and the control group B. The groups are switched in alternating weeks. Thus each student ends up receiving the intervention in alternate weeks only, as shown in the figure below. The effectiveness of self-study materials will be evaluated by comparing  pre- and post-quiz scores. The effectiveness of the intervention will be evaluated by comparing the performance of the control and intervention groups during in-class quizzes and related topics of course exams.

  • IRB protocols: Because the study relies on data sources embedded in normally required course work (with the new intervention becoming part of normal course work), we guarantee anonymity and confidentiality, and students only need to consent to their data being used in the analysis, we used an exempt IRB protocol applied to low risk studies in an educational context. To be fully aware of all research compliance issues, we recommend that anyone pursuing this type of inquiry consult with the IRB office at their institution before proceeding.
  • Actions: If the self-study materials are revealed to be inadequately effective, we have to look for ways to revise them and make them more effective, for example by shortening them, breaking them into smaller bits, adding examples or exercises, or converting them to traditional lectures. If the post-quizzes do not appear to improve retention of self-study materials, we have to consider withdrawing the intervention and trying alternative incentives and assessment strategies. If we get positive results, we will retain the interventions, keep measuring, and fine-tune the strategy with an eye to further improve student outcomes.

Status

We are in the middle of conducting the TAR study. Our results should be available by early Spring. Stay tuned for a sneak peek.

Acknowledgements

We are grateful to the Eberly Center staff Drs. Chad Hershock and Soniya Gadgil-Sharma for their guidance and help in designing the TAR study. Judy Books suggested the low-stakes assessment strategy. The section explaining the TAR approach is drawn from Eberly Center workshop materials.

Further Information

For further information on the TAR approach, visit the related page by  Center for the Integration of Research, Teaching and Learning. CIRTL is an NSF-funded network for learning and teaching in higher education.

Video Lectures: The Good, the Bad, and the Ugly

In my previous post on flipped classroom, I touched on a key benefit of this approach: Immediate faculty feedback during in-class activities enabling​ rapid and effective​ learning.

In this post, I will cover video lectures: The videos that students watch online before coming to class, in order to prepare for in-class activities. We’ll look at the good, the bad, and the ugly of video lectures.

Let’s start with the “good” 

Most students appreciate online videos, because they can watch them (potentially repeatedly) at their own time and pace. Students like the fact that the videos are short and focused on teaching them the key concepts to remember before class.

As faculty, we also appreciate those videos, because they reduce our preparation time before each class, every semester. Indeed, they eliminate the need to review a large slide deck before class in order to get ready for a long monolog. Instead, during class, students do most of the talking and thinking by solving problems related to the concepts introduced during the videos. Faculty preparation is mostly reduced to remembering how to introduce those problems to the students, facilitate the problem-solving activity, and highlight the activity takeaways.

Video lectures have their drawbacks, so let’s continue with the “bad”

Producing and maintaining video lectures could be extremely time consuming. Below are some advices (taken from Flipping a Graduate-Level Software Engineering Foundations Course) that we received from mentors who helped us produce videos for our Foundations of Software Engineering course:

  • Aim for “good enough”. Shooting perfect videos could take days if one aims for the perfect background, angle, lighting, audio, elocution, timing, etc. Even-though all these elements are important, imperfection in the context of video lectures is perfectly acceptable. Hence the video production process could be accelerated greatly by aiming for “good-enough”.
  • Keep videos short and focused. Videos should be created to retain students’ attention and maximize learning: they should be kept short (e.g. about 10 minutes at most) and convey a limited number of concepts and key messages. The key messages should be easy to summarize at the end.
  • Include required elements. Elements that should be included in a video are: a (catchy) opening with motivation, agenda, learning objectives, and summary of key messages.
  • Favor pictures over text. Prefer graphics and pictorials over text in visuals.
  • Ask for participation. A video lecture may encourage active participation of the viewer. For example it may pose a question and ask the viewer to pause and ponder the question or solve a problem.
  • Assess understanding during live sessions: Because a faculty is not present when students consume online videos, it is hard to assess students’ understanding of the content. To overcome this challenge, we often start a class (also called live session) with a Q&A to clarify or complement the content of video lectures. In addition, incorporating graded quizzes to Q&A sessions might help “motivate” students to watch videos.

Here is where it gets “ugly”

Because video lectures are initially long to create and later on difficult to maintain, they have the tendency of freezing the course content. Here are some advices (taken from Flipping a Graduate-Level Software Engineering Foundations Course) to address this problem:

  • Favor principles over fashion. Videos should focus on principles and foundational concepts versus technology and fads to maximize their relevance in fast-evolving subjects. Keep timeless components in; remove volatile components that are likely to become stale. These volatile components could be introduced during live sessions using mini-lectures (e.g. short tech-talk on how to use git) for instance.
  • Stabilize before recording. Video lectures should ideally be created once the content has been tested and stabilized. Unfortunately, we could not follow this advice. We were designing the course almost entirely from scratch, and took many risks with untested content. We later had to revisit and edit existing videos to make changes (which was extremely time-consuming). We also had to eliminate content that did not work. Be prepared to rework or trash some portion when designing a flipped classroom from scratch.

Conclusion for faculty: Is flipped-classroom right for you?

If being an activity facilitator makes you uncomfortable, you might want to stay away from flipped-classroom. Otherwise, do not let the “bad” and the “ugly” discourage you. If like me, you are not fond of slide presentations but deeply enjoy facilitating workshops, this teaching approach could clearly make teaching easier and more pleasurable. Also, note that it is very possible to replace video lectures with selected readings and videos made by others. That way you retain the benefits of flipped-classroom without the drawbacks of video lecture production.

A Toolkit for Pedagogical Innovation in Software Engineering

I was asked to share some of the pedagogical innovations from two books I recommended once during a talk: Pedagogical Patterns and Training from the Back of the Room. In this post I will focus on Pedagogical Patterns, and leave Training from the Back of the Room for my next post. I will provide an overview of the book and share the insights that I have put in practice. I hope you will be tempted to read them and apply some of these ideas.

Pedagogical Patterns, Advice for Educators, is a collection edited by Joe Bergin with the help of a board of editors, including Jutta Eckstein and Helen Sharp, and the result of the collaborative work of many authors. The book is applicable beyond software engineering, but most of the examples deal with computer software issues. One of the things that resonated with me the most while reading this book is the loving warmth that the writer/teacher’s voice conveys for the work of teaching, love of both students and the experience of teaching itself.
The book is organised as a pattern catalog, following the Alexandrian format, and also grouped according to their main theme (active learning, feedback, experiential learning, different perspectives). Here are some of the patterns that have impacted me the most:

* Abstraction Gravity — Fron High to Low: Introduce concepts that require understanding on two levels of abstraction at the higher level of abstraction and then at the lower level. We apply this pattern in several forms in our first programming course at UNTREF. First, we apply it in the context of Meyer’s Inverted Curriculum, so that we focus first on understanding and using objects, and then we move on to implementing them. Also, we go from a very abstract use of Java objects using a dynamic interpreter to a more static compiled environment as the course progresses. Some of the issues in this approach are making sure that the link between the two abstraction levels is clear, and that more than one example is described.

* Active Student: Keep students active and engaged in and outside the classroom to make it more likely that they will learn. This means shifting some activities from listening and reading to speaking, writing, solving problems and interacting. The Different Approaches related pattern promotes organising activities with varied focus among different sensory modalities: visual, auditory and kinesthesia, that cater to different types of people. We find that active students are more likely to participate during the course, and that is why we make sure that during the first and second classes students are actively engaged physically and mentally (we go out of the classroom, separate in “tribes” throughout the classroom, and solve minor puzzles).

* Early Warning: Provide early feedback on students falling behind or not having a clear understanding of issues that might impact them when future activities need to build on previous knowledge. After the first few weeks of our programming course, we start working on exercises. We apply this pattern by explicitly marking the expected rhythm of progress on the exercises by publicly stating which exercise workbook corresponds to that week. For this to actually have impact, we make visible how many students have solved a given amount of exercises on the current workbook. In order to do this. we ask students to raise their hand when we say out loud the number of exercises from the current workbook they have solved up to that point. We count hands for 0, 1, 2, etc. up to the number of exercises in that workbook. This not only marks which one is the current workbook that they should be working on, but provides gentle peer pressure on students when they see each other’s progress. We have measured noticeable improvement in the amount of exercises performed on the current guide as the course progresses.

The core tenets I see behind these patterns are a caring look on students and an effort to keep then connected to the learning experience. These are only a few examples of the patterns you will find in the book and how you can apply them.

I would love to hear your comments on your own experiences and how these ideas resonate with you.

Flipped Classroom versus Online Learning

Flipped classroom was a hot topic during the SECM workshop. It started with a paper by Nicolas Paez on A Flipped Classroom Experience Teaching Software Engineering, and continued with many exciting discussions throughout the day.

In parallel to the workshop, Hakan and I presented a paper at ICSE’17 on Flipping a Graduate-Level Software Engineering Foundations Course based on our experience co-designing and co-teaching the course over several semesters. In this blog, I would like to distill some of the tips from the paper, so others have the opportunity to discuss related experiences.

But before we start, let me share my suprise when I noticed that our ICSE paper on flipped classroom was part of a session called “Online learning”.  Online learning?

Yes, true, in flipped classroom students have to watch videos “online” before coming to class. But the parallel with online education stops here. In fact, flipped classroom is the antithesis of online learning, in the sense that most of the learning happens within the classroom, where students come physically to perform in-class activities under the guidance of a faculty​.

Most of the value of flipped classroom comes from immediate feedback that students receive when trying to solve a problem right in front of a faculty. Immediate feedback enables​ rapid and effective​ learning. It is specially important in fields like software engineering, where nothing is black or white, everything has nuances, and there are exceptions to pretty much every rule. In such a context, forget about having “robots” answering students’ questions online! And who wants to wait 24 hours for a human being to get back to a question posted online? The most effective way (that I know of) of​ creating​ short feedback loop​s​​ and generating rapid and effective learning​ ​i​s ​with​ face-to-face interaction​s​ between students and faculty.​ This is what flipped classrooms are all about!