All You Need is Love … And a Rubic, Part II

By Yael Silk

What does a love of learning look like? Can you observe it in a classroom? What is unique about a love of Jewish learning? How might you measure this in a religious school setting? These are only some of the questions that educators and volunteers at the Temple David Weiger Religious School in Monroeville, PA grappled with as they worked on developing a range of student assessments this past year.

After an unsuccessful search for existing student assessments that would meet the religious school’s needs, the Temple David team committed to designing original assessments with guidance and support from the Agency for Jewish Learning (AJL) in Pittsburgh. These efforts were an outgrowth of AJL work focusing on school improvement. In an eJewish Philanthropy article published in November 2013, we shared the journey this group took going from school program objectives to determining what measuring change would look like. The driving question for this work was – how would we know if the changes we are making were making a difference for our students?

This follow-up article focuses on how one of these assessments transitioned from page to stage, ready for full implementation for the 5775 school year. The most difficult program objective to address through assessment was, every child will display a love of learning about Judaism. Based on a textual analysis of Pirkei Avot, the volunteer Assessment Team designed a classroom observation protocol under the guidance of an AJL assessment consultant. Ultimately, the Love of Learning assessment included the following components:

• A training unit for all volunteer raters
• An observation protocol (specific instructions for before, during, and after the observation)
• A rubric clearly articulating three Love of Learning criteria across a continuum
• A classroom observation form for volunteer raters to use during their observations and submit immediately following the observation
• A teacher questionnaire to capture the faculty’s perspectives of Love of Learning in their classrooms

The final versions of the items above are all available on Temple David’s website.

The Love of Learning assessment pilot included the following four criteria:

1. Readiness to learn – being a mensch
   The basics are in order. This is about classroom behavior and readiness to learn.
2. Seeking wisdom and knowledge
   The student is responsible for seeking answers, seeking a teacher. It’s not only the taller person in the front of the room – anyone can be a teacher.
3. Active peer learning
   Based on the idea of chavrutah. We learn best by pilpul (lively back and forth debate).
4. Learning and doing
   There is a flow of communication between home and classroom in terms of how this is getting translated – e.g., if they learn about Shabbat, they do some observances at home.

The Assessment Consultant led a training session for all volunteer raters in February 2014. This training included a thorough review of the Love of Learning rubric and observation form. After reviewing each criterion in depth, the group viewed a ten-minute video segment of a Temple David class. During the video viewing, all participants took independent notes using the observation form and then submitted scores to the Assessment Consultant. The group then had a discussion beginning with the raters who gave the highest scores and practiced using observational evidence to select a rubric score. Once the group determined a consensus score for each rubric criterion, they were ready to pilot the observation protocol in Temple David classrooms.

The team of five raters completed a total of 12 classroom observations from mid-March through late April 2014. Each of the six faculty members was observed twice. The raters completed these observations in pairs (different combinations of raters observed different lessons), filled out their observation worksheets independently, and submitted their rubric scores directly to the Assessment Consultant. The consultant reviewed each observation worksheet to see if the comments supported the scores and to determine the reliability of the rubric (to what extent two raters can independently come up with the same, or nearly the same scores). The rater agreement results are in the table below:

Raters scored the active peer learning with the highest level of agreement with raters assigning the exact same score nearly half of the time. In addition to active peer learning, raters also scored readiness to learn and seeking wisdom and knowledge reliably (either in complete agreement or a 1-point difference) more than 80% of the time. The weakest rubric criterion was learning and doing. Raters scored this criterion reliably less than 60% of the time and even scored with a 3-point difference 18% of the time (i.e., in these cases one rater scored a 1 and the other a 4, the extremes of the rubric).

The Assessment Consultant reviewed and discussed these pilot findings with the faculty and again at a later date with the Education Director. The ultimate conclusion was that classroom observation was not the most appropriate method for measuring learning and doing. In addition to being a methods issue, the Education Director realized that learning [and] doing aligned more closely with a different school program objective: All students will have the knowledge and skills to proudly practice their Jewish faith. The faculty responded to this new understanding with a commitment to brainstorm with the Education Director around ways to create a continuous dialogue among teachers, students, and their family members about connecting Jewish learning to Jewish practice.

This process of designing student assessments for a part time religious school has been both incredibly rewarding and challenging. The following are some of the lessons learned for those who are interested in brining this work to their own schools:

• Developing assessments around complex objectives is a time consuming and iterative process. It is difficult to speed things up and requires regular revisiting of past work with fresh eyes and consensus building among all involved stakeholders.
• The part time nature of religious school exacerbates some of time limitation and communication challenges. Working on a deeply nuanced and detailed project like this over a long period of time with volunteer and part time staff requires significant communication via document sharing, emails, phone calls, and in person meetings. Meetings are often spread out and typically require beginning with a review of decisions made at the previous gathering.
• The faculty advocated for developing a more immediate feedback loop so that they could quickly debrief with one of the raters in-person immediately following the observation in addition to getting timely feedback (within two weeks) on their rubric performance from the Education Director.
• It is essential that all stakeholders know why the volunteers are observing classrooms and what their roles are during these observations. These stakeholders include: the Education Director, faculty, volunteer raters, high school teacher assistants, students, and parents.

After more than a year of ongoing development, the faculty at Temple David is ready to start the 5775 school year with a full Love of Learning assessment plan in place. They will use ongoing data from this effort to reflect on their instructional practice and gauge their progress towards instilling a love of Jewish learning in all of their students.

Those interested in learning more about these efforts can contact Nicolette Canterna at ncanterna@ajlpittsburgh.org or (412) 521-1101 x3207.

Yael Z. Silk is an Educational Consultant at Pittsburgh’s Agency for Jewish Learning. She is the founder of Silk Strategic Arts, an independent consultancy focusing on program evaluation, student assessment, facilitation, and strategic planning in the education and arts fields. She can be reached at yael@silkstrategicarts.com.