Strategies for a Successful Curriculum Survey: Building Evaluative & Curricular Thinking

Strategies for a Successful Curriculum Survey: Building Evaluative & Curricular Thinking

In the process of conducting evaluative work, you may want to think about how you are going to infuse learning process as you engage in assessment or evaluation. For example, in recent Discovery Experience Survey Project, the project team infused specific processes to build evaluative and curricular thinking--mindsets that incorporate intentional questioning of assumptions and valuing of systematic data-gathering for educational decision-making (evaluative thinking; Patton, 2012), and intentional curricular and instructional design that is guided by learning outcomes and principled reasoning for organizing learning experiences (curricular thinking). 

Strategies

Using survey planning to stimulate evaluative thinking

Strategy 1. Stimulate use from the get-go

Discuss what decisions and actions you are trying to inform/influence by learning from data. Gather reactions from leadership by listing potential decisions and actions. Create anevaluation design matrixthat shows the relationships between the intended uses of evaluation, evaluation questions, and survey questions. 

Strategy 2. Stimulate use before gathering data:Anticipate findings

Set expectations on what can come out of the report and give no surprises. Draft and have intended report users (VCUE, Deans) react to a mock-up survey reportprior to data gathering for the primary intended users of the report. This was done to modify the report language and format to readers’ expectations.  

Data collection as a socialization/educative tool for curricular thinking

Strategy 3. Challenge and model learning

We wanted the survey instrument to generate a positive backwash effect on faculty to think about educational design in a certain way--in our case, thinking deeply about Discovery Learning. We challenged the respondents to (collectively) describe mastery-level Discovery Learning outcomes and learning progression, but we also modeledways to describe learning. 

Strategy 4. Challenge and model learning

Make respondents think about learning and curriculum in a certain way. We constrained programs to think about curriculum in three-tier learning progression: Introductory, develop & refine, and mastery. 

Ensure usable data

Strategy 5. Obtain high response rate (Achieved 100 % response rate)
All communication strategies (pre-survey warning, invitation message, survey response rate summary to Deans, reminder messages, phone calls, etc.) 
are set in place, so we can obtain a high response rate by the closing date. Response rate achieved via: a “heads up” email four weeks before the survey launched, repeated email reminders from both Provost and Vice Chancellor, deadline extension, discussing survey status in Deans meetings so they had a stake in their departments responding, targeted emails to department managers and personal contacts/allies, repeated phone calls and voice messages.

Strategy 6. Obtain high quality response

Survey belief tied to action and decisions: (1) Chairs are motivated to see the value to their department in engaging with the survey (including that survey results will inform fundraising campaigns). (2) Deans and chairs understood that the EVCP and VCUE were invested in survey success and learning from responses. (3) The project didn’t come across as a top-down demand or something that would lead to “do more with less” mandates from above   

Checking & filling out missing responses: We reviewed every response and flagged departments with partial responses. We called and/or emailed the department chairs and managers to get responses directly, and entered the data for the department to avoid requiring them to go back to the instrument. 

Additional strategies for a successful campus survey project

Strategy 7. Forming an evaluation team

Bring the right talents around the table: Working group membership was intentionally formed with attention to expertise required to carry out the survey project, such as knowledge of the evolution of the Discovery Experience Initiative and Berkeley’s undergraduate curricular structures, as well as competencies in curricular design and program evaluation, survey design and implementation, quantitative and qualitative data analysis, campus communication, and project management.

Strategy 8. Monitor project progress and keep reasonable project pace 
Monitoring project progress and having regular check-ins to keep pace of the project are important success  for teams to stay on track and maintain a reasonable workload. Project manager and communications lead sat in the project sponsor’s office and was in most of the campus leadership meetings. This was helpful in quickly seeking and integrating feedback from sponsors and leadership and also understanding the nuances of politics inherent in such a wide-reaching and public project 

Strategy 9. Engage multiple stakeholders for buy-in

Multiple stakeholders were engagedin the survey development process to ensure content validity. The survey wasreviewed by the Council of Undergraduate Deansand pilot-testedwith three departments. A few distinguished chairs of undergraduate education(Berkeley Collegium) from various disciplines also provided feedback. These efforts ensured that the survey resonated with faculty and garnered buy-in to the process and findings.