It’s partly how you construct the program:
- Bottom-up perspective — allow flexibility into the programs so that they can support the students
- Build programs that incorporate heterogeneity of perspectives (differing tribes)
- Environment: instructor is facilitator, students active, involved, lead discussions
- For “away” programs- allow time for students to go home and then return (10 week programs)
- Support hands-on experiences
- Take native students on field trips
- Involve other students “on the margin” to broaden participation
- Lessons learned– students came in from “academic department” rather than recruited by native program
- Some issues can be resolved by having participation from local evaluators on projects.
- Division by age is counterproductive
But also how you construct the evaluation
- Use evaluation to build up not tear down
- We need more data, but be careful with cause and effect
- Design evaluation to meet community?
- Benefit to the community
- Evaluators need to be part of the team from the beginning of the project-formative
- Evaluation is made easier or more effective with a long-term relationship between teachers and students. Teachers should be with class for more than a single semester. Perhaps more than one year? How could this be done?
- Establish a relationship with the audience you are working with- have a conversation
- Go to audience you want to learn from
- Growing our own evaluators
- Talk to community first, then design evaluation protocol and report back to community
- Time spent developing partnerships pays off
- Top-down or devise own metrics
- Try to identify tools that work at the tribal level, or even smaller groups
- Use as a way to inform future iterations of the program
- Use language to define the role.
Related to Surveys:
- Allow open ended responses
- But Native students must live in the world of standardized tests, so they must be prepared to take those types of tests as well.
- “Adaptive Assesment” (Adaptive tests present different questions to different test takers, depending upon how well they have answered earlier questions on the test. Someone who has answered poorly on earlier questions will be switched to easier questions for a period, while someone who has answered well on earlier questions will be presented with more challenging questions. Adaptive testing allows the assessment of knowledge, achievement and ability with fewer questions than traditional testing for all but the very poorest-performing test takers. However, the development of adaptive assessments is a complex task http://www.ehow.com/how_7518054_create-adaptive-assessment-tests.html)
- Debriefing/follow-up meetings to discuss survey findings
- Don’t try to create “Pan-Native” assessments: that’s standards all over again. Cultures and tribes differ greatly
- Have a local evaluator to do the survey/interviews (would save $)
- Have an external evaluator as a consultant
- How an elder might do it: tell a student something, then ask student to explain it back.
- How to ask the right questions
- Look to your family for examples. How was/is Traditional Knowledge passed down by elders?
- Most appropriate to do it formatively- throughout a course or program not just one big summative test at the end. Use oral means of assessing
- Natives- always evolving- outcomes not set in stone
- Poorly designed surveys can be problematic
- Standards-based evaluation/assessment: not appropriate for all students? Not just inappropriate for native students?
- Watch for creating own bias
- Will the survey have any impact?
- Working with the tribe/building trust
- Can have two types of responses:
- Automatic response
- In-depth response
Metrics:
- Are they doing science for their own tribe?
- How is success defined?
- Measure attitudinal/value changes
- Need to address the difference in the ideas of success among tribal people. What counts as success (i.e. publications, tenure, etc) is maybe not the direction or need of Native communities. More important may be cultural and linguistic achievements.
- There are different ways of expressing knowledge
- Cultural Identity
- Disconnect between different sets of standards (if success= standards based)
- How do we define success? Define effectiveness?
- How do we make evaluation to capture community building?
- How is traditional knowledge evaluated? It is a continuous process.
- Is education accessible to students?
- Number of people who return to their community after their post-secondary education (or doing work based on their community?) (or other indigenous communities?)
- Persistence of Students
- Prove Success of program
- Ways of student perception of what is going on
- Good standard—measure persistence rather than # of students in degree program
- Strategic plans: Look at community and university strategic plans and match metrics to that
- To be Native Friendly, get the cultural references straight: Must be accurate and appropriate to the specific culture in which the lesson or assessment is done. How can non-native educators learn this? Requires time and maybe retreats?
- What are the goals, and have you met the goals?
- Goal encourage NDN students into geosciences
Problems and needs
- Western science- disrespectful of local native communities
- Need more program directors/project evaluators @ NSF
- Educating NSF/DOD etc
- Build in time in project to build community
- Create culturally responsive evaluation
- Funding
- How do we conduct evaluation in a big project? One that we were thrown in at the last minute
- NDN populations (<1.5%) fall into “margin of error”
- Need to get publications in Native friendly methodologies
- Start a journal
- Problem of undergrads not learning basic knowledge- must spend time remediating
- Multi-age education
Resources:
- Indigenous evaluation: New Zealand is a good model
- LCO-Purdue-GLITWC NAU-EEOP
- EPA 2010 conference
- Northwestern Menominee American Indian Center