AEA - Evaluation of Second Life EP, Evaluation enhancement using web tools

Enhancing Evaluation and Evaluation Practice Using Web 2.0

One of the sessions this morning from the American Evaluation Association conference is a veritable meta-hot topic for my National Network of Library of Medicine (NN/LM) and medical librarian colleagues: Using Adobe Connect for evaluation, and evaluating a Second Life class for health administrators who used it for an emergency preparedness (EP) training scenario.

Some highlights and fun

• What was life like BT (Before Twitter)?
• President Obama uses Adobe Connect for meetings to cut travel costs
• Great collaboration possibilities in Adobe Connect I never even thought of before
• Traditional eval areas: Knowledge, skills, application, decision making
• Newish eval areas: Participation, interactivity, constructivism, situativity, visualization, collaboration
• Challenge: Recognizing that learning will be made of immersive experiences rather than knowledge transfer between teacher and student.

I was quite disappointed that one of the presentations I was looking forward to about the use of wikis to engage stakeholders in meaningful discussions about their program and its evaluation was not offered due to the presenter's change of jobs. That's always a risk with the time lag between proposals and conferences but the first time I've had it happen with something I really wanted to learn more about!

AEA - Evaluation of (Distance) Learning

The Evaluation of Distributed Learning and Computer-Enabled Environments to Support Instruction

I'm highlighting this session from the American Evaluation Association conference since it's not only of high interest to me but many other information professionals who are offering instructional resources and training via web-based and other distance learning modalities.

The session (my notes available in title link) refers to distributed learning (what's that?) and this is honestly the first time I've come across that term although it's been around for a while. My guess is it may be in more frequent use within the evaluation and education fields instead of the terms 'distance' or 'hybrid' that I'm used to in order to describe education settings exclusive of face-to-face instruction or including some along with distance modalities. It's good to get out of my library & medical terminology areas once in a while to take a look at other fields', but for the purpose of my own I'm using mine in my blog so I sound like me.

The three paper presentations were about

• The role of computer-assisted instruction in the field of statistics (a strongly positive indicator was found for face to face instruction along with a strongly negative one in distance learning, but observe the caveat in my notes)
• The role of collaborative communication in a hybrid engineering course (Communication? Engineering? Surprise, these two things aren't mutually exclusive after all and those who exhibit that are more engaged with exploring education tools)
• The role of discussion board quality in online professional development (Dive in and D'OH with Homer Simpson about the structure and importance of the instructor's own posts. I didn't make D'OH up, the presenters said it!)

We had quite a discussion afterward until we were kicked out of the room due to time. I hope this information is beneficial to others working in distance education and the challenges of effective discussion boards in their classes, and I'm looking forward to seeing more research along these lines.

AEA 2009 Orlando FL Day 2 - Afternoon

Unfortunately I cut out of the first workshop a bit early through no fault of the facilitators; I lost all the time I had planned to edit & post about the morning sessions in an insanely long line for lunch. I plan to remedy this tomorrow by brown bagging it from the grocery store!

How To Do Participatory Impact Pathways Analysis
by Boru Douthwaite and Sophie Alvarez

Abstract: In this workshop, participants will be introduced to Participatory Impact Pathways Analysis (PIPA) and develop impact pathways for their own program. PIPA is a practical planning and evaluation approach fast being adopted for use with complex programs in the water and food sectors (see http://impactpathways.pbwiki.com). PIPA begins with a participatory workshop where stakeholders make explicit their assumptions about how their program will achieve impact. Participants construct problem trees, carry out a visioning exercise and draw network maps to help them clarify their program theory in the form of 'impact pathways'. Impact pathways describe which actors need to change to achieve the program vision, what are those changes and which strategies are needed to make them happen. PIPA goes beyond the traditional use of logic models and logframes by engaging stakeholders in a structured participatory process, promoting learning and providing a framework for 'action research' on processes of change.

Next up were some interesting applications of a Key Evaluation Checklist (KEC) in the role of strategic planning management (apparently the term 'strategic planning' is used more often in the corporate world and 'strategic management' for academia) that I'm looking forward to exploring more when I return since I was part of an environmental scan for strategic planning (even though I'm an academic) & want to learn more about it.

Enhancing the Strategic Management Process Through the Use of Evaluation Measures
(Chair Michael Scriven unable to attend, Michelle Woodhouse-Jackson & Nadini Persaud presenting)

Abstract: Research has shown that companies (both for profit and non-profit) that engage in formal strategic planning tend to be more successful than companies that do not. Therefore, the strategic management process should be an integral part of every company or organization. Strategy evaluation is identified as one of the three phases of this process; however, the evaluative nature of this process has limitations which could be revamped using principles from evaluation methodology. This multi-paper session will focus on the evaluative nature of the strategic management process, with special emphasis on the widely-used Fred David strategic model, and will also highlight the similarities and differences between this model and the KEC (Key Evaluation checklist), a practical tool which can be used to conduct evaluations. The session will end with suggestions on how evaluation methods could potentially improve the strategic management process.

I'll cover the third session tomorrow morning in a post of its own, I'm still thinking about our great group discussion afterward that continued past the ending time until we were kicked out of our meeting room!

AEA 2009 Orlando FL Day 2 - Morning

I began the day not at the plenary session (sorry, just couldn't get going early enough after visiting with friends the night before and a 3 hour jetlag in full force!) but at a workshop with James Altschuld from The Ohio State University (why is it so important to include the The?) whom I remembered from last year in Denver as being a particularly engaging and thoughtful speaker.

I was pleased to discover that was the case again today and we dove into a great discussion of

Setting Those Needs-Based Priorities

Abstract: Many evaluators, while familiar with what needs are and procedures for assessing them, are much less knowledgeable about ways to formally establish priorities in situations where many needs exist. This short workshop will begin with questions asked of participants as to how they work with organizations to select priority needs. From that starting point a short overview will be given of criteria commonly used to prioritize needs (importance, feasibility, risk factors, etc.) and methods (weighting criteria, screening needs candidates, variations of rank ordering techniques, and so forth) employed. Participants will apply some of the methods on typical scenarios that might occur in needs assessments. The workshop concludes with a group discussion of perceptions of the prioritizing process and what might work best in different settings and why it is important to fully consider the nature of how final needs are chosen.

No Google doc of notes as his slides were very text heavy and I want to digest their content when I receive them later. The terms 'needs' and 'needs assessment' are tossed around a lot though and it is helpful to keep in mind for the context of evaluation that need is defined as the measurable discrepancy between what is and what should be, not about projecting solutions as part of them but using verbs (desired, likely to occur, etc) to describe them. Needs assessment is a systematic way of setting and making decisions about needs-based priorities. (both terms Witkin & Altschuld, 1995)

As a medical librarian, I briefly wondered what (if any) needs assessment work was done regarding the new PubMed layout.... but I won't get into that here!

Next up was a deviation from the title and abstract that made for a great presentation and discussion with the group. The deviation? There Is No Checklist :) The title is also a link to the Google doc of my notes.

Establishing Effective Relationships: Presentation of a New Checklist to Help Evaluators Understand and Work With Diverse Clients
with Gary Mikron and Nakia James, Western Michigan University

Abstract: This skill-building session will introduce a new checklist that is designed to help evaluators establish relationships and work effectively with their clients. Broadly speaking, the checklist covers a list of issues and common obstacles that evaluators face when working with diverse clients. Checkpoints will highlight strategies and practices that will help ensure effective relationships are built and maintained. The checklist draws upon three key sources of information: (i) relevant literature, (iii) interviews with experienced evaluators and program officers that oversee evaluation contracts, and (ii) the national and international experience of the presenters. While the checklist is intended to be concise and provide only prompts for evaluators, the presentation and paper will allow a more in-depth description of the do's and don'ts when it comes to working with evaluation clients.

AEA 2009 Orlando FL Day 1 Coverage

For the next few days I am attending the American Evaluation Association (AEA) conference in Orlando FL, which is just about as far as you can get from Seattle in everything from geographic distance to climate. I had to find my summer stash of clothing which I had packed away until next year when I was packing for here, with temperatures in the 80s... about as hot as it normally gets in Seattle during the summer!

The opening plenary began with a polite inquiry from a passerby in the aisle as I was booting up my computer: "Excuse me, is that Office 07 on your computer?"

When the answer was affirmative, he then asked if he could borrow my laptop (!)

It turns out that the passerby was Leonard Bickman, one of the plenary panel members. The panel had saved their Powerpoint in 2007 and it was incompatible with the 2003 version the presentation computer had. I suggested getting a flash drive of the presentation that I'd be happy to convert for them, did so, and on they went.

The hyperlink of the title leads to a Google document of my notes from the plenary& below that is a paraphrased abstract.

The Impact of Context on Evaluation Choices: Lessons for Three cases

Jody Fitzpatrick moderated a discussion with Leonard Bickman, Ross Conner, and Katrina Bledsoe about specific evaluations they conducted. The discussion focused on ways in which the contexts of those evaluations influenced the choices they made about the questions to be addressed in the evaluation, the methods used, the role of stakeholders, and their efforts regarding use.

Enhancing Organizational Learning with Technology: Implications of Diversity, Improving Response Rates, and Increasing Evaluation Capacity

A multipaper session sponsored by the Integrating Technology into Evaluation topic interest group.

Friday Foolery #38: The Beacon of Evaluation turns into...

This is me on a 2 hour bus ride to Boston on Sunday, about to face 8.25 hours of plane rides on two flights after a week at Woods Hole:


At the time I claimed the top of my head really hadn't gone missing but I was wrong.

At work on Wednesday, I scanned through the table of contents for the recent journal of the American Evaluation Association and promptly misread the word 'beacon' as bacon and 'evaluation' as elevator.

Lo and behold,


The Bacon Elevator, brought to you by the World Famous Sporks of the Disney University.

It goes to show you that no matter what the subject is, bacon already has it covered.

Blog-o-Remiss-a-Versary Time!

(Snowflake cupcake @ abakedcreation)

Here's what I'm still in a state of disbelief about

• I've been blogging here for 1 year today
• People actually read what I write (I think)
• People share with others what I write here (thanks AEA) and on Twitter
• I've been able to know many wonderful people around the world because of this blog
• I've been in remission from Löfgren’s syndrome* for 8 years this month

Here's what I'll do my best to continue until the next blog-o-remiss-a-versary

• Share coverage of conferences, classes & events I'm at that are helpful to know about
• Strongly advocate for the credibility of appropriate social media channels for outreach, education, feedback & evaluation
• Not reinvent the wheel by 100% rehashing what others have already said well
• Respect and value your time and perspectives
  
• Welcome more guest blog entries (I'm looking at another Eagle, you know who you are!)
• Share my abiding love & learning of health informatics (and bacon)
• Fridays are meant for Foolery
• Stay healthy

Are you with me still?

Thank you. I sincerely and humbly appreciate it.

*Löfgren’s syndrome - an acute form of sarcoidosis. I was diagnosed as stage II in 1999 but symptomatic by the summer of 1998 before it simmered down only to later appear severely with every Löfgren’s symptom at the same time. The snowflake is a symbol for sarcoid, each case is unique and I had it long before House made its obscurity trendy. The reality is sarcoid sucks. Despite extensive research they still don't really know what causes it, but it's likely I have the HLA-DRB1*03 allele and that combined with a trigger (possibly evergreens) led to a really scary few years. I never take health or the ability to breathe deep lungfulls of air for granted anymore.

AEA Day 4: Ten thousand, four hundred and thirty one

That's how many words of notes I took over the duration of the American Evaluation Association 2008 conference here in Denver, not including full session & paper titles (let alone presenter names) in the mad typing rampage.

I'm tired and I want to go home now, but not before a nice breakfast with my aunt before I catch a plane tomorrow morning.

Lots of great information was shared today (social network analysis, relevant cultural information, distance education evaluation practices that make sense compared to what I endured as a grad student, etc.) I've identified the future need for me to see about modifications in Google Docs to make it easier to skip directly to relevant information, but now is not the time.

For the collective record:
Saturday, November 8th notes
Friday, November 7th notes
Thursday, November 6th notes
Wednesday, November 5th notes

I won't be able to coherently discuss any of this information for quite some time until I'm able to synthesize it a bit!

AEA Day 3: So, what's your B-HAG?

It's a reliable rule of thumb since my graduate school trips to Texas in 2006 that I go a bit insane on the third night away from my family. This conference is no exception to the rule.

My favorite line of the conference today is the discussion of the B-HAG, which stands for Big, Hairy Audacious Goal. I love this and think I'll informally begin every program or class project I have in mind with a serious B-HAG brainstorming session before going any further with it. Today I also rescued a presenter from having to verbally describe a website and its functionality because his wireless connection decided to fail. I have a camera phone pic of most of the 30 participants in the room gazing at my laptop but am having issues uploading it. I'll include it in the Google Docs when I can.

Despite third night insanity, I've polished up most of my notes from today (6 sessions) as well as yesterday's and the day before's. Tomorrow should go very well with my scheduled Sanity Break involving shopping and a long lunch!

Friday Foolery #7: 'Cause rational discourse was not on the test

Tom Chapin puts into song (MP3 here) what I've been thinking for years, and why I am immensely thankful our son's teacher has a Master of Fine Arts and is not afraid to use it in her classroom even if it's not on the test or grading rubrics.

(Photo credit: success in standardized testing went to his head, by woodleywonderworks

Awareness credit: Gary Minron, Western Michigan University, who played the MP3 for us before a conference session about survey data)

AEA Day 2: Those with authority may fear sharing it

The most frequent searches that unsuccessfully lead people to my blog involve my old Quizno's/EBM post with 'I fear change, I will keep my bushes.'

It was the title statement by the president of the American Evaluation Association during a plenary today that made me think about organization structures. Those in hierarchies both know and execute the flow of authoritative power because it is a controlled factor that also carries an easy amount of blame on 'the system.' I use it myself to describe how things are easily lost in the massive bureaucracies we juggle.

These flows of power don't work the same way in a horizontal organization structure, yet the latter seem to be where most social networking and collaboration occurs. Sharing is not only easy, it thrives here with substantially decreased turnaround expectations compared to the hierarchy. Why is David Rothman apologizing for what really isn't a delay? In my current conference (which is rather un-Web 2.0), the onus is on us to contact presenters via email for copies of presentations. Is part of social networking an acceleration on turnaround time and the expectation that we as creators of knowledge resources will always & immediately share our material online? I'm not certain.

In the nature of sharing, (I saw others with laptops today, yay!) today's conference notes are here containing mostly unedited raw material that I'll turn into at least one lengthy blog post in the future and possibly two. I'm too pooped to put all 5 proper session titles, all the presenter names & affiliations and will edit them later. I need sleep because I have 6 sessions on tap for tomorrow!

A solo laptop blogging AEA 2008?

Lately there has been plenty of coverage in the library section of the blogosphere of trendy library conferences.

Then there's me, here at the American Evaluation Association (AEA) conference in Denver, Colorado, where the plenary speaker features a Pushmi-pullyu (she had an uncited one, mine is from the Llama Rescue) in the Powerpoint.

I am distinctive here not because I'm a librarian (although I've yet to run into another one besides my colleague roommate) but because I'm becoming aware of a rather odd factor: with the exception of presenters, no one else is running around with a laptop to take notes. This is particularly odd (to me) in the session sponsored by the graduate student & new evaluator interest group.

If you're involved in library evaluation and assessment you'd probably be interested in some of the sessions. My notes from today are up here. If I can locate outlets I'll hopefully be able to cover each day but there are none alongside walls... maybe in the very back?

There is a very high acronym usage and specialized jargon level I'm completely unfamiliar with here since I've had no prior training in evaluation/assessment beyond logic models. I did manage to explain to someone from the National Institute of Health (NIH) exactly how the National Network of Libraries of Medicine (NN/LM) fits in with the National Library of Medicine (NLM) & talked with the presenter of a poster about the assessment of community foundation grants (including consumer health) who had no idea NN/LM existed & seemed rather excited by it.