About Me

I'm a 28+ year academic health sciences career chimera whose views in no way represent the institution.
Showing posts with label evaluation. Show all posts
Showing posts with label evaluation. Show all posts

Friday, November 13, 2009

AEA - Evaluation of Second Life EP, Evaluation enhancement using web tools

Enhancing Evaluation and Evaluation Practice Using Web 2.0

One of the sessions this morning from the American Evaluation Association conference is a veritable meta-hot topic for my National Network of Library of Medicine (NN/LM) and medical librarian colleagues: Using Adobe Connect for evaluation, and evaluating a Second Life class for health administrators who used it for an emergency preparedness (EP) training scenario.

Some highlights and fun

  • What was life like BT (Before Twitter)?
  • President Obama uses Adobe Connect for meetings to cut travel costs
  • Great collaboration possibilities in Adobe Connect I never even thought of before
  • Traditional eval areas: Knowledge, skills, application, decision making
  • Newish eval areas: Participation, interactivity, constructivism, situativity, visualization, collaboration
  • Challenge: Recognizing that learning will be made of immersive experiences rather than knowledge transfer between teacher and student.

I was quite disappointed that one of the presentations I was looking forward to about the use of wikis to engage stakeholders in meaningful discussions about their program and its evaluation was not offered due to the presenter's change of jobs. That's always a risk with the time lag between proposals and conferences but the first time I've had it happen with something I really wanted to learn more about!

AEA - Evaluation of (Distance) Learning

The Evaluation of Distributed Learning and Computer-Enabled Environments to Support Instruction

I'm highlighting this session from the American Evaluation Association conference since it's not only of high interest to me but many other information professionals who are offering instructional resources and training via web-based and other distance learning modalities.

The session (my notes available in title link) refers to distributed learning (what's that?) and this is honestly the first time I've come across that term although it's been around for a while. My guess is it may be in more frequent use within the evaluation and education fields instead of the terms 'distance' or 'hybrid' that I'm used to in order to describe education settings exclusive of face-to-face instruction or including some along with distance modalities. It's good to get out of my library & medical terminology areas once in a while to take a look at other fields', but for the purpose of my own I'm using mine in my blog so I sound like me.

The three paper presentations were about

  • The role of computer-assisted instruction in the field of statistics (a strongly positive indicator was found for face to face instruction along with a strongly negative one in distance learning, but observe the caveat in my notes)
  • The role of collaborative communication in a hybrid engineering course (Communication? Engineering? Surprise, these two things aren't mutually exclusive after all and those who exhibit that are more engaged with exploring education tools)
  • The role of discussion board quality in online professional development (Dive in and D'OH with Homer Simpson about the structure and importance of the instructor's own posts. I didn't make D'OH up, the presenters said it!)

We had quite a discussion afterward until we were kicked out of the room due to time. I hope this information is beneficial to others working in distance education and the challenges of effective discussion boards in their classes, and I'm looking forward to seeing more research along these lines.

Thursday, November 12, 2009

AEA 2009 Orlando FL Day 2 - Afternoon

Unfortunately I cut out of the first workshop a bit early through no fault of the facilitators; I lost all the time I had planned to edit & post about the morning sessions in an insanely long line for lunch. I plan to remedy this tomorrow by brown bagging it from the grocery store!

How To Do Participatory Impact Pathways Analysis

by Boru Douthwaite and Sophie Alvarez

Abstract: In this workshop, participants will be introduced to Participatory Impact Pathways Analysis (PIPA) and develop impact pathways for their own program. PIPA is a practical planning and evaluation approach fast being adopted for use with complex programs in the water and food sectors (see http://impactpathways.pbwiki.com). PIPA begins with a participatory workshop where stakeholders make explicit their assumptions about how their program will achieve impact. Participants construct problem trees, carry out a visioning exercise and draw network maps to help them clarify their program theory in the form of 'impact pathways'. Impact pathways describe which actors need to change to achieve the program vision, what are those changes and which strategies are needed to make them happen. PIPA goes beyond the traditional use of logic models and logframes by engaging stakeholders in a structured participatory process, promoting learning and providing a framework for 'action research' on processes of change.

Next up were some interesting applications of a Key Evaluation Checklist (KEC) in the role of strategic planning management (apparently the term 'strategic planning' is used more often in the corporate world and 'strategic management' for academia) that I'm looking forward to exploring more when I return since I was part of an environmental scan for strategic planning (even though I'm an academic) & want to learn more about it.

Enhancing the Strategic Management Process Through the Use of Evaluation Measures

(Chair Michael Scriven unable to attend, Michelle Woodhouse-Jackson & Nadini Persaud presenting)

Abstract: Research has shown that companies (both for profit and non-profit) that engage in formal strategic planning tend to be more successful than companies that do not. Therefore, the strategic management process should be an integral part of every company or organization. Strategy evaluation is identified as one of the three phases of this process; however, the evaluative nature of this process has limitations which could be revamped using principles from evaluation methodology. This multi-paper session will focus on the evaluative nature of the strategic management process, with special emphasis on the widely-used Fred David strategic model, and will also highlight the similarities and differences between this model and the KEC (Key Evaluation checklist), a practical tool which can be used to conduct evaluations. The session will end with suggestions on how evaluation methods could potentially improve the strategic management process.

I'll cover the third session tomorrow morning in a post of its own, I'm still thinking about our great group discussion afterward that continued past the ending time until we were kicked out of our meeting room!

AEA 2009 Orlando FL Day 2 - Morning

I began the day not at the plenary session (sorry, just couldn't get going early enough after visiting with friends the night before and a 3 hour jetlag in full force!) but at a workshop with James Altschuld from The Ohio State University (why is it so important to include the The?) whom I remembered from last year in Denver as being a particularly engaging and thoughtful speaker.

I was pleased to discover that was the case again today and we dove into a great discussion of

Setting Those Needs-Based Priorities

Abstract: Many evaluators, while familiar with what needs are and procedures for assessing them, are much less knowledgeable about ways to formally establish priorities in situations where many needs exist. This short workshop will begin with questions asked of participants as to how they work with organizations to select priority needs. From that starting point a short overview will be given of criteria commonly used to prioritize needs (importance, feasibility, risk factors, etc.) and methods (weighting criteria, screening needs candidates, variations of rank ordering techniques, and so forth) employed. Participants will apply some of the methods on typical scenarios that might occur in needs assessments. The workshop concludes with a group discussion of perceptions of the prioritizing process and what might work best in different settings and why it is important to fully consider the nature of how final needs are chosen.

No Google doc of notes as his slides were very text heavy and I want to digest their content when I receive them later. The terms 'needs' and 'needs assessment' are tossed around a lot though and it is helpful to keep in mind for the context of evaluation that need is defined as the measurable discrepancy between what is and what should be, not about projecting solutions as part of them but using verbs (desired, likely to occur, etc) to describe them. Needs assessment is a systematic way of setting and making decisions about needs-based priorities. (both terms Witkin & Altschuld, 1995)

As a medical librarian, I briefly wondered what (if any) needs assessment work was done regarding the new PubMed layout.... but I won't get into that here!

Next up was a deviation from the title and abstract that made for a great presentation and discussion with the group. The deviation? There Is No Checklist :) The title is also a link to the Google doc of my notes.

Establishing Effective Relationships: Presentation of a New Checklist to Help Evaluators Understand and Work With Diverse Clients
with Gary Mikron and Nakia James, Western Michigan University

Abstract: This skill-building session will introduce a new checklist that is designed to help evaluators establish relationships and work effectively with their clients. Broadly speaking, the checklist covers a list of issues and common obstacles that evaluators face when working with diverse clients. Checkpoints will highlight strategies and practices that will help ensure effective relationships are built and maintained. The checklist draws upon three key sources of information: (i) relevant literature, (iii) interviews with experienced evaluators and program officers that oversee evaluation contracts, and (ii) the national and international experience of the presenters. While the checklist is intended to be concise and provide only prompts for evaluators, the presentation and paper will allow a more in-depth description of the do's and don'ts when it comes to working with evaluation clients.

Wednesday, November 11, 2009

AEA 2009 Orlando FL Day 1 Coverage

For the next few days I am attending the American Evaluation Association (AEA) conference in Orlando FL, which is just about as far as you can get from Seattle in everything from geographic distance to climate. I had to find my summer stash of clothing which I had packed away until next year when I was packing for here, with temperatures in the 80s... about as hot as it normally gets in Seattle during the summer!

The opening plenary began with a polite inquiry from a passerby in the aisle as I was booting up my computer: "Excuse me, is that Office 07 on your computer?"

When the answer was affirmative, he then asked if he could borrow my laptop (!)

It turns out that the passerby was Leonard Bickman, one of the plenary panel members. The panel had saved their Powerpoint in 2007 and it was incompatible with the 2003 version the presentation computer had. I suggested getting a flash drive of the presentation that I'd be happy to convert for them, did so, and on they went.

The hyperlink of the title leads to a Google document of my notes from the plenary& below that is a paraphrased abstract.

The Impact of Context on Evaluation Choices: Lessons for Three cases

Jody Fitzpatrick moderated a discussion with Leonard Bickman, Ross Conner, and Katrina Bledsoe about specific evaluations they conducted. The discussion focused on ways in which the contexts of those evaluations influenced the choices they made about the questions to be addressed in the evaluation, the methods used, the role of stakeholders, and their efforts regarding use.

Enhancing Organizational Learning with Technology: Implications of Diversity, Improving Response Rates, and Increasing Evaluation Capacity

A multipaper session sponsored by the Integrating Technology into Evaluation topic interest group.

Thursday, August 6, 2009

Health Information Technology Bibliography

The most beautiful combination of my interest in health information technology (HIT)/health informatics and librarianship I've seen for a while:



The Health IT Bibliography
from the Agency for Healthcare Research and Quality (AHRQ) has a list of broad subjects (Organizational Strategy, Technology, Evaluation, Operations, and Populations & Perspectives) with topics listed first and then the acronym, such as Health Information Exchange (HIE). Bookmark it now, the URL is very long and not memorable!

The first click in on a topic brings you to a page of peer-reviewed sources selected by subject experts (several were at Woods Hole!) that represent "the best known evidence on the benefits, challenges, and best practices" associated with it. They have also made an attempt to list freely accessible articles although some may need institutional/individual access permissions.

Scrolling down to the bottom of the subject page to the 'Click here for more' link leads to supplemental resources to the core listings on the subject page that may not be peer-reviewed but are still valuable contributions to the field. They welcome additional submissions and there are resources from 2008-2009 so it is actively maintained.

This resource has been live since February 2008 but I don't recall hearing about it either in the blogosphere or at Woods Hole. Have I been living under a HIT rock?

Tuesday, June 30, 2009

Live near DC and have an hour to help NLM?

From the MEDLIB-L listserv

Volunteers Needed for Web Site Evaluation

The Lister Hill National Center for Biomedical Communications at the
National Library of Medicine is seeking people interested in
evaluating a web site developed by NLM. The feedback received from
evaluators will help NLM develop effective and easy-to-use web
interfaces. Participants will be asked to spend one hour using a web
site on the Bethesda campus. Federal employees, contractors and
members of the public who can travel to NIH are encouraged to respond.
Although no monetary compensation can be offered, evaluators will
receive a small gift for their time. Forward this message to anyone
who might be interested in participating. Contact Sarah Ward at
saward@mail.nih.gov or (301) 496-9301 for more information.

Friday, June 12, 2009

Friday Foolery #38: The Beacon of Evaluation turns into...

This is me on a 2 hour bus ride to Boston on Sunday, about to face 8.25 hours of plane rides on two flights after a week at Woods Hole:


At the time I claimed the top of my head really hadn't gone missing but I was wrong.

At work on Wednesday, I scanned through the table of contents for the recent journal of the American Evaluation Association and promptly misread the word 'beacon' as bacon and 'evaluation' as elevator.

Lo and behold,


The Bacon Elevator, brought to you by the World Famous Sporks of the Disney University.

It goes to show you that no matter what the subject is, bacon already has it covered.

Thursday, May 7, 2009

Medlib's Round delay & near-pandemic social media progress

Due to a combination of people preparing for the Medical Library Association's upcoming conference in Hawaii and the global outbreak of H1N1 influenza, the Medlib's Round blog carnival is delayed until June 6th with publication on June 9th.

Many are, as usual, wondering if "the media" has overblown the H1N1 outbreak. I haven't watched any television news or talk shows recently (I rarely to never do anyway) so I can't comment there. However, when it comes to online and social media outlets, two things in particular impress me for signs of how the National Library of Medicine (NLM) and the National Institutes of Health (NIH) are opening up to these tools as high quality information resources during a near-pandemic situation.

The first is MedlinePlus and PubMed with the placement of the Department of Health and Human Services (HHS) H1N1 widget on the front pages. In addition, the very first hyperlink offered for the H1N1 Flu (Swine Flu) MedlinePlus page is @CDCemergency, an official Centers for Disease Control & Prevention (CDC) Twitter account. When I began writing this post the morning of May 6th, it had 104,266 followers. As of 10:25 that same evening the number had jumped up to 108,698! Others without Twitter accounts may be subscribing via RSS feed, those numbers are not included.

The second is the inclusion of Patricia Anderson's iGoogle H1N1 swine flu tab as an information resource within the NIH Library. To be clear, this is not NLM as NLM is the medical library for everyone and the NIH Library is for the 27 entities that make up the National Institutes of Health.

Are these uses of online & social media and the resources wiki I started all hype? Of course not. As is always the case for work and our own personal sanity, we must enable our own filters to narrow in on quality resources, share them, and tune out misinformation. A bit of humor now and then for you is good for you too.

This is a good opportunity to review your own emergency preparedness plans too... H1N1 isn't too severe now, but nobody can say that will continue to be the case during the flu season this fall. Will your library be prepared to still function if social distancing measures are enacted? Check out a pandemic planning exercise for more details.

I'm still hopeful for the day when we will see an NLM social media presence. CDC are truly the 2009 HHS social media pioneers who have had their hands full between the peanut recall and now H1N1. Hopefully things will settle down to allow for evaluation to determine some best practices to be shared by all, but in the meantime keep up the amazing work!

Thursday, March 19, 2009

ACRL 2009 - Information Literacy Assessment

One would think I was already done with my blog coverage of ACRL, but the truth is I had to turn all my attention to work and the upcoming Washington Medical Librarians Association (WMLA) conference I'm program chair for next week! After this entry I still have one more ACRL entry mulling on the back burner that integrates thoughts and experiences from the two keynotes given by Sherman Alexie and Ira Glass.

The last session I attended at ACRL was The Right Tool for the Job: Picking the Best Method for Information Literacy Assessment. This was nothing fancy nor glitzy compared to other session options going on at the time, but provided a good introduction and overview of how and when knowledge tests, integrated assessments & rubrics are appropriate assessment measures to use as part of library instruction.

As usual, a brief summary of my more extensive session notes

  • Knowledge tests focus on cognitive domain (recall, understanding, knowledge, analytical)
  • Use knowledge tests to assess at the cognitive level, not for behaviors or attitudes, or opinions.
  • Integrated assessments involve collaborating with professors to include information literacy assessments within the course assignment in meaningful, authentic way
  • Audience response to this is that annotated bibliographies have worked very well for students as an integrated assessment tool
  • Rubrics describe learning in 2 dimensions (parts/criteria and levels of performance)
  • Benefits of rubrics include focus on deep learning and higher order thinking skills, and a way to provide direct feedback

These assessment methods cover nearly any type of instruction and gave me some food for thought regarding how I can develop better ones for the online courses I teach to my colleagues that provide all of us with valuable information as part of the learning experience. I'm not satisfied to rest in the knowledge that the continuing education courses I teach are free; they are investments of my students' time and I strive to make it the best possible and productive experience for them.

Saturday, March 14, 2009

ACRL 2009 - Social Networking Literacy Competencies for Librarians

Excellent, thought-provoking papers about the use of social networking in the library and by librarians by both speakers that were well worth dragging my still-sick self out of bed and onto a Metro bus to slog through the rain at 8am on a Saturday morning for!

As usual, some teaser thoughts from my notes (can you tell my brain is on format autopilot?)

  • It is not enough to know what social networking is but able to understand nature and roles, importance in research, communication and information life cycle of the social networking site.
  • We have to evaluate the tools just as we do everything else and not jump on board just because it's new.
  • Librarians who are social networking-literate must be able to apply their current skills and curiosity to emerging and evolving resources.
  • Professional associations and library schools can support librarians with CE initiatives
  • Require new hires to possess social networking skills (yikes!)
  • How frequently do students use Facebook for academic purposes? Most ranked rarely and they feel Facebook impacts their academic study negatively.
  • When developing a virtual presence, consider students' actual perceptions of the library. Talk to your students, not student workers or this study's students.
  • Becoming a fan of the library involves less effort/awkwardness than friending.
  • Don't send apps or poke the students on Facebook.

My thoughts - It is impossible to verify with 100% accuracy any institutional online presence on Twitter. An individual may use a library or library product name, create a profile with the company logo, link to the company website and fill out all the appropriate profile details, follow (and be followed by) others in the field and 'talk the talk'... and not be sanctioned by the library or institution to represent them at all.

I am not going to give further attention to the individuals on Twitter I'm aware of who have done so with National Library of Medicine (NLM) by linking to them here, but be aware that they do exist. As it was in January and still the case now, there is no official NLM presence on Twitter.

Friday, March 13, 2009

ACRL 2009 - Peer to peer teaching/evaluating

I absolutely adore teaching online and am constantly on the lookout for strategies, methodologies and structures to enable my students to have the best learning experience possible there.

While this session was geared towards library instruction by librarians for students at the University of Alberta in a face to face setting, I wonder about the possibilities of implementing colleague peer-review teaching/observing relationships for online education and the valuable feedback that could be obtained by it.

Some teasers from my notes:

  • Why experiment with peer to peer research process? A desire to share love of subject matter that shares your passion and enthusiasm, improve and enhance skills, overcome ‘silo-ization’ to talk and learn from one another
  • Used a 3-step approach of Preparation, Observation phase, Closing phase
  • Challenges of teacher: being observed by peers made them extremely nervous. Ironic since same academics ok with peer review of their research but threatened by having a peer in the classroom (paraphrase of Washer, 2006)

This also introduced me to skrbl, which I had not heard of before and want to check out later!

Friday Foolery #25: Bacon & Ramen & Pie (Charts)? Oh My!


Hat tip to Susan Barnes at the Outreach Evaluation Resource Center (OERC) for guiding me to GraphJam.com, and in particular this entry. Unlike this fine example, I'm fairly certain Ramen was at the base of my college food pyramid though.

Otherwise, I pledge my life and my fortune to the pie!

Saturday, November 8, 2008

AEA Day 4: Ten thousand, four hundred and thirty one

That's how many words of notes I took over the duration of the American Evaluation Association 2008 conference here in Denver, not including full session & paper titles (let alone presenter names) in the mad typing rampage.

I'm tired and I want to go home now, but not before a nice breakfast with my aunt before I catch a plane tomorrow morning.

Lots of great information was shared today (social network analysis, relevant cultural information, distance education evaluation practices that make sense compared to what I endured as a grad student, etc.) I've identified the future need for me to see about modifications in Google Docs to make it easier to skip directly to relevant information, but now is not the time.

For the collective record:
Saturday, November 8th notes
Friday, November 7th notes
Thursday, November 6th notes
Wednesday, November 5th notes

I won't be able to coherently discuss any of this information for quite some time until I'm able to synthesize it a bit!

Friday, November 7, 2008

AEA Day 3: So, what's your B-HAG?

It's a reliable rule of thumb since my graduate school trips to Texas in 2006 that I go a bit insane on the third night away from my family. This conference is no exception to the rule.

My favorite line of the conference today is the discussion of the B-HAG, which stands for Big, Hairy Audacious Goal. I love this and think I'll informally begin every program or class project I have in mind with a serious B-HAG brainstorming session before going any further with it. Today I also rescued a presenter from having to verbally describe a website and its functionality because his wireless connection decided to fail. I have a camera phone pic of most of the 30 participants in the room gazing at my laptop but am having issues uploading it. I'll include it in the Google Docs when I can.

Despite third night insanity, I've polished up most of my notes from today (6 sessions) as well as yesterday's and the day before's. Tomorrow should go very well with my scheduled Sanity Break involving shopping and a long lunch!

Thursday, November 6, 2008

AEA Day 2: Those with authority may fear sharing it

The most frequent searches that unsuccessfully lead people to my blog involve my old Quizno's/EBM post with 'I fear change, I will keep my bushes.'

It was the title statement by the president of the American Evaluation Association during a plenary today that made me think about organization structures. Those in hierarchies both know and execute the flow of authoritative power because it is a controlled factor that also carries an easy amount of blame on 'the system.' I use it myself to describe how things are easily lost in the massive bureaucracies we juggle.

These flows of power don't work the same way in a horizontal organization structure, yet the latter seem to be where most social networking and collaboration occurs. Sharing is not only easy, it thrives here with substantially decreased turnaround expectations compared to the hierarchy. Why is David Rothman apologizing for what really isn't a delay? In my current conference (which is rather un-Web 2.0), the onus is on us to contact presenters via email for copies of presentations. Is part of social networking an acceleration on turnaround time and the expectation that we as creators of knowledge resources will always & immediately share our material online? I'm not certain.

In the nature of sharing, (I saw others with laptops today, yay!) today's conference notes are here containing mostly unedited raw material that I'll turn into at least one lengthy blog post in the future and possibly two. I'm too pooped to put all 5 proper session titles, all the presenter names & affiliations and will edit them later. I need sleep because I have 6 sessions on tap for tomorrow!

Wednesday, November 5, 2008

A solo laptop blogging AEA 2008?

Lately there has been plenty of coverage in the library section of the blogosphere of trendy library conferences.

Then there's me, here at the American Evaluation Association (AEA) conference in Denver, Colorado, where the plenary speaker features a Pushmi-pullyu (she had an uncited one, mine is from the Llama Rescue) in the Powerpoint.

I am distinctive here not because I'm a librarian (although I've yet to run into another one besides my colleague roommate) but because I'm becoming aware of a rather odd factor: with the exception of presenters, no one else is running around with a laptop to take notes. This is particularly odd (to me) in the session sponsored by the graduate student & new evaluator interest group.

If you're involved in library evaluation and assessment you'd probably be interested in some of the sessions. My notes from today are up here. If I can locate outlets I'll hopefully be able to cover each day but there are none alongside walls... maybe in the very back?

There is a very high acronym usage and specialized jargon level I'm completely unfamiliar with here since I've had no prior training in evaluation/assessment beyond logic models. I did manage to explain to someone from the National Institute of Health (NIH) exactly how the National Network of Libraries of Medicine (NN/LM) fits in with the National Library of Medicine (NLM) & talked with the presenter of a poster about the assessment of community foundation grants (including consumer health) who had no idea NN/LM existed & seemed rather excited by it.