Borlund started by explaining that the Retrieval Evaluation in Context (REX) project aims to increase the research standard of interactive information retrieval (IR) IR projects, improving requirements for the use of a simulated work task situation. A simulated work task situation (SWTS) is a “description of a scenario that invites the test participant to search for information”: something which may be used in evaluating an IR system, or carrying out some types of information seeking studies.
Borlund gave an example of an SWTS and highlighted how the situation should be something that the searcher can identify with, of topical interest, and with enough context for him/her to understand the situation. However it was also necessary to “include test participants personal information needs as baseline”, to pilot, to rotate the order of tasks, and to report the SWTS when presenting results.
This leads to Borlund’s aim “how to frame different types of information needs in SWTS.” Her own study involved 25 teenagers and 6 teachers, from a Danish school. Each had searches , with 3 SWTS and 1 personal information need. There was a pre search questionnaire, logging of search interaction and post search interviews. The 3 SWTS used one each of needs identified by Ingwersen: verificative (checking something factual), muddled (searching new topic to you) and conscious topical (a topic known to you). Borlund set different scenarios for the teachers and teenagers.
As an example, the “muddled” SWTS for the teachers was about the upcoming Danish curriculum reforms (thus it was directly relevant to them as teachers). She found it most difficult to find a verificative scenario that was relevant to the teenagers (something that they both didn’t know and might be expected to be interested in personally).
When Borlund compared the log data for the three types of search she used the variables: Number of search terms used, no. of unique search terms, no. of search iterations, no. of favourites and time spent searching. She also compared the search behaviour with the personal information need and the SWTS, the results correlated, which was seen as validating the search behaviour in the SWTS as being realistic. Borlund found that both groups’ search behaviour varied across the 3 types of search, and also the search behaviour of teenagers and teachers differed. The teenagers spent a lot less time on searching than the teachers did, on all searches (e.g. 4.33 vs. 13.36 minutes on average for the muddled topic). It is often assumed that amount of time spent is an indication of interest, however it did seem to be a different search style (e.g. the teenagers did more clicking of pictures, and one said “I don’t have to read it now, now I know where to find it” i.e. assuming that reading and understanding was not part of the task).
As part of the evaluating the SWTS, Borlund compared the scale (high, low or medium) of the 5 variables (no. of search terms etc.) with what was predicted: this identified potential issues with the nature of the task (e.g. whether the “muddled” scenario could be answered rather as a fact-finding scenario) . The interviews also gave insight into how useful the SWTS were.
Borlund gave several conclusions, which were recommendations for this kind of study involving SWTS. Unfortunately these whizzed off the screen before I had time to note them down, but they flowed out of her presentation e.g. advising getting even more information from participants to tailor the SWTS effectively; that it was valuable to include a personal information need search; the importance of piloting and using questionnaire/interviews.
Photo by Sheila Webber: window in Dunnottir castle, Scotland, June 2015
No comments:
Post a Comment