Annual Conference
Association of Moving Image Archivists

November 17-22, 1997

Deriving shot-level indexing from audio description texts

James M. Turner
Professeur adjoint
École de bibliothéconomie et des sciences de l'information
Université de Montréal
voice +1 514 343 2454
fax +1 514 343 5753
turner@ere.umontreal.ca
http://tornade.ere.umontreal.ca/~turner

Separation line

What’s audio description?
Deriving shot-level indexing?
Premises for this work
But don’t we want to get rid of text?
Background work
Follow-up studies
How would all this work exactly?
Some preliminary results
Conclusion

Separation line

What’s audio description?

Technique for tv, theatre, movies, that uses the «second audio program» (SAP) channel to insert a voiceover description of what’s happening in the picture.
Developed for the visually impaired.
Uses available empty sound spaces (when nobody is speaking and there is no other foreground sound).
In existence since the early 1990s.
Seen mostly on PBS and largely through the work of the Descriptive Video Service (DVS) at WGBH in Boston.
Less than 10 describing agencies in the U.S., a bit of activity in Canada, some activity in France.

Separation line

Deriving shot-level indexing?

The wordprocessing file from which the narrated text is read into a microphone for inclusion in the production provides a pre-existing automated file describing the image.
Although created for the blind, these text files might be recycled as the source for automatically generating shot-level indexes to the described material.

Separation line

Premises for this work

A premise for shot-level indexing: we will have automated indexes to individual moving-image shots or we will have none at all.
Audio description is just one of several components of a global strategy for generating indexing to moving images from existing text. Others include closed captioning, narration, dialogue, scripts, editors’ logs, production notes, camera reports
Each offers only a partial solution, but they could work together to constitute high-quality indexes.
Other strategies, using the image, are members of the same «retrieval team»

Separation line

But don’t we want to get rid of text?

Even if computers could offer online access to the moving image as a component of the database and display 400 shots at a time (this is not going to happen any time soon), we still need the text
This is because the text contains information useful for searching but which is not available from the image (e.g. where was this shot taken, who’s that woman in the red dress, what kind of dancing is that, when was this shot taken?)
Another reason: a quick scan of text descriptions of the shots drastically reduces viewing time by helping eliminate most shots
Ultimately, the trick is to recycle existing text, avoid having to create new text, and automate the processes for integrating the text into the database used for managing information about the shots

Separation line

Background work

Previous work in the area of shot-level indexing for storage and retrieval of moving images (Turner 1994) showed that users tend to provide retrieval cues by naming objects and events that appear in the frame.
Stockshots from the National Film Board of Canada were compiled onto a tape, and 200 participants screened the tape
Their task: for each of the 44 shots, write 1 to 5 descriptors that could be used for retrieving the shot from some information system
Variables: • The participants: student or worker, visually-oriented or not • The shots: still or moving, simple or complex
Analysis: how many terms (2+), what were the terms named most often?

Separation line

Follow-up studies

In one follow-up study (Turner 1995), the top terms were compared with indexing terms assigned by professionals for the same shots
The top terms were also compared against the running description («visual synopsis») created for use in the source database
Summary of results

Shots in the database % Shots in the card file %

Term appears
in indexing
Term appears
in description 9/11

9/11 82

82 28/31

25/33 90

76
The results strongly suggest that automating subject access to non-art images can be successfully achieved using existing textual representations
However, the basis for making this claim needs to be confirmed, because the results are MUCH better than those usually obtained in information studies of all sorts
In another follow-up study, the top terms provided by French-speakers were compared against those provided by English-speakers to see how well they correspond
This is with a view to automating the indexing in bilingual and multilingual information systems
The analysis is not yet complete, but the results so far indicate that it will be possible to automate this aspect too
Observation: probably the encouraging results have to do with the nature of the material: shots of things anybody can name (as opposed to more abstract objects, e.g. command names for software, textual expressions of concepts)
Replication studies are underway (1996-1998) to confirm all these results
The results will provide (we hope!) a solid theoretical foundation for deriving indexing from existing text


Term appears in indexing Term appears in description	9/11 9/11	82 82	28/31 25/33	90 76

Separation line

How would all this work exactly?

These studies are being done in a research context, but the long-term goal is to build practical applications
In the context of the ongoing automation of the processes of production, the text components will become more and more integrated
Thus, the overhead of transferring and massaging files will be eliminated
Stopword lists and a markup language can be used to identify useful indexing terms (the text may be already marked up for other purposes; alternately, names of things could be checked against a dictionary and a parser, and tagged automatically)
The indexing derived from the text would be transferred to the appropriate fields in the database, and a pointer to the image installed
NB: We’re not there yet. For the moment, our work is concentrated on identifying the characteristics of audio description texts

Separation line

Some preliminary results

There are 187 shots

Digitised and compressed, this still represents over 1 GB of data for images 11 x 8 cm (about 4 x 3 inches) on a computer screen

The smallest file size is 0.8 MB; the largest is 31.2 MB, and the average is 6.4 MB (the mode is 3.5 MB)

The shortest shot is less than a second, the longest is 45 seconds, and the average is 9 seconds (the mode is 3 seconds)

Some other figures:


Number of shots with any audio description at all: Number of «episodes» of audio description: Some description corresponds to the shot on screen: Descr. entirely spoken while corresponding shot shown: Description entirely contained in a sequence of shots: Description refers to the next shot: Description refers to the last shot: Description does not overlap shots: Description refers to the second last shot: Description trails from the preceding shot: Description anticipates next shot:	80 57 59 22 4 13 21 56 1 13 8	(=43%) (=32%) (=12%) (=7%) (=11%) (=30%) (=7%) (=4%)

Separation line

Conclusion

Whether the description is spoken as the corresponding shot is shown or not, less than half the shots are described at all.
However, the narrator provides much additional description (not compiled here).Other sources (people speaking, other sound) provide additional information. All this should show up in closed-captioning text, which is another member of the «retrieval team»
The indexing model which connects the description to the shot will need to take the neighbouring shots into account, since sometimes the description overlaps (11%), and even more often it is entirely within the preceding or the following shot (21%).
The information gleaned from this sample may not be typical of audio description in general.
It may not even be typical of other documentary programmes; in addition, there may be differences in patterns between types of programmes.
Furthermore, different agencies may provide different levels / quality of description.
Thus only exploratory results are reported here, but they will be fleshed out over the next two years.

Separation line

Conclusion

Main menu | EBSI | Comments

Deriving shot-level indexing from audio description texts

Table of contents

What’s audio description?

Deriving shot-level indexing?

Premises for this work

But don’t we want to get rid of text?

Background work

Follow-up studies

How would all this work exactly?

Some preliminary results