Toward Text-to-Picture Synthesis

author: Andrew B. Goldberg, University of Wisconsin - Madison
published: Jan. 19, 2010,   recorded: December 2009,   views: 2572


Related Open Educational Resources

Related content

Report a problem or upload files

If you have found a problem with this lecture or would like to send us extra material, articles, exercises, etc., please use our ticket system to describe your request and upload the data.
Enter your e-mail into the 'Cc' field, and we will keep you updated with your request's status.
Lecture popularity: You need to login to cast your vote.


It is estimated that more that 2 million people in the United States have significant communication impairments that result in them relying on methods other than natural speech alone for communication [2]. One type of commonly used augmentative and alternative communication (AAC) system is pictorial communication software such as SymWriter [8], which uses a lookup table to transliterate each word (or common phrase) in a sentence into an icon. This is an example of converting information between modalities. However, the resulting sequence of icons can be difficult to understand. We have been developing general-purpose Text-to-Picture (TTP) synthesis algorithms [10, 5] to improve understandability using machine learning techniques. Our goal is to help users with special needs, such as the elderly or those with disabilities, to rapidly browse documents through pictorial summaries (e.g., Figure 5). Our TTP system targets general English. This differs from other pictorial conversion systems that require hand-crafted narrative descriptions of a scene [1, 9], 3D models [3], or special domains [6]. Instead, we use a concatenative or “collage” approach. In this talk, we discuss how machine learning enables the key components of our TTP system.

See Also:

Download slides icon Download slides: nips09_goldberg_ttp_01.pdf (1.2 MB)

Help icon Streaming Video Help

Link this page

Would you like to put a link to this lecture on your homepage?
Go ahead! Copy the HTML snippet !

Write your own review or comment:

make sure you have javascript enabled or clear this field: