About Research
Our current research efforts are based on previous arts-related research projects in computer science, but is also forward looking in its focus on integrative research . A project in computer music, for example, can be cast into the broader context of animation, games, and even theater studies. A project in computer graphics may have relevance simultaneously in architecture and art history. It is the development of these many symbiotic relationships that we feel provide the most value from the C2 initiative.
In the following we describe current research directions in the broad areas of “New Digital Media”, “History of Art and Culture” and “Design and Production”. Additional areas will continuously evolve as C2progresses. Detailed topics and projects, as well as full text of publications can be found under “Research by topic”
New Digital Media
Computer music research at Yale has focused on the design of new ways to express the synthesis of sound and, at a higher level, the composition of music. Our approach has been language centric: the use of advanced ideas in programming language research to allow the instrument designer or music composer to express his or her ideas in an abstract, high-level, declarative way, devoid of the many details normally associated with programming. We have designed, and are continuing to evolve, a Haskell library and domain-specific language called Euterpea that allows the expression of sound synthesis and instrument design, as well as high-level music composition. Euterpea is used exclusively in teaching a two-term computer music sequence at Yale.
Research topics involving the use of Euterpea include extensions for real-time interaction with musicians, new modes of expression based on ideas from composers (the domain experts), novel algorithmic techniques for composition and synthesis (based, for example, on fractal or stochastic mechanisms, as well as geometric chord spaces), and optimization techniques to allow, ultimately, real-time sound synthesis. We are also exploring visualization techniques, not just static graphics but also computer animations, that bridge the gap between music and the visual arts.
At the foundational level, we have developed a theory of polymorphic temporal media (PTM) that refines the traditional notion of “form vs. function” to include syntactic, temporal, and semantic properties. This theory is independent of the base media type, and thus includes animation, sound, video clips, music, and even dance (or, more concretely, humanoid robot motion). In related work, functional reactive programming (FRP) exposes commonalities of time-varying quantities and reactive behaviors over a variety of domains, including reactive animation, sound synthesis, and robotics. On a grander scale, our goal is to develop an integrative, hierarchical framework for describing, creating, organizing, evolving, storing, and retrieving digital artifacts across a variety of artistic domains. One approach to achieving these results is an effective extension to and integration of PTM and FRP into a single, coherent framework that exposes commonalities, while still permitting domain-specific specializations of behavior. A distinguishing aspect of our research is a declarative approach that provides a rigorous, specification-like interface that is easy to use, easy to reason about, and easy to change, because it more directly reflects the structure of the underlying domains.
History of Art and Culture
Our past experience in the area of history of art and culture has centered around capturing a model of Michelangelo’s Florence Pietà http://www.research.ibm.com/pieta/ and artifacts from the Egyptian museum in Cairo http://www.eternalegypt.org. The goal of the first project was to provide a model and tools to an art historian to study the form and design of the sculpture. The goal of the second project was the development of materials for public education about the history of Egyptian culture.
Underlying research issues revealed in these projects include problems in geometric and visual analysis and in digital representations to facilitate user interaction with complex models. In the Piet`a project, geometric algorithms such as “ball pivoting” to mesh point clouds and image adjustment methods to estimate reflectance and dense normals for visual representations were developed. For the Egyptian Museum project techniques such as image-based object editing were developed as a step in managing and modifying complex geometric objects. In addition to numerous publications, a series of ten patents have been issued for basic technical innovations resulting from our work on these cultural heritage projects.
We continue building on this experience by developing projects with Yale museums centered around 3D capture and display. A particular challenge is the effective presentation of artifacts in context. We have recently initiated a project to develop a novel representation and display system for the images, data and artifacts related to the 1930’s Dura Europos excavation that are held by the Yale Gallery http://artgallery.yale.edu/pages/collection/permanent/pc_ancient_over.html.
Design and Production
In developing methods and tools for the design and production of physical products, we draw on our experience in lighting design, acoustics, and advanced sketching techniques. We have previously developed tools for the design and simulation of opera lighting and projection effects in collaboration with the Metropolitan Opera in New York. This project posed challenges such as the simulation and real-time control of lighting systems involving hundreds of lights and elaborate textured, geometric models, the real-time rendering and control of complex time-varying illumination, and the simulation of projected background scenery. The outcomes of this work included methods for projections and deformation of projections, interaction techniques that allow users to indicate intent by painting surfaces, and optimization techniques to achieve visual effects.
We have also developed an inverse, interactive acoustic design approach that helps a designer produce an architectural configuration that achieves a desired acoustic performance. For a new building, the system may suggest optimal configurations that would not otherwise be considered; for a hall with modifiable components or a renovation project, it may assist in optimizing an existing configuration. Broader outcomes of this work include visualization techniques for displaying multidimensional results in 3D and goal-directed design using optimization techniques .
We build on lighting and acoustic design experience as well as our expertise in modeling materials in studying the problem of material design. Our goal is to develop end-to-end systems that start with the characterization of microstructure provided by material scientists and end with renderings of materials used in manufacturing as they appear through time. The research develops algorithms to translate the mathematical models of materials and their interaction with light into a mode that is useful for designers concerned with appearance and end function.
In addition to these analytical tools, we build on previous work in sketching. Previously we have investigated a sketching paradigm that attempts to bridge the gap between 2D drawing programs that have fixed views, and 3D modeling programs that allow arbitrary views. The resulting application utilizes a projective representation of points – i.e. points that lie on the surface of a unit sphere centered at the viewpoint. This representation facilitates the production of novel re-projects generated from an initial perspective sketch and gives the user the impression of being immersed in the drawing or space. In addition, we explored image-based modeling and editing of scenes. Our approaches allow editing from different viewpoints, extracting and grouping of image-based objects, and modifying the shape, color, and illumination of these objects.
We continue the sketching theme by considering tools that complement current CAD tools for design. Today’s CAD systems are based on a “top down” approach of defining large masses in 3D and refining and decorating these masses. We are developing an alternative approach in which designs are developed from sketches and photographs of varying levels of detail arranged by a user in a 3D environment. This system will allow the user to tour the 2D sketches in a manner that lets the model emerge by the fusion of images by the user’s own visual system. After the design is detailed by sketches, computer vision techniques may be used to extract a three-dimensional model from the user’s input. The advantage of the approach is that the user works in a familiar 2D mode for defining details, without the limitations and time delays of a full 3D model being defined at each step in the process.
Related to the problem of sketching proposed designs, is indexing and accessing past designs. Over a period of years designers create volumes of sketches that can serve as important reference material. Storing and retrieving either scanned or directly computer sketched drawings is challenging. We will pursue methods to analyze and organize sketches for ready retrieval based on both graphic and text input. To represent the sketches compactly we propose to extend our current work using geometric harmonic analysis to find meaningful basis functions to characterize salient features of sketches. Previous work on sketch searching has proposed new interfaces for search. The problem however involves much deeper issues of mathematical representation and organization – not just how sketches are presented on the screen to the user. In this work we are collaborating with the Yale Libraries, and in particular are working with Yale’s extensive Visual Resources Collection http://images.library.yale.edu/vrc/.
