Method and system for managing graphics objects in a graphics display system

ABSTRACT

A method and system for controlling the algorithmic elements in 3D graphics systems via an improved 3D graphics API is provided. In one aspect, in a 3D graphics system having privatized formats with privatized drivers used to increase the efficiency of display, existing problems are eliminated that are associated with multiple redundant copies of the publicly formatted graphics data made in host system memory pursuant to various graphics operations e.g., lock and unlock operations. The ability to make a system copy of publicly formatted data is exposed to the developer, eliminating the creation of unnecessary, and redundant copies. Efficient switching between the privatized and public format remains hidden from the developers so that applications execute efficiently while removing consideration thereof from the developers. Thus, developers are free to consider other tasks. In the event that a developer wishes to make a copy of the data, the data is copied pursuant to an operation that the developer calls and over which the developer has control, ensuring that multiple redundant copies of the graphics data are not made in host system memory.

INFORMATION

DETAILED DESCRIPTION OF THE INVENTION

BRIEF DESCRIPTION OF THE DRAWINGS

The system and methods for controlling the algorithmic elements in a 3D graphics system are further described with reference to the accompanying drawings in which:

FIG. 1 is a block diagram representing a suitable computing system environment in which the present invention may be implemented;

FIG. 2 is a block diagram representing an exemplary network environment in which the present invention may be implemented;

FIG. 3A is a block diagram illustrating various connections between surfaces and objects in a graphics system running multiple applications simultaneously, wherein a connection may be lost;

FIG. 3B is a block diagram illustrating various connections between surfaces and objects in a graphics system running multiple applications simultaneously wherein the connections are preserved according to the present invention;

FIG. 4A illustrates the unification of 2D and 3D graphics APIs in accordance with the present invention;

FIG. 4B illustrates various ways in which applications could perform a texture download in accordance with prior 3D graphics APIs;

FIG. 4C illustrates a single way in which applications perform a texture download in accordance with the 3D graphics API of the present invention;

FIG. 4D illustrates various ways in which applications could perform a resolution change in accordance with prior 3D graphics APIs;

FIG. 4E illustrates a single function by which applications perform a resolution change in accordance with the 3D graphics API of the present invention;

FIG. 5 illustrates the downloading of specialized graphics functions to a 3D graphics chip in accordance with the 3D graphics API of the present invention;

FIG. 6A illustrates prior art techniques of interacting with procedural shaders;

FIG. 6B illustrates techniques of communicating with procedural shaders in accordance with the 3D graphics API of the present invention;

FIG. 7A illustrates a private driving format in connection with which the 3D graphics API of the present invention may be implemented;

FIG. 7B illustrates the 3D graphics API of the present invention that operates seamlessly with respect to the private driving format of FIG. 7A;

FIG. 8A illustrates a prior art technique of switching data among memory types with previous graphics APIs;

FIG. 8B illustrates the automatic switching of data among memory types in a graphics system in accordance with a cache managing algorithm of the 3D graphics API of the present invention;

FIG. 9A illustrates a prior art technique whereby data is transmitted to a graphics chip serially; and

FIG. 9B illustrates a technique in accordance with the 3D graphics API of the present invention whereby data is transmitted to a graphics chip in parallel.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Overview

The present invention provides a new and improved API as a layer between application developers and the current state of the art of graphics hardware and the pipeline that renders and processes the graphics data. As related in the background, a current problem with rasterizer/display units in 3D graphics systems exists with respect to the privatization of the formats used to increase the efficiency of display. The number and frequency of transformations between the privatized representation utilized with modern graphics hardware and the public representation utilized in host computing device system memory is currently out of the control of the developer, and consequently in many cases, multiple redundant copies of the same publicly formatted graphics data are made in host system memory, pursuant to various graphics operations e.g., lock and unlock operations.

The present invention thus exposes the ability to make a system copy of publicly formatted data to the developer, eliminating the creation of unnecessary, and especially redundant copies. At the same time, the process, whereby switching occurs between the privatized and public format, is hidden from the developers so that efficiency is gained behind the scenes of the application processes. In the event that a developer wishes to make a copy of the data, the data is copied pursuant to an operation that the developer calls, ensuring that multiple redundant copies of the graphics data are not made pursuant to multiple graphics operations that remove the element of control from the developer.

Various techniques for memory management are described in the following patents which are herein incorporated by reference: U.S. Pat. No. 5,801,717, entitled “Method and system in display device interface for managing surface memory”; U.S. Pat. No. 5,844,569, entitled “Display device interface including support for generalized flipping of surfaces”; U.S. Pat No. 6,078,942, entitled “Resource management for multimedia devices in a computer”; and U.S. Pat. No. 6,134,602, entitled “Application programming interface enabling application programs to group code and data to control allocation of physical memory in a virtual memory system”.

Exemplary Computer and Network Environments

FIG. and the following discussion are intended to provide a brief general description of a suitable computing environment in which the invention may be implemented. Although not required, the invention will be described in the general context of computer-executable instructions, such as program modules, being executed by one or more computers, such as client workstations, servers or other devices. Generally, program modules include routines, programs, objects, components, data structures and the like that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments. Moreover, those skilled in the art will appreciate that the invention may be practiced with other computer system configurations. Other well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to, personal computers (PCs), server computers, hand-held or laptop devices, multi-processor systems, microprocessor-based systems, programmable consumer electronics, network PCs, minicomputers, mainframe computers, gaming platforms and the like. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network or other data transmission medium. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

FIG. 1 illustrates an example of a suitable computing system environment in which the invention may be implemented. The computing system environment is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. For example, graphics application programming interfaces may be useful in a wide range of platforms. Neither should the computing environment be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment .

With reference to FIG. 1, an exemplary system for implementing the invention includes a general purpose computing device in the form of a computer . Components of computer may include, but are not limited to, a processing unit , a system memory , and a system bus that couples various system components including the system memory to the processing unit . The system bus may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus (also known as Mezzanine bus).

Computer typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by computer and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CDROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by computer . Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer readable media.

The system memory includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) and random access memory (RAM) . A basic input/output system (BIOS), containing the basic routines that help to transfer information between elements within computer , such as during start-up, is typically stored in ROM . RAM typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit . By way of example, and not limitation, FIG. 1 illustrates operating system , application programs , other program modules , and program data .

The computer may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only, FIG. 1 illustrates a hard disk drive that reads from or writes to non-removable, nonvolatile magnetic media, a magnetic disk drive that reads from or writes to a removable, nonvolatile magnetic disk , and an optical disk drive that reads from or writes to a removable, nonvolatile optical disk , such as a CD ROM or other optical media. Other removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The hard disk drive is typically connected to the system bus through an non-removable memory interface such as interface , and magnetic disk drive and optical disk drive are typically connected to the system bus by a removable memory interface, such as interface .

The drives and their associated computer storage media discussed above and illustrated in FIG. 1, provide storage of computer readable instructions, data structures, program modules and other data for the computer . In FIG. 1, for example, hard disk drive is illustrated as storing operating system , application programs , other program modules , and program data . Note that these components can either be the same as or different from operating system , application programs , other program modules , and program data . Operating system , application programs , other program modules , and program data are given different numbers here to illustrate that, at a minimum, they are different copies. A user may enter commands and information into the computer through input devices such as a keyboard and pointing device , commonly referred to as a mouse, trackball or touch pad. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit through a user input interface that is coupled to the system bus , but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). A monitor or other type of display device is also connected to the system bus via an interface, such as a video interface . In addition to the monitor, computers may also include other peripheral output devices such as speakers and printer , which may be connected through an output peripheral interface .

The computer may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer . The remote computer may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer , although only a memory storage device has been illustrated in FIG. . The logical connections depicted in FIG. 1 include a local area network (LAN) and a wide area network (WAN) , but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.

When used in a LAN networking environment, the computer is connected to the LAN through a network interface or adapter . When used in a WAN networking environment, the computer typically includes a modem or other means for establishing communications over the WAN , such as the Internet. The modem , which may be internal or external, may be connected to the system bus via the user input interface , or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer , or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation, FIG. 1 illustrates remote application programs as residing on memory device . It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

As mentioned, a computer, such as described above, can be deployed as part of a computer network. Further, the present invention pertains to any computer system having any number of memory or storage units, and any number of applications and processes occurring across any number of storage units or volumes. Thus, the present invention may apply to both server computers and client computers deployed in a network environment, having remote or local storage. More and more, graphics applications are becoming deployed in network environments. FIG. 2 thus illustrates an exemplary network environment, with a server in communication with client computers via a network, in which the present invention may be employed. As shown, a number of servers , , etc., are interconnected via a communications network (which may be a LAN, WAN, intranet or the Internet) with a number of client computers , , , etc. In a network environment in which the communications network is the Internet, for example, servers can be Web servers with which the clients , , etc. communicate via any of a number of known protocols such as hypertext transfer protocol (HTTP). The invention may also leverage peer to peer networking techniques. Each client computer and server computer may be equipped with various application program modules , other program modules and program data , and with connections or access to various types of storage elements or objects, across which files may be stored or to which portion(s) of files may be downloaded or migrated. Each client computer and server computer may also be connected to additional storage elements , such as a database. Thus, the present invention can be utilized in a computer network environment having client computers for accessing and interacting with a network and server computers , , etc. for interacting with client computers.

Maintenance of Connection between Surfaces and Objects

As mentioned in the background, a resource contention issue sometimes occurs due to the demands of multiple devices and applications requiring graphics system resources simultaneously. Current 3D graphics APIs sometimes improperly manage resources such that if multiple applications running simultaneously are maintaining connections to multiple surfaces from various objects of the graphics system, sometimes these connections to surfaces can become severed or disconnected. If the connection to surface memory for any one application is severed, a user may have to restart the application or begin certain portions of the application again in order to recreate a proper connection.

FIG. 3A illustrates some elementary components of a 3D graphics system. A rasterizer processes graphics data from surface memory in preparation for its transmission to a display device having display memory . Any number of objects , to may be created through graphics API when applications and or more are running. Objects , to may be part of the graphics API or may be provided separately for use with the graphics system. As FIG. 3A illustrates, a connection between surface space and, for example, object may become severed due to improper maintenance of system resources when performing thousands of high speed operations for multiple applications. For example, some objects are responsible for building and moving frames to surface space. Over the course of operation of a graphics application, thousands and thousands of such build and move operations may take place. Presently, there is only one point in the API code that performs a ‘check’ to see if the connection is properly maintained to achieve such operations. Previously, during the course of building, processing and presenting frame data, multiple checks were performed, essentially each time any operation occurred with respect to the data. This, however, is wasteful of computing resources in the event that a connection is lost since each check consumes computing resources on its own.

In accordance with the present invention, by performing a check and monitoring connections and system resources each time a frame is presented, the consumption of valuable resources between present operations according to various operations is avoided. As shown in FIG. 3B, the connection between surface and object is treated as if it persists between present operations in accordance with the present invention, even where the connection has been lost. For example, where a typical check for the persistence of the connection between surfaces and objects includes returning a flag, such as true or false, depending upon the whether the connection persists, these operations may be skipped or spoofed such that between present operations, the flag continues to indicate that the connection persists, even if it has been lost, thus avoiding multiple, redundant checks for the persistence of the connection. As a result, operations are streamlined between present calls such that checks are made when expedient to do so. Thus, the present invention prevents wasteful consumption of computing resources due to the loss of connection between surfaces and objects when multiple devices and applications demand premium system resources simultaneously.

In an exemplary embodiment, the present invention provides a solution by unifying the command structure that previously checked for lost connections to surfaces. Previously, there were innumerable places where the application checked for and handled resource issues between present calls. This distributed the checkpoints and created multiple, if not hundreds or thousands of places throughout the application where checks occurred between present calls, creating inefficiencies and further opportunities for error due to wasteful devotion of computing resources. In accordance with the present invention, each time data is ‘presented’ to the surface memory space according to a ‘present’ function call, the 3D API of the invention checks for these resource contention issues. Thus, instead of having many different cases occurring at different times for which the API might determine that a connection has been lost, the 3D API of the invention checks each time a ‘present’ function call is made, thereby providing a single case for a lost connection such that resources may be newly allocated according to the same unified procedure. This check may be performed before or after the present call, or included therein. It can be appreciated that any call, such as a present function call, that is cyclical and occurs once per frame e.g., at a typical 30 or 60 frames per second, or other recurring event may also be utilized to achieve the above goals.

Unification of Previous API Command Structure into Concrete Algorithmic Elements

The subsequent versioning of a software product to meet the needs of an evolving operating environment sometimes results in inefficiencies wherein once separate or merely related modules may be more efficiently placed together, rewritten or merged. Thus, the present invention unifies existing API command structures into concrete, atomic algorithmic elements that ease the task of development.

For example, while the creation, processing and rendering of 3D objects by a 3D API utilizes algorithms and function calls of the 2D API, a single set of APIs does not exist for the purpose of creating both 2D and 3D objects. There are thus typically multiple choices for a developer to make, when creating, processing or rendering an object, which makes the developer's work more complex. For example, with reference to FIG. 4A, there are numerous instances where the current 2D graphics API shares functionality with the current 3D graphics API, because, for example, both include the same function calls. There are also instances wherein to perform a function with the 3D graphics API may involve a function call to function of the 2D graphics API, and vice versa. While the overlap in the figure is illustrated simply with dashed lines, the reality of current interoperation is far from simple, and leaves the developer with too many choices. The present invention thus provides a single 2D and 3D graphics API, providing a unified programming body with which developers may work.

Currently, there are no 3D graphics APIs that unify 2D and 3D graphics data types. Historically, due to the evolution of 2D and 3D graphics processing, with ‘modern’ 2D graphics applications beginning as early as the 1970s and with 3D graphics applications developing in parallel fashion at a later time, 2D and 3D data types have simply been treated differently by different applications i.e., memory allocation, memory structures and memory management have been different as between 2D and 3D data types. Thus, because the present invention unifies data types and choices with respect to 2D and 3D data types, memory management techniques have been unified in accordance with the present invention, eliminating ad hoc memory management based on whether the data was a 2D data type or a 3D data type. Due to the unification of 2D and 3D data formatting, the definition of data is simplified from the perspective of the developer. Thus, a long felt need in the art for a unified 2D and 3D API is addressed by the present invention.

In the above scenario, there is an overlapping of functionality among API objects that is not exploited. There are also other instances in which the number of choices available to developers for the same or similar functionality can both complicate the development process and create a source of inefficiency. For instance, there are three ways for a developer to perform a texture download depending upon the hardware involved, wherein data is transferred from a system memory surface to a 3D display memory surface. Textures are rectangular arrays of colored pixels, which tend to be square in graphics applications, and when processed in quantity, represent a lot of data. A texture download, in this regard, is important because ultimately slow texture downloading can become a bottleneck in graphics data processing. Different texture downloads, depending upon the hardware used incident thereto, have different data transfer rates associated therewith and thus the choice of texture download made by the developer can affect the overall system performance. The optimization of this data transfer rate, if required of the developer, can involve considerable time, effort and skill.

As illustrated in FIG. 4B, three different functions or ways , and of graphics API are currently available to a developer for use in connection with a texture download depending upon which hardware , . . . is involved. When multiple hardware , . . . is present, and a developer has three choices to make regarding texture downloading data from a system memory surface , . . . to 3D display memory surface , the decision is undesirably time-consuming, complex and requires an understanding of the underlying hardware in order to make efficient decisions. The present invention thus provides a single fast texture download. Instead of having a plurality of mappings from an application to API objects, and multiple commands that perform the same or similar actions, the present invention provides a single mapping. The present invention thus centralizes the diversity of current texture downloads and provides a unified singular command structure, thereby reducing the number of diverse, and redundant, mappings to API texture downloads. Instead of choosing among API objects , or of FIG. 4B, or a combination thereof, or having to write an optimization routine that optimizes the use of , or , the present invention provides a single texture download object _td for use by the developer, as shown in FIG. C.

Optimization of graphics components used incident to a texture download, such as hardware , , . . . , is thus performed by the API object _td in accordance with the present invention, thereby freeing the developer to be concerned with other aspects of the graphics application. For example, according to one optimization, the number of times used and order of hardware components , , etc. utilized in connection with a texture download is hidden from the developer. For example, in one embodiment, object _td keeps track of how well the hardware objects , , etc. are keeping up in terms of free memory (capacity) and speed of data processing and transfer. This may be configured staticly beforehand, so as to control the number and order of hardware components , , etc. that may be used and in connection with a texture download, or this may be performed dynamically based upon an evaluation of the performance of texture downloading, with feedback from the hardware components , , etc.

In addition, there are a number of instances in which existing 3D graphics APIs inconvenience the developer by requiring the developer to write substantially more complex code than is necessary in view of today's computing environments. For example, currently it requires at least five programming steps to effect a resolution change, inconveniencing the developer each time a resolution change is desired. While coding five steps is still better than interfacing directly with graphics system components, the present invention unifies the command structure of a resolution change, allowing a developer to effect a resolution change with a single API command. The present invention thus provides a single command to effect a resolution change, insulating the developer from the detailed changes that are made in the graphics system in order to effect the change. This is yet another example where current graphics APIs require the developer to have an overly detailed understanding of the underlying graphics hardware. As shown in FIG. 4D, there are currently five steps or commands _rc, _rc, _rc, _rc and _rc that a developer D must enter in order to effect a graphics system resolution change RC. Each of commands _rc, _rc, _rc, _rc and _rc has an independent bearing on the graphics system which can involve overlapping functionality or redundant arguments. Thus, as FIG. 4E illustrates, the present invention provides a single efficient API object _rc to achieve a resolution change. Thus, in these and other instances, the present invention unifies existing API command structures into concrete, atomic algorithmic elements that ease the task of development for a developer.

Downloading of 3D Algorithmic Elements to 3D Chip and Unique Algorithmic Elements for use with Procedural Shaders

The API of the present invention enables programmability of a 3D chip, wherein programming or algorithmic elements written by the developer can be downloaded to the chip, thereby programming the chip to perform those algorithms.

Thus far, the functionality of 3D hardware chips or components has been relatively fixed from the vantage point of the developer, leaving little flexibility at the developer end to perform efficient sets of operations that are tailored to an application or that allow the developer to control the algorithmic elements that are performed by the 3D graphics chip. There are innumerable circumstances where the state of the art of graphics hardware has made it desirable to take operations previously customized by a developer for an application, and make these operations downloadable to a 3D chip for improved performance characteristics. Since 3D graphics chips have evolved to be very high performance compared to host system processing space, it would be advantageous to allow a developer to download functionality to the 3D chip.

Thus, by way of the API of the present invention, 3D algorithmic elements written by a developer can be downloaded to the 3D chip for improved performance characteristics. FIG. 5 illustrates this process whereby a developer D writes a routine that may be downloaded to 3D graphics chip . Similar to this case where a developer may write a routine downloadable to the 3D chip , there are also a set of algorithmic elements that are provided in connection with the API of the present invention (routines that do not have to be written by developer D, but which have already been programmed for the developer D), that are downloadable to the programmable chip for improved performance. As shown in FIG. 5, a developer D may download preexisting API objects , , . . . to 3D graphics chip . While graphics applications generally involve a performance specification that includes fast processing and fast rendering, the ability to control 3D algorithmic elements in this fashion is very advantageous, because it allows a developer access to the fastest, highest performance portions of the graphics processing system, enabling the developer to download efficient algorithmic elements to the graphics chip that are tailored to the application at hand.

In an exemplary embodiment, a developer adheres to a specific format for packing up an algorithmic element, or set of instructions, for implementation by a 3D graphics chip. The developer packs the instruction set into an array of numbers, by referring to a list of ‘tokens’ understood by the 3D graphics chip. This array of numbers in turn is mapped correctly to the 3D graphics chip for implementation of the algorithmic element by the 3D graphics chip. Further background, hereby incorporated by reference in its entirety, may be found in U.S. Patent Appln. entitled “API Communications For Vertex And Pixel Shaders” having inventors Boyd and Toelle (Attorney Docket No. MSFT-0238).

With respect to unique algorithmic elements for use with procedural shaders in accordance with the present invention, some general background is instructive: rendering and displaying three dimensional graphics typically involves many calculations and computations. For example, to render a three dimensional object, a set of coordinate points or vertices that define the object to be rendered must be formed. Vertices can be joined to form polygons that define the surface of the object to be rendered and displayed. Once the vertices that define an object are formed, the vertices are transformed from an object or model frame of reference to a world frame of reference and finally to two dimensional coordinates that can be displayed on a flat display device. Along the way, vertices may be rotated, scaled, eliminated or clipped because they fall outside the viewable area, are lit by various lighting schemes, colorized, and so forth. Thus, the process of rendering and displaying a three dimensional object can be computationally intensive and may involve a large number of vertices.

A triangle has many helpful properties that make it ideal for use in rendering three dimensional surfaces. A triangle is completely defined by three vertices and a triangle also uniquely defines a plane. Thus, many systems will use a plurality of triangles to render a three dimensional surface. If each triangle is passed separately to the graphic subsystem that renders the three dimensional object, then three vertices for each triangle must be passed and processed by the graphic subsystem. However, the number of vertices that must be passed and processed by the graphic subsystem can be reduced through “vertex sharing.” Vertex sharing relies on a property of shared sides among triangles. Although it takes three vertices to define one triangle, it only takes four vertices to define two triangles if they share a common side. In order to take advantage of vertex sharing to reduce the number of vertices needed to render an object, pipelined systems have been developed that divide a three dimensional object into triangle strips that can then be processed and displayed efficiently. Indeed, specialized 3D software objects and/or hardware components such as procedural shaders have been created or designed for the purpose of carrying out specialized graphics functionality upon graphics data, in order to speed the process of rendering complex graphics objects. Procedural shaders, such as vertex and pixel shaders, have traditionally been used to perform such complex transformations on pixels and/or arrays of pixels or triangles.

However, the functionality of these procedural shading software objects or hardware components has been relatively fixed, leaving little flexibility at the developer end to perform efficient sets of operations that may be efficiently tailored to a particular graphics application or task.

Thus, with previous 3D APIs, the API did not provide the developer with flexibility as to operations that could be performed in connection with procedural shaders, such as vertex and pixel shaders. Vertex and pixel shaders, which may be implemented with software or in hardware or with a combination of both, have specialized functionality. Currently, in order to utilize useful algorithmic elements of a procedural shader, or otherwise use fixed and limited functionality of the procedural shader, a developer has to invariably design software procedural shader algorithms from scratch for each application. While the core commands for use with the procedural shaders were available to the developer, the effective or efficient combination of these commands is left to the developer. Consequently, algorithms that are unique, common and useful in connection with typical 3D graphics processes, such as for typical use in connection with procedural shaders, are developed from the ground up for each application. Conceptually, these elements for acting on procedural shaders have been customized by necessity for each application and thus provided ‘above’ the API, programmed as part of the graphics application itself. As shown in FIG. 6A, developer D, with access to a specification for a procedural shader , programs an inflexible object so as to work with or control the procedural shader . As FIG. 6A illustrates, developer D develops a shading algorithm with code. After customization by the developer D, object interacts with shader(s) via graphics API .

With present hardware designs of procedural shaders, however, a specialized set of assembly language instructions has been developed creating flexibility of procedural shader use. The developer still has access to the specialized set of instructions as in the past. Advantageously, with the present invention, this specialized set of instructions, or list of tokens packed as an array of numbers, can be combined in such ways as to create useful algorithmic elements. The present invention takes these useful combinations and exposes the algorithmic elements to the developer by way of the API . Conceptually, therefore, the present invention provides these useful algorithmic elements for acting on procedural shaders below or inside the API. As FIG. 6B illustrates, the present invention provides API objects _ps, _ps, . . . _psn, which are useful combinations of procedural shader 's instruction set for acting upon shader . In this fashion, algorithmic elements for use with procedural shader are exposed to the developer D.

For more concrete examples of algorithmic elements that used to be customized by necessity above the API, which are now provided for use below are any of the following types of techniques, taken alone or in combination: lighting, colorizing, mapping, texturizing, surfacing, shading, enhancing, and other image processing techniques.

Some exemplary code or definitional pseudocode for a procedural shader, such as a vertex shader, is provided below. The declaration portion of an exemplary procedural shader defines the static external interface of the procedural shader. The information in the declaration includes assignments of vertex shader input registers to data streams. These assignments bind a specific vertex register to a single component within a vertex stream. A vertex stream element is identified by a byte offset within the stream and a type. The type specifies the arithmetic data type plus the dimensionality (1, 2, 3, or 4 values). Stream data that is less than four values are preferably expanded out to four values with zero or more 0.F values and one 1.F value.

The information in the declaration also includes assignment of vertex shader input registers to implicit data from the primitive tessellator. This controls the loading of vertex data which is not loaded from a stream, but rather is generated during primitive tessellation prior to the vertex shader.

Moreover, the information in the declaration also includes loading data into the constant memory at the time a procedural shader is set as the current procedural shader. Each token specifies values for one or more contiguous 4 DWORD constant registers. This allows the procedural shader to update an arbitrary subset of the constant memory, overwriting the device state (which contains the current values of the constant memory). Note that these values can be subsequently overwritten (between DrawPrimitive calls) during the time a procedural shader is bound to a device via the SetVertexShaderConstant method.

Declaration arrays are single-dimensional arrays of DWORDs composed of multiple tokens each of which is one or more DWORDs. The single DWORD token value 0xFFFFFFFF is a special token used to indicate the end of the declaration array. The single DWORD token value 0x00000000 is a NOP token with is ignored during the declaration parsing. Note that 0x00000000 is a valid value for DWORDs following the first DWORD for multiple word tokens.

The stream selector token is desirably followed by a contiguous set of stream data definition tokens. This token sequence fully defines that stream, including the set of elements within the stream, the order in which the elements appear, the type of each element, and the vertex register into which to load an element.

Streams are allowed to include data which is not loaded into a vertex register, thus allowing data which is not used for this shader to exist in the vertex stream. This skipped data is defined only by a count of DWORDs to skip over, since the type information is irrelevant.

defines stream zero to consist of four elements, three of which are loaded into registers and the fourth skipped over. Register is loaded with the first three DWORDs in each vertex interpreted at FLOAT data. Register is loaded with the fourth, fifth, and sixth DWORDs interpreted as FLOAT data. The next two DWORDs (seventh and eighth) are skipped over and not loaded into any vertex input register. Register is loaded with the ninth and tenth DWORDs interpreted as FLOAT data.

Placing of tokens other than NOPs between the Stream Selector and Stream Data Definition tokens in disallowed.

While the above exemplary computer-executable instructions have been described in the context of a procedural shader, these concepts and principles may be applied to any 3D hardware rendering device utilized in connection with a graphics display system.

Improvements on the Display (Rasterizer) Side of the API

As mentioned above, while direct video memory access was once a possibility, it is no longer a possibility according to today's currently utilized graphics architectures. In accordance with today's graphics pipeline architecture, specialized or private drivers and surface formats are used in connection with very fast graphics accelerators. With direct rasterizer/processor access to display memory surfaces, “chunks” of surfaces can be moved around according to the specialized surface format, and pulled for processing as efficiency dictates. Thus, the pipeline between display memory surface space and the display itself has been made more efficient. With reference to FIG. 7A, an example of the type of modern ‘chunk’ manipulation is illustrated at a microcosmic level i.e., only 4 squares or chunks of data are illustrated. Private driver causes chunks _ through _ to be grabbed as efficiency dictates and are subsequently manipulated with a rasterizer into an intermediate form , wherein the original image may be unrecognizable. Then, data is moved along the graphics pipeline to render the final image on display , whereby band B_ of data may translate to band B_ in the displayed image. These mathematical transformations, and timing thereof, have advanced algorithms for determining the efficiency of chunk grabbing and placement. In essence, many images involve redundant data, or data that can be exploited based upon temporal and spatial knowledge, and these algorithms exploit such knowledge to create an extremely efficient and fast graphics data rendering pipeline.

Without the API of the present invention, however, display memory surface space must be set up properly by the developer to work with this privatized format. FIG. 7B illustrates API in accordance with the present invention. The API object _ of the present invention hides the efficiencies gained from the privatized driver format and rasterizer operation as described above from applications and developers. As far as the developer D writing application is concerned, the display memory surface receives a rectangular image that is then rendered upon the display , when in fact many efficient operations and data manipulations take place behind the scenes. The present invention thus implements API object _ such that it exposes more of these efficient pipeline operations to the developer D so that developer D need be less concerned with the performance of hidden operations, such as locking and unlocking the privatized formats pursuant to various commands and function calls over which the developer has no control.

Currently, when data is stored in hardware memory, the data is implemented in the privatized format illustrated in FIG. A. When graphics data is stored in main memory, it is stored in the public, more easily understood format. The privatized driver performs this transition. However, previously when graphics data stored in the hardware memory was asked for pursuant to some command or function call, the data was shuffled back to the public format, and then transmitted according to the private format for hardware purposes. Thus, upon an ‘unlock’ command, the data was copied to system memory in the public format, and then the data was transformed to the privatized format where necessary. Unfortunately, a problem arises wherein the same data may be ‘unlocked’ from hardware memory according to multiple function calls or commands, and consequently, multiple copies of the same data may be copied to the system memory. The present invention addresses this problem by only allowing data to be copied into system memory space when the developer specifically makes such a request, and thus ensuring that only copies that the developer knows about are resident in the system. Reductions in performance as a result of multiple copies resident in main memory, when unnecessary, are thus avoided. In recognition that the data does not always need to be accessed from system memory space, the present invention thus allows the developer more freedom to command when data is copied to system memory when stored in the privatized format associated with hardware memory.

Optimal Switching of Data Objects among Memory Locations

As described in the background, there are generally two types of containers or data structures that the API presents to a developer for use: one for pixels and one for polygons. Essentially, through passing arguments to the API (placing data into the containers), the developers can manipulate various graphics data structures. Once these containers are filled with data, there are various places, such as system memory or on a 3D card or chip, where this data may stored for further manipulation. The decision as to where to place this data is generally a performance issue. For instance, data for which fast access is not necessary can be stored in system memory, whereas data for which speed of access is the most important may be stored on a chip designed for ultra fast access. It is also sometimes the case that it is desirable to switch data or chunks of data from one memory location to another memory location at different stages of processing.

As illustrated in FIG. 8A, in the past, when a developer switched data from one memory location to another, the developer had to code the switching the data i.e., by destroying the data in the old location and recreating the data in the new location. Developer D, after creating a data container or deleting and recreating a data container via graphics API , has the decision to make regarding where to place the new data container . While a developer may choose into which of system memory , graphics chip memory and other memory data container is placed, oftentimes, a developer D may write a custom cache managing algorithm for application in an effort to efficiently manage resources. In theory, the cache managing algorithm would try to account for all of the system parameters and attempt to manage memory based upon data location, frequency of accessing or processing the data, and processing times associated with certain locations. However, this is a great inconvenience to developer D who has to custom build each cache managing algorithm for each new application , and who through oversight may not manage resources as efficiently as possible.

As shown in FIG. 8B, in accordance with the present invention, the switching of data containers from one location to another is performed automatically by an API object _cm. Thus, polygon or pixel data objects are automatically transitioned between memory types such that the switching is seamless. It appears as if the data chunks/containers last forever to the developer, whereas in reality, the API hides the fact that the data is being transitioned to optimize system performance. For example, it would in theory be desirable to keep all data on the faster hardware chip memory to process data. However, in reality, there is little room for such on chip data, sometimes as few as a hundred registers. Thus, typically a cache managing algorithm optimizes the tradeoff between host system memory and video memory on the 3D card so as to keep a maximum amount of data for processing in graphics hardware memory without causing overflow. As alluded to above, currently, a developer has to write such a cache managing algorithm for every application that is individually tailored to the programming task at hand. The API of the present invention hides an optimal cache managing algorithm from developer D so that developer D need not be concerned with the optimal tradeoff of system resources, and so that efficient switching of data can take place ‘behind the scenes’ simplifying the developer's task. Data containers are thus efficiently placed in storage to maximize data processing rates, and storage space, whether a data container is newly created, or switched from one location to another.

Parallel Feeding of Data Objects to 3D Chip for Processing

There are also current issues with respect to the transmission of data containers , either pixel and polygon, to a 3D chip. Currently, when a developer goes about specifying multiple data objects to fill multiple containers, these data objects are fed to the 3D chip one by one, or in a serial fashion. As illustrated in FIG. 9A, currently, to feed two data containers and to graphics chip memory , developer D must feed the objects serially to memory . In the figure, t

Thus, an optimization in accordance with the present invention is that a developer coding an application may specify that multiple of these data objects wherever originated or located at the time of operation, may be fed to the 3D chip simultaneously or in parallel. As illustrated in FIG. 9B, both containers and may be fed to 3D graphics chip memory at the same time. At t, the data containers and are retrieved or created, and at t, containers and are fed to 3D graphics chip memory in parallel. While FIG. 9B illustrates the parallel transmission of two data containers to 3D graphics chip memory , any number of data containers up to the maximum storage of 3D graphics chip memory may be fed in parallel. Thus, in the case of the above example where data is being transmitted to 3D graphics chip memory , wherein the data includes the same spatial position of pixel(s), but only the orientation or color is changing, the data may be loaded into two separate containers and , with a header description understood by the graphics chip and implemented by graphics API , whereby a single copy of the position data can be loaded into container , and the changing color or orientation data may be loaded into container . Thus, when received by the graphics chip , the data is loaded correctly into register space and processed according to the header description. In an exemplary implementation, up to 8 data objects , , etc. may be fed in parallel to the graphics chip . In such a configuration, the exploitation of redundancies may be performed in connection with any of the 8 values utilized to represent a pixel's location, orientation, etc. The present invention thus supplements previous systems in which data could only be fed serially to a graphics chip with parallel feeding of graphics data.

The term graphics data as used herein is intended to cover not only video and/or audio data in any pre-processed, processed or compressed form, but is additionally intended to cover communications of data of any kind along the graphics pipeline among the developer, the software interface of the present invention and various hardware and/or software components utilized incident to the graphics system.

The various techniques described herein may be implemented with hardware or software or, where appropriate, with a combination of both. Thus, the methods and apparatus of the present invention, or certain aspects or portions thereof, may take the form of program code (i e., instructions) embodied in tangible media, such as floppy diskettes, CD-ROMs, hard drives, or any other machine-readable storage medium, wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the invention. In the case of program code execution on programmable computers, the computer will generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. One or more programs are preferably implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language, and combined with hardware implementations.

The methods and apparatus of the present invention may also be embodied in the form of program code that is transmitted over some transmission medium, such as over electrical wiring or cabling, through fiber optics, or via any other form of transmission, wherein, when the program code is received and loaded into and executed by a machine, such as an EPROM, a gate array, a programmable logic device (PLD), a client computer, a video recorder or the like, the machine becomes an apparatus for practicing the invention. When implemented on a general-purpose processor, the program code combines with the processor to provide a unique apparatus that operates to perform the indexing functionality of the present invention. For example, the storage techniques used in connection with the present invention may invariably be a combination of hardware and software.

While the present invention has been described in connection with the preferred embodiments of the various figures, it is to be understood that other similar embodiments may be used or modifications and additions may be made to the described embodiment for performing the same function of the present invention without deviating therefrom. For example, while exemplary embodiments of the invention are described in the context of graphics data in a PC with a general operating system, one skilled in the art will recognize that the present invention is not limited to the PC, and that a 3D graphics API may apply to any computing device, such as a gaming console, handheld computer, portable computer, etc., whether wired or wireless, and may be applied to any number of such computing devices connected via a communications network, and interacting across the network. Furthermore, it should be emphasized that a variety of computer platforms, including handheld device operating systems and other application specific operating systems are contemplated, especially as the number of wireless networked devices continues to proliferate. Therefore, the present invention should not be limited to any single embodiment, but rather construed in breadth and scope in accordance with the appended claims.

CLAIMS

1. A method for communicating between a 3D graphics API and surface memory space having a privatized format and a public format associated therewith in a 3D graphics system, comprising: requesting via the 3D graphics API a first operation upon 3D graphics data stored in at least one display memory surface of the surface memory space to be rendered; determining whether a second operation has been requested; and transforming said 3D graphics data from data formatted according to said privatized format to data formatted according to said public format and copying the publicly formatted data into host system memory pursuant to said first operation if said second operation has been requested; and requesting a third operation upon said 3D graphics data to be rendered, wherein publicly formatted data is copied into host system memory pursuant to said third operation only if said second operation has been requested; whereby if said first and third operations are requested, and said second operation has also been requested in connection with both said first and third operations, only one copy of publicly formatted data is copied into host system memory pursuant to said first and third operations.

2. A method according to claim 1, wherein said first operation is an unlock operation.

3. A method according to claim 1, wherein said first operation is a lock operation.

4. A method according to claim 1, wherein said second operation includes a specific request for said copying of publicly formatted data into host system memory.

5. A computer readable medium bearing computer executable instructions for carrying out the steps of: requesting via a 3D graphics API a first operation upon 3D graphics data stored in at least one display memory surface of the surface memory space to be rendered; determining whether a second operation has been requested; and transforming said 3D graphics data from data formatted according to a privatized format to data formatted according to a public format and copying the publicly formatted data into host system memory pursuant to said first operation if said second operation has been requested; and computer executable instructions for requesting a third operation upon said 3D graphics data to be rendered, wherein publicly formatted data is copied into host system memory pursuant to said third operation only if said second operation has been requested; and computer executable instructions for copying only one copy of publicly formatted data into host system memory pursuant to said first and third operations if said first and third operations are requested, and said second operation has also been requested in connection with both said first and third operations.

6. A modulated data signal carrying computer executable instructions for performing the method of claim 1.

7. A computing device having a 3D graphics software interface as a layer between executing applications and the graphics pipeline that renders and processes the graphics data, comprising: a 3D graphics API; wherein when a first operation upon 3D graphics data to be rendered stored in at least one display memory surface of the surface memory space is requested via the 3D graphics API, said 3D graphics data is transformed from data formatted according to a privatized format to data formatted according to a public format and then copied into host system memory according to the public format, only if a second operation has been requested; wherein further when a third operation upon 3D graphics data is requested via the 3D graphics API, said 3D graphics data is transformed from data formatted according to said privatized format to data formatted according to said public format and then copied into host system memory according to the public format, only if said second operation has been requested; and whereby if said first and third operations are requested, and said second operation has also been requested in connection with both said first and third operations, only one copy of publicly formatted data is copied into host system memory pursuant to said first and third operations.

8. A computing device according to claim 7, wherein said first operation is an unlock operation.

9. A computing device according to claim 7, wherein said first operation is a lock operation.

10. A computing device according to claim 7, wherein said second operation includes a request for said copying into host system memory.

COPYRIGHT

User acknowledges that Fairview Research and its third party providers retain all right, title and interest in and to this xml under applicable copyright laws. User acquires no ownership rights to this xml including but not limited to its format. User hereby accepts the terms and conditions of the License Agreement.