Personal tools
You are here: Home downloads Gem Documentation Manual Ye olde Gem Manual
Document Actions

Ye olde Gem Manual

Note: This is the print view with all the Reference Manual pages on one page. The paginated version is available here, if you prefer that.

some online-documentation on how to use GEM

1. GEM - Introduction

a short introduction to Gem
GEM - Introduction

Introduction

GEM is the Graphics Environment for Multimedia. It was originally written by Mark Danks to generate real-time computer graphics, especially for audio-visual compositions. Because GEM is a visual programming environment, users do not need any experience in traditional computer languages.

GEM is a collection of externals which allow the user to create OpenGL graphics within Pd, a program for real-time audio processing by Miller Puckette (of Max fame).

There are many different shapes and objects, including polygonal graphics, lighting, texture mapping, image processing, and camera motion. All of this is possible in real-time without any previous programming experience. Because GEM is an add-on library for Pd, users can combine audio and graphics, controlling one medium from another.

GEM is supported in part by a grant from the Intel Research Council for the The Global Visual Music project of Vibeke Sorensen, Miller Puckette and Rand Steiger.

An important thing to remember is that GEM is NOT an application.  It is a library that Pd loads at run-time.  Make sure that you see the section on using GEM with Pd.  This manual assumes that you have Pd working correctly and can load up patches already.  If you do not have that working yet, look at the Pd manual and the GEM FAQ.  Also, it is assumed that you have a basic understanding of how to use Pd and the idea behind the data flow model.  In other words, if I ask you to pass a message with 3 floats into an object, you would know what I mean.

The system requirements vary depending on your system and what you are trying to do.  In general, you should have the most powerful computer available and the best graphics accelerator on the market.  In reality, people have been doing some amazing work with a Pentium II and an nVidia Riva TNT or 3Dfx Voodoo2 card.  If you are on an SGI, then everything from an O2 up seems to be okay.  The biggest requirement is that you have some kind of OpenGL graphics accelerator.  This means that a Matrox Millennium II will not run very quickly.

The other factor is what you are trying to do.  Pushing real-time video around requires a fast bus, which really only exists on SGIs.  Doing thousands of texture mapped polygons is great on a PC...if it is a constant texture.  There are many issues which mean that there is no one answer to "Is this system good enough?".  In general, you will have to try and see.

GEM is now maintained by IOhannes m zmölnig. So any bug-reports and donations should go to him instead of Mark...

[return]
 

2. Using GEM with Pd

How do I get Gem running?
Using GEM with Pd

Using GEM with Pd

    An important fact is that GEM is NOT an application. It is a library that the application Pd loads in at run-time.  Most of this information is taken directly from the GEM FAQ.

How do I install GEM on IRIX?
How do I install GEM on Win95/NT/2k?
How do I install GEM on linux?
How do I install GEM on macOS-X?
How do I run GEM on IRIX?
How do I run GEM on Win95/NT/2k?
How do I run GEM on linux?
How do I run GEM on linux?
Why doesn't GEM run?

How do I install GEM and Pd on IRIX?

See the readme for installing Pd.

Uncompress and untar the GEM file that you downloaded.  GEM should be located at

pd/gem

depending on where you have installed Pd.

If you run the shell script, GEM.INSTALL.sh, then all of the example files and documention
should be put in the correct locations.

How do I install GEM and Pd on WinNT?

See the readme for installing Pd.

TODO: there should be a install package somewhere

Unzip the GEM file that you downloaded so that it is at

pd\gem

depending on where you have installed Pd.

If you run GEM.INSTALL.bat, then all of the example files and documentation should be put in the correct locations.

How do I install GEM and Pd on linux?

See the readme for installing Pd.

Uncompress and untar the GEM file that you downloaded so that it is at

pd/gem

depending on where you have installed Pd.

chdir into <pd/gem>/src/Gnu

read the README.build

run ./configure and afterwards make

If you run make install, then all of the example files and documentation should be put in the correct locations.

How do I install GEM and Pd on macOS-X?

See the readme for installing Pd.

TODO: there should be a install package somewhere

How do I run GEM on IRIX?

   To use GEM type something like:

/usr/people/mdanks/pd/bin/pd -lib /usr/people/mdanks/pd/gem/Gem

(where /usr/people/mdanks is the path to the pd directory). Check out the README for Pd to see examples of the -lib flag. If you just try to "run" GEM, you will get an error! Notice that last word is a capital Gem. If you get a "can't find gem_setup" error, then that is the problem. Look in the GEM FAQ
for trouble shooting suggestions.

If you don't see startup messages from GEM, then something went wrong.
Also, you might need to add pd/bin to your PATH environment variable.

How do I run GEM on Win95/NT?

    It is best to start Pd from a DOS command line.  If you go to the Start menu, you should find an application called "Command Prompt" under the Program menu.  You need to change to the drive where you installed Pd.  For instance, if it is on your D: drive, just type d: at the prompt.

   To use GEM type something like:

\pd\bin\pd -lib /pd/gem/Gem

depending on where you installed Pd.

    Check out the README for Pd to see examples of the -lib flag. If you just try to double click GEM, you will get an error! Notice that last word is a capital Gem. If you get a "can't find gem_setup" error, then that is the problem. Look in the GEM FAQ for trouble shooting suggestions.

If you don't see a startup message from GEM, then something went wrong.

    Most people use the command shell to start Pd.  It is difficult to configure Pd to run from double-clicking on the icon.

    Also, you might need to add pd/bin to your PATH environment variable.

How do I run GEM on linux?

   To use GEM type something like:

/usr/people/mdanks/pd/bin/pd -lib /usr/people/mdanks/pd/gem/Gem

(where /usr/people/mdanks is the path to the pd directory). Check out the README for Pd to see examples of the -lib flag. If you just try to "run" GEM, you will get an error! Notice that last word is a capital Gem. If you get a "can't find gem_setup" error, then that is the problem. Look in the GEM FAQ
for trouble shooting suggestions.

If you don't see startup messages from GEM, then something went wrong.
Also, you might need to add pd/bin to your PATH environment variable.

How do I run GEM on macOS-X?

   To use GEM type something like: /usr/local/bin/pd -lib /Users/zmoelnig/pd/Gem

(where /usr/local/bin/pd is the path to the pd directory and /Users/zmoelnig/pd is the path where the Gem.pd_darwin resides). Check out the README for Pd to see examples of the -lib flag. If you just try to "run" GEM, you will get an error! Notice that last word is a capital Gem. If you get a "can't find gem_setup" error, then that is the problem. Look in the GEM FAQ
for trouble shooting suggestions.

If you don't see startup messages from GEM, then something went wrong.
Also, you might need to add pd/bin to your PATH environment variable.

Why doesn't GEM run?

    Notice that the -lib flag always requires Unix styles slashes, even if you are on Windows.  This means that you need to do -lib /gem/Gem, not -lib \gem\Gem

    You may also want to use the -nosound flag.  For instance, my PC has problems using audio (it leaks memory), so I just turn off the audio part of Pd.  However, other people can't get GEM to work if the -nosound is used (on Win95).  You can also try the -dac or -adc flags (for digital-analog-conversion only and analog-digital-conversion only).

[return]
 

3. Basic Objects

These are the objects you need for getting started
Basic Objects

Basic Objects


There are a number of objects which are the foundation for GEM. These objects are used in every patch and control the graphics and rendering.

[gemwin] - The window manager
[gemhead] - The start of a rendering chain
manips - Move an object in the window
geos - Render a shape

[gemwin]

The graphics window is created and destroyed with the [gemwin] object. With the [gemwin] object, you can set the default size of the graphics window, create and destroy the graphics window, turn on and off rendering, etc. All basic GEM patches will have the following [gemwin] object with these messages:

The create and destroy messages will display and remove the graphics window. The 1 and 0 messages start and stop rendering.

[gemhead]

The [gemhead] object is the start of every rendering chain. A simple patch, which is located in examples/gem_basic/gem1.redSquare.pd looks like:

This patch will render a red square. The [gemhead] object signifies the start of rendering. The [color] object sets the color for all objects after it in the chain. The [square] object renders a square into the graphics window based on the current color, texturing, and transformations. In this case, there is no texturing and no transformation.

Every rendering chain MUST start with a [gemhead]. If you do not put a [gemhead] at the beginning of the chain, then nothing will be rendered for that part of the patch.

manips

In the patch 01.basic/02.cube.pd, the [translateXYZ] object is introduced.

The graphics are transformed and moved by the manipulator objects, or the manips. GEM has the following manips:

[color] - set the color with a vector
[colorRGB] - set the color with 3 discrete values
[rotate] - rotate with an angle and vector
[rotateXYZ] - rotate with 3 discrete values
[scale] - scale with a vector
[scaleXYZ] - scale with 3 discrete values
[translate] - translate with a vector
[translateXYZ] - translate with 3 discrete values

To understand the difference between the vector and discrete values version, realize that everything in is defined in 3 dimensions. These dimensions can be XYZ values, or RGB colors.

The two translate objects above will do exactly the same thing in a patch, but they provide two different ways to do it. [translate] accepts a scalar and vector. [translateXYZ] accepts three floats which specify a point in space. The manips will transform any object which appears after it in the rendering chain.

geos

Up above, we saw the [square] and [cube] objects. The other primary geos are:

[square] - render a square
[circle] - render a circle
[triangle] - render a triangle
[cube] - render a cube
[sphere] - render a sphere
[cone] - render a cone

The [square], [circle], [cube], and [triangle] objects have a right-hand inlet to set the size of the shape. The default size is 1.

The [cone] and [sphere] objects are not perfectly smooth. They are actually composed of a number of polygons. In order to control the rendering better, the middle inlet is the size of the object, while the right-hand inlet is the number of slices to define the shape. Take a look at the patch gem_basic/gem3.sphere.pd to see how the number of slices can change the look of a sphere. Don't worry about the [world_light] object, it is just there to make it easier to see the difference in the number of slices. Make sure to click the 'lighting 0' message before closing the patch (if you don't, then other patches will probably be completely black until you quit and restart pd/GEM).

Your graphics window should look like this for 5 and 15 slices:

Obviously, the more slices that you use, the better the sphere looks. However, each slice adds more polygons, which can slow down your frame rate. In computer graphics, there is always a trade off between resolution and speed.

[return]

4. Images

Dealing with Images
Images

Dealing with Images

Images are files which are loaded into GEM. The images can be manipulated, applied to objects, and used in any number of different ways. In this section, you will load in an image and display it on the screen. This section will not apply the images to a geo; that occurs in the next part of the manual.

The pix objects are GEM objects which deal with pixels. They do everything from loading in images to applying filters to the data. The objects in this section of the manual only load in pix data from outside sources. How you actually display the image is up to you. The most common usages are with [pix_draw] and [pix_texture].

Warning: [pix_draw] is almost always slower than [pix_texture]. Because [pix_draw] is easier to use than [pix_texture], it is used in these examples. However, in any real usage or piece, [pix_texture] should always be used instead. [pix_draw] is slow because PC graphics accelerators do not provide hardware acceleration for that functionality. [pix_texture] does have hardware acceleration and will be much faster.

[pix_image] - load in an image
[pix_multiimage] - load in multiple images
[pix_movie] - load in a movie file
[pix_video] - use a real time video source

[pix_image]

[pix_image] is used to load in images. Images can be in a variety of different formats, including TIFF, JPEG, and SGI formats. The patch gem_pix/gemImage.pd is the simplest use of the [pix_image] object. In this patch, the [pix_image] object loads in the file dancer.JPG.

As is the case with every GEM chain, this patch starts with the [gemhead] object. The next object is [pix_image], which actually loads the image. [pix_image] makes the file dancer.JPG the current pixel data, which will be used in all subsequent operations in the chain. The [translateXYZ] object is used to move the image around. Finally, the [pix_draw] object renders the pixel data to the screen.

The patch mentions that changing the Z in [translateXYZ] does not change the size of the image, as would occur with a geo object like [square]. This is because [pix_draw] simply draws the pixel at the current raster position, without any transformation. If you want to change the size on the fly and rotate the image, you need to texture map the pix, which is described in the next section.

[pix_multiimage]

The [pix_image] object only loads in one image at time. If you try to change the image rapidly while the patch is running, you will notice a lag every time it has to load in a new file. To avoid this lag, there is another object called [pix_multiimage]. If you look at patch gem_pix/gemMultiImage.pd, you will see this object in action.

Basically, the * in the file name is replaced by the number that you pass in. This allows you to play sequences of images with random access. The one downside is that every image is loaded into memory when the object is created, so you need to have a lot of RAM to use it.

[pix_movie]/[pix_film]

These objects are used to read movie-files from disk (or if supported from the internet). The movie is streamed off of disk, using whatever decompression libraries are installed on your computer. On Windows AVI movies seem to work fine, but there is also a prelaminary support for quicktimes (and mpeg). On macOS-X all formats supported by the system (basically: quicktime) should work ok. On linux the support is highly depending on what libraries are installed during compile time. There is support for MPEG (with libmpeg1 or (preferred:) libmpeg3), quicktime (either libquicktime or quicktime4linux; most likely you will not be able to decode quicktimes with proprietary codecs) and AVI (with libavifile which is able to utilize windows-dlls for (proprietary) codecs). There is also some rudimentary support for FFMPEG. The right inlet of [pix_movie] accepts a number to specify the frame to display. Look at 04.pix/04.movie.pd for an image.

A key fact of [pix_movie] is that it immediately sends the movie data to OpenGL as a texture map. This means that you do not need the [pix_texture] object in your chain. This also means that you cannot process the movie data with pix objects. The main reason for this is that it removes the need for a copy of all of the movie data. If you want to apply some image-processing, you will have to use [pix_film] (and [pix_texture] for texture-mapping).

Some of the geos will not texture map the [pix_movie] data correctly. Cone and sphere do not use texture coordinates when they are provided, so when you display a movie on one of these objects, you will have a black region (unless your movie size is a power of two...however, most movies are 320x160 pixels or something). This will be fixed in a future release.

pix_video

The "image" can come from the [pix_video] object. This means that you can use a real-time video source and display it on the screen.

You can play with [pix_video] with the patches in 04.video/. The patches are explained in more depth in the advanced section of the GEM manual.

[return]

5. Texture mapping

Texture mapping is the act of applying pixel data to a geometric object.
Texture mapping

Texture Mapping

Texture mapping is the act of applying pixel data to a geometric object. In GEM, this is achieved with the [pix_texture] object. It is important to understand that the [pix_texture] object merely sets the pix as the current texture. It does not do any rendering! You need to use a geo object which does texture mapping. All of the basic geo objects can texture map, such as [square] or [sphere].

A simple example of texture mapping is the following patch:

This patch can be found at 07.texture/01.texture.pd. Change the number box connected to the rotate object to see what a texture map on a cube looks like.

The [pix_image] object loads in the fractal image file. The [pix_texture] object says that the pix data should be used as a texture map. Notice that this is different than the previous manual section when we used the [pix_draw] object. The final object in the chain is the [cube] object. Because we have enabled texture mapping with the [pix_texture] object, the cube takes the pix data and applies it to the geometry.

Texture mapping can be used with any GEM object. In the previous manual section, you saw how to load in pix data with a variety of objects, including [pix_multiimage] and [pix_video]. All of these objects can be used with the [pix_texture] object.

Because the pix data is applied to geometry, you can move, rotate, and scale the image. This is extremely useful on the [square] object. Instead of doing a one-to-one pixel mapping as occurs with the [pix_draw] object, you can resize and reshape the image.

OpenGL originally required that images must have dimensions that are power-of-2, such as 64, 128, or 256. This restriction has been released with recent gfx-cards (like some radeon/nvidia products). However, if the width or height of an image is not a power of two, then the [pix_texture] object will take care of this, and still render it (depending on you hardware with some tricks). You can thus texture images of any size, but since this is based on tricking the texture-coordinates, [pix_coordinate] might not give the wanted result any more.

The example patch 07.texture/02.moveImages.pd is a much more complex patch which uses alpha blending to create a transparent object, in this case, the dancer. Make sure to turn on the rotation with the [metro] object.

People have been asking how textures are handled in GEM. Here is a long explanation from an email which I wrote.

Here is how textures are dealt with under OpenGL and hardware accelerators. This can obviously change in the future, but right now, I am fairly certain that the info is correct (I make games in my day job, so I have vested interest in this :-)

The amount of memory (VRAM) on the card (12mb for Voodoo2, 16mb for TNT, 64mb for GeForce2, etc) is used for both textures (TRAM) and frame buffer space. If you have a large rendering window, like 1600x1200, it will take up 1600x1200x4x3 in 32-bit mode with double buffering and a Z buffer (or 23mb). Most people run at TV resolution, like NTSC, so it takes 640x480x4x3 = 3.7mb All of the space left is for textures onboard the card (FYI, if you have heard that people are having problems with the PlayStation2, notice that it only has 4mb of VRAM...not much onboard texture space, huh? :-) Thankfully it has an extremely fast DMA bus)

Sooo, when GEM "creates" a texture, it immediately tries to send the texture to the card, which uses some of the left over space in the VRAM. If you had a 640x480 window on a Voodoo2, you have ~8mb of texture space left over. On a GeForce2, ~60mb. The problem is what happens if you want more textures than can fit into TRAM. OpenGL requires that the video drivers deal with the problem, so GEM doesn't care too much (more about this later).

In most cases, the drivers cache the textures in main memory and if a texture is requested for rendering and it isn't resident on the card, it will download it. If you have AGP, then this is pretty quick, although none of 3dfx cards really take advantage of this (ie, those cards are about the same speed as the PCI bus). So depending on the number of textures, and how complex the scene is, you might be able to display more textures than you have TRAM.

One slowdown that can happen with GEM is that it makes a copy of the image before sending it down the chain of objects. If you are constantly changing images with a pix_multiimage, this can be a performance hit, but you can modify the actual pixel data with the pix objects. The pixels aren't sent to the graphics card until the pix_texture object is reached.

GEM tries to help with this with a few objects. pix_imageInPlace acts much the same as pix_multiimage, but it downloads _every_ image in the sequence to the card when a download message is recieved. It also immediately turns on texturing, instead of making a copy (ie, you don't need a pix_texture object). Much faster, but not as flexible. pix_movie does much the same thing. It sends the pixel data without copying it if there is a new frame to display.

The entire pix system uses a caching system so that the copying and processing only occurs if something actually changes. For example, if you had a pix_threshold object, it would only process when rendering started...and every time that the values actually changed. You can use pix_buf to isolate parts which don't change from those that do, but it involves another copy.

On the Voodoo2, the hardware itself limits textures to 256x256...this will never change. The newest Voodoo5 boards have a higher texture size.

If you load the _exact_ same image (this means the exact same file/path name), then the pix_image has a cache system which means that it is only loaded into the
computers memory once. However, each pix_image still sends its own copy down to the gfx card.

You could use a single [pix_image]/[pix_texture] with [separator] to do this...I have done it a lot in the past.

The reason that [pix_image] doesn't share the actual texture data is that you can modify the pixel data with other pix objects...[pix_image] doesn't actually send the texture data to the gfx card, [pix_texture] does.

[return]

6. Pixes

Image Processing
Pixes (image processing)

Image processing

The pix objects are used to do image processing to pixel data. If you load in an image with [pix_image], then you can change what the image looks like before rendering it out

In general, processing images is extremely expensive, so you probably cannot have that many active pix objects. GEM only reprocesses images when the source image changes or one of the parameters for a pix object changes. This means that GEM will only process an image when something is different, instead of every frame. If you want to do a lot of processing at start up, but then not change anything once the patch is running, GEM will only do the computation once.
Modern CPUs use SIMD (Single Instruction - Multiple Data) (like MMX, SSE2, altivec) to make pixel-processing more effective (by processing data parallely). Until now, only the macOS version of Gem has support for SIMD for some pix-objects. MMX/SSE2 boosts will hopefully come in future Gem-releases.

The pix objects are divided into two general groups, those which take one input, and those which require two input images. For example, [pix_invert] will "invert" all of the pixels (if a pixel is white, it will change to black), while [pix_add] will add two images together.

Only some of the pix objects are described here. Look in the reference patches for explanations for the other pix objects.

[pix_invert] - invert the pixel data
[pix_add] - add two pixes together
[pix_mask] - create an alpha mask
[pix_convolve] - convolve a pix with a kernel

[pix_invert]

[pix_invert] inverts the pixels in an image. To use [pix_invert], simply make sure that you have already loaded an image into the chain. In the following patch, the fractal image will be inverted.

Here is the difference between the fractal image and the inverted version.

pix_add

[pix_add] does what you would expect. It adds two images together.

This patch adds the fractal image with a car image. The processed image will often contain a lot of white pixels, because the data is just added together. This occurs in the resulting image, shown below.


pix_mask

[pix_mask] is used to create an alpha mask from another image. In the following example (gem_pix/gemMaskDancer.pd), the fractal image's alpha channel is replaced by the dancer image. If the [alpha] object was removed, then you would just see the solid fractal image (because the alpha channel wouldn't be used).

In other words, images are composed of a red, a green, a blue, and an alpha channel. The alpha channel is the transparency of the pixel. [pix_mask] only modifies the alpha channel and does not touch the red, green, or blue data.

The result is this image.

pix_convolve

[pix_convolve] convolves pix data with a convolution kernel. Basically, you can get really nice effects if you choose the correct kernel...and garbage if you choose the wrong one.

Edge detection is done with a convolution kernel, as is smoothing. The biggest problem is that convolving an image is about the most expensive operation that you can do in GEM.

Look at gem_pix/gemPixConvolve.pd to get an idea of some of the kernels that you can send to [pix_convolve] and the effects that you can get.

If you want to learn the math behind convolution, then find any standard image processing (or audio processing book, this is just 2D convolution).

[return]

7. Lighting

addings Lights and Magic
Lighting

Lighting

Lighting is an important factor is how we perceive the quality of an image. For example, without lighting and shading, a sphere would just look like a circle. GEM provides two types of lights, a local light and world light.

OpenGL uses a vertex lighting model. This means that for every vertex in the scene, the influence of the light is calculated. The color for the polygon is then modified by the light value of all of the vertices. This generally produces a very smooth effect, but you will occasionally run into rendering artifacts, especially if you use local lights. For example, imagine you have a local light close a large square. The corners of the square are far away from the light, so none of them will be lit very brightly, even though the light itself is very close to the surface of the square.

It is important to realize that lighting is an expensive operation to use. The number of polygons that you will be able to render will be much lower if lighting is turned on. As usual, the complexity of the scene and the speed of your computer and graphics card will greatly affect your frame rate.

GEM has only a maximum of 8 lights at one time. If you try to create more lights than that, you will get an error message.

Activate lighting
[world_light] - A directional light
[light] - A point light in the world
Moving lights

Activate lighting

Lighting is activated by sending a message to [gemwin]. If you send "lighting 1", then lighting will be turned on. If you send "lighting 0", then lighting will be turned off. The lighting state is kept even if you destroy the gemwin. This means that if you close a patch and open another one, the lighting will still be the same.

Individual lights can be turned on and off by sending a 1 or 0 to the left inlet of the light object itself.

[world_light]

A [world_light] is a light which exists infintely far away. This reduces the computation needed, so your patch can run faster, but it means that all of the light rays are parallel. The [world_light] is good for objects like the sun and other lighting affects. This means that translating a [world_light] has no effect, although rotation does.

The following patch is 03.lighting/01.world_light.pd.

The [world_light] has one extra inlet. The right inlet accepts three floats to set the color of the light. A [color] object would do nothing. In this case, the light is being set to purple. The [world_light] also accepts a debug message. The debug message turns on and off a graphical representation of the light in the scene. The [world_light] looks like a cone. The cone shows the direction that the light is coming from. Remember that the actual position of the light does not matter, so geos behind the cone will still be lit. It is the direction of the light that matters. This is why you can rotate the light.

[light]

A [light] object generates a point light in the world. Because the light is local to the scene, there is more math to generate the effect of the light on the vertices. However, unlike a [world_light], you can translate the [light] object.

Below is the patch 03.lighting/02.light.pd.

The [light] object has a right inlet for the color, just light the [world_light] object. As this patch shows, the light can be moved around the scene with both [rotate] and [translate] objects. If you were to set the translate X value equal to 1.0, then the sphere would not be lit at all. This is because the light would be inside of the sphere. When you turn on the debug representation, it is a sphere with its origin where the light it. The [light] object does not have any size. It exists as a point source.

Moving lights

The patch 03.lighting/03.controlLights.pd allows you to move a [light] and [world_light] object in the same scene to see the difference between the two objects.

The patch 03.lighting/04.moveSpheres.pd is an example which moves two spheres around the world. Turn on and off the individual lights for a demonstration of a local versus infinite light.

The patch 03.lighting/05.materials.pd uses the material objects to selectively control the color of the object. Notice that the diffuse object sets the "overall" color, while the specular objects sets the bright reflective area where the light directly shines.

[return]

8. Utilities

a number of object that used to be part of Gem, but can now be found in a separate library "markEx"
Utility objects

Utility objects

There are a number of objects which were written to make it easier to use both GEM and pd.  For instance, you often pass around 3 floats at a time in GEM, either for position or colors.  To help with this, there are a collection of vector objects.  Use the list below to find out about the objects.

These objects used to be in a separate library called MarkEx, but they have now been folded into GEM.

counter - count the number of bangs
average - average a series of numbers together
change - only output when there is a change in the number
invert - invert a number
randomF/randF - floating point random number
tripleLine - line object for 3 values
tripleRand - three random numbers
vector objects - process a series of numbers
hsv2rgb and rgb2hsv - convert between RGB and HSV color space

counter

The inlets are:
bang (increment or decrement the counter)
set direction (1 = count up, 2 = count down, 3 = count up and down)
set low value
set hight value
The outlet is the current count.

So in this case, the top counter will count up from 1 to 10.  The bottom counter will count up from 2 to 5.

The counter also accepts the messages reset and clear.  Reset immediately sets the counter to its low value and outputs the value.  The clear message means that the next bang will set the counter to its low value.

average

The average object just averages a series of numbers as they come in.  The left inlet accepts a single float.  It then outputs the current average.  The default number of floats to average together is 10, but that can be changed by sending a new value to the right inlet.

The average object also accepts the messages clear and reset.  Clear will immediately set all of the values that the object has been storing for averaging to 0.  With the reset message, you must pass in a number to set all of the values.

change

Change only accepts a number into its left inlet.  If the number is the same as the last number sent to the change object, then it does nothing.  If the number is different, then the change object will output the new number and store it for the next comparision.

This object is very useful for the == object and others like it, since they send a 0 or a 1 every time they do a comparision, and you usually only care when the state actually changes.

invert

The invert object is very simple.  If the number sent to its left inlet is equal to 0., then invert outputs a 1.  If the number is not equal to 0., the invert outputs a 0.

randomF/randF

One problem with the random object in pd is that it only sends out integers.  This a real problem in GEM, where you often want a value between 0 and 1.  randomF is exactly like the random object.

When the left inlet gets a bang, randomF outputs a random number between 0 and the given range.  The range can be set with a number to the right inlet.

randF is just an alternate name for randomF.

tripleLine

The line object is really great for dealing with a single number.  To do a line with 3 values, like an RGB color value, means that you have to unpack, do a line, then repack the number.  Not only is it a pain, but it expensive computationally.

tripleLine behaves just like the line object, only it accepts three numbers to interpolate between.  In the example, tripleLine will interpolate from the current values to 1., .2, .4 over 1000 milliseconds.  The default output resolution is 50 milliseconds, which is the same default rendering time.  Going faster with GEM objects will not produce any benefit, unless you increase the frames per second.

tripleRand

Just as using tripleLine makes it easier to interpolate between 3 values at once, tripleRand makes it easy to generate three random values.  In the above example, when the bang is sent, tripleRand will create three values and output them, with the first between 0 - 1, the second between 0 - .5, and the third from 0 - .8.

Vector objects

The vector math objects are
vector+ or v+
vector- or v-
vector* or v*
vector/ or v/
All of the above objects perform math on a list of numbers.  The left inlet accepts a list of numbers of any length.  The right inlet accepts a single value, which is the operand for the computation.  In other words, they work just like the normal *, +, -, and / objects, except they can handle more than one number in the left inlet.

There are two other objects which are also useful.

The first is vectorabs or vabs. It computes the absolute value on a list of numbers.

The second object is vectorpack or vpack. vpack accepts a list of numbers in the left inlet and a single number into the right inlet.  The output is a single list of numbers that is the vector with the single number appended to the end.  This is very useful when you want to change the time for a tripleLine without unpacking and repacking all of the data.

hsv2rgb and rgb2hsv

These two objects convert three numbers between HSV and RGB color space.  HSV stands for hue, saturation, and value.  The simple way to think of HSV space is that hue is the "color", such as red, blue, etc, the saturation is how intense the color is, and the value is how bright the color is.

You can get some really nice effects by varying the hue of a color, because the brightness will not change while you do it.

[return]

9. Writing new objects

Gem can be extended just like Pd...
Writing new objects

Creating new GEM objects

Look at the source code :-)  GEM is written in C++, which means that you have to jump through some hopes to interact properly with Pd, which is written in C.  If you look in Base/CPPExtern.h, you will see a collection of macros which you can use to help you create new objects.  Use one of the GEM objects which is closest to what you want to do as a template.

One problem on SGI...you will need to

setenv LD_LIBRARY_PATH "/where/ever/pd/gem"

so that rld (the run-time linker) can find the GEM dso.  Because you are linking with GEM, Pd isn't involved with the run time linking process; it is all done when Pd calls dlopen.

On NT, there is much the same problem...

set your PATH environment variable to \where\ever\pd\gem

or

make sure that your new .dll is located in the same directory where GEM is.

On NT, all of the classes and functions are exported through declexport/declimport.  You shouldn't have to do anything to call the functions.  I have not had any problems making other dll's which are loaded into Pd at runtime.  You need to make certain that you are exporting the correct functions.  If your dll cannot find the gem.dll, then it will silently fail.

And of course, e-mail IOhannes m zmölnig (zmoelnig@iem.at) if you have any problems, questions, or solutions

[return]
 

10. Gem FAQ

random questions with answers
Gem FAQ

GEM FAQ


* : new question
+ : changed question

QUESTIONS

GENERAL
1.1) What is GEM?
1.2) What is Pd?
1.3) What platforms do GEM and Pd run on?
1.4.0) How do I install GEM and Pd on IRIX?
1.4.1) How do I install GEM and Pd on linux?
1.4.2) How do I install GEM and Pd on WinNT?
1.7) What is a good intro to OpenGL?
1.8) Are there any web sites for Pd or GEM?
1.9) What libraries does GEM use? (aka: Who does Mark want to thank?)
1.10) Are there any restrictions on GEM?
1.11) How do I use GEM in a performance?

USING GEM
2.1) How do I (???)
2.2) How do I make GEM run?
2.3) Why doesn't GEM run?
2.4) I've got it running. Now what?
2.5) On IRIX 5.3, why does GEM dump with an rld error?
2.6) Why can't I compile GEM on IRIX 5.3?
2.7) Why is GEM slow in general?
2.8) Why is GEM slow on IRIX?
2.9) Why is GEM slow on WinNT/Win95?
2.10) Why is GEM slow on Linux?
2.11) If I resize the window, everything looks strange.
2.12) Can GEM run on a 3Dfx Voodoo card?
2.13) Will GEM support hardware transform and lighting (T&L) ?
2.14) I get an error "GEM needs Truecolor visual support".

VIEWING OBJECTS
3.1) Why does everything seem dim?
3.2) Why does everything seem dark?

TEXTURE MAPPING
4.1) My image doesn't appear. What is going on?
4.2) My image looks strange. What is going on?
4.3) Why does GEM say that it can't handle a gray image?
4.4) What image formats can GEM handle?
4.5) What movie formats can GEM handle?
4.6) Why is pix_draw so slow?

WORKING WITH PD
5.1) Why do I get clicks in the audio?
5.2) How do I get audio data to GEM?
5.3) Why can't GEM find an image/model file?
5.4) How can I optimize my patches?

WRITING NEW GEM OBJECTS
6.1) How do I write a new GEM object?
6.2) What are the default OpenGL states?

OBJECT SPECIFIC
7.1) Why doesn't <object> exist on <platform>?
7.2) Why doesn't gemtablet work?
7.3) I don't want GEM to take over my tablet. How do I stop it?
7.4) Why doesn't gemmouse work in IRIX/Linux?
7.5) Why doesn't gemorb work?
7.6) What is wrong with pix_video in WinNT?

ANSWERS

GENERAL


1.1) What is GEM?

GEM is the Graphics Environment for Multimedia. It was originally written by Mark Danks to generate real-time computer graphics, especially for audio-visual compositions. It originally ran under FTS/Max (which is why you might see some papers reference it), but all new development is under Pd.

You can get GEM at http://gem.iem.at/

GEM was sponsored by a grant from Intel (http://www.intel.com)

GEM was ported to linux by Günter Geiger

GEM is now maintained by IOhannes m zmölnig.

the core-development team consists of

  • chris clepper
  • günter geiger
  • daniel heckenberg
  • james tittle
  • IOhannes m zmölnig
lots of contributions are made by various people (thanks to all of them)

----
1.2) What is Pd?

Pd is a real-time environment for audio and MIDI. It was written by Miller Puckette, who created FTS/Max when he was at IRCAM. Basically, Pd can be seen as the next generation of real-time visual programming languages. GEM runs inside of the Pd environment.

You can get Pd at http://www.crca.ucsd.edu/~msp/software.html

Pd is sponsored by a grant from Intel (http://www.intel.com)

----
1.3) What platforms do GEM and Pd run on?

GEM and Pd run on Windows (95, 98, ME, NT 4.0, 2000, XP), linux and macOS-X (>10.2). SGI-Irix (> 6.2) used to be supported but i don't have any prove that it still works). Günter Geiger has done an initial port of GEM and Pd to Linux http://gige.epy.co.at).

GEM is now maintained by me and developed by a team of several independent programmers (see section 1.1)

----
1.4) How do I install GEM ?

----
1.4.0) How do I install GEM and Pd on IRIX?

See the readme for installing Pd.

GEM should be at

pd/gem

If you run GEM.INSTALL.sh, then all of the example files and documention should be put in the correct locations.

----
1.4.1) How do I install GEM and Pd on linux?

See the readme for installing Pd.

GEM should be at

chdir to <gem>/src/Gnu and build Gem following the instructions in the README.build (./configure; make)

If you then make install, then all of the example files and documention should be put in the correct locations.

if you are using debian, Gem should be available via apt

if you are using an rpm-based distribution, check out the builds at planetCCRMA

----
1.4.2) How do I install GEM and Pd on WinNT?

See the readme for installing Pd.

unzip GEM so that it is at

pd\gem

If you run GEM.INSTALL.bat, then all of the example files and documentation should be put in the correct locations.

there is also an installer for windows.

----
1.4.3) How do I install GEM and Pd on macOS?

See the readme for installing Pd.

there is also an installer for macOS.

----
1.7) What is a good intro to OpenGL?

The best book is the OpenGL Programming Manual by Mason and Woo. This is also called the "Red Book". If you search the web, there are many sites on OpenGL. A good starting point is http://www.opengl.org. Also, Mark Kilgard (who used to work for SGI) has a wonderful site with lots of links (http://reality.sgi.com/mjk) Also, Normal Lin has written another great book on 3D-graphics under linux

----
1.8) Are there any web sites for Pd or GEM?

Except for the ones noted above, there is the Japanese installation page at
http://www.rinc.or.jp/~kotobuki/gem/index.htm

There is a Pd mailing list. Subscription info is on IEM's site http://www.iem.at/mailinglists/pd-list

One of pd's unofficial home-pages is at http://pd.iem.at hosted by the Institute of Electronic Music and Acoustics, Graz, Austria

Also hosted by the iem is the site of the pd-community http://www.pure-data.org

An interesting place might also be Günter Geiger's size http://gige.epy.co.at/

there are lot's of other cool pages (search the net...)

----
1.9) What libraries does GEM use? (aka: Who does Mark want to thank?)

All copyrights and license info can be found in
GEM.LICENSE.TERMS
Thanks to Sam Leffner for libTiff, the TIFF image loader.
sam@engr.sgi.com
ftp://ftp.sgi.com/graphics/tiff/
Thanks to Masayuki Matsumoto for fstimage for OpenGL, the SGI
image loader.
matumot@dst.nk-exa.co.jp
Thanks to the Independent JPEG Group for libjpeg, the JPEG image loader.
jpeg-info@uunet.uu.net
ftp://ftp.simtel.net/pub/simtelnet/msdos/graphics/
Thanks to Mark Kilgard at al. (and SGI) for glut, the openGL Utility Toolkit
http://www.pobox.com/~ndr
Thanks to Stephane Rehel for GLTT, the OpenGL TrueType render.
rehel@worldnet.fr
http://home.worldnet.fr/~rehel/gltt/gltt.html
Thanks to David Turner, Robert Wilhelm, and Werner Lemberg for
Freetype, a TrueType font rendering engine.
turner@enst.fr
robert@physiol.med.tu-muenchen.de
a7971428@unet.univie.ac.at
http://www.physiol.med.tu-muenchen.de/~robert/freetype.html
Thanks to the MPEG Software Simulation Group, for libmpeg, the
MPEG-2 Encoder/Decoder
mssg@mpeg.org
http://www.mpeg.org/MSSG/
Thanks to Heroine for quicktime4linux a quickime Decoder and libmpeg3, another MPEG-2 Encoder/Decoder
MPEG-2 Encoder/Decoder
mssg@mpeg.org
http://heroinewarrior.com/
Thanks to LCS/Telegraphics for Wintab, the Windows tablet library.
wintab@pointing.com
Thanks to David McAllister for the Particle System library.
davemc@cs.unc.edu
http://www.cs.unc.edu/~davemc/Particle/
Thanks to John Stone for the Space Orb library, libOrb
j.stone@acm.org
http://www.umr.edu/~johns/projects/liborb/

----
1.10) Are there any restrictions on GEM?

GEM is under the Gnu Public License. This basically means that it will always be free software.Check out http://www.gnu.org for more information and read the full license in GnuGPL.LICENSE in the GEM release.

----
1.11) How do I use GEM in a performance?

This is a constant problem, because there is no consistent way to display video on any platform. Also, you usually do not want to send the entire screen, but only the GEM window. It is also useful to be able to edit/control the Pd patch window while the patch is actually running.

On SGIs, the best way is to get a video out option. On the SGI O2, Impact, and Onyx (Mark has used all of these), there is a simple connector or breakout box to do video. If you run the video out program, then you will get a rectangle on your screen which shows what is being sent out the video connector. Make your GEM window a little larger than 640x480 and center it in the rectangle. You can now project this with a standard video projector.

On PCs it is a bit harder. Several modern video-cards have the possibility to output several screens (either 2 (or more) VGA-screens or 1 VGA-screen and 1 TV (Composite or S-HVS) or a combination with DFTs) If you have a Canopus Voodoo2 card it has a video and s-video output on it. As described in question 2.12, you can get a Voodoo to work with GEM. If any one else has a better solution, please let me know. The nVidia Riva TNTs require that you output the full screen, so this is not a very good option. You can use a video scan convertor. Some of them only display a part of the scene, which is exactly what you want.

With modern multi-headed cards it is more simple: Configure your card to display the desktop spread over your multiple screens (e.g.: from left-to-right). On windows and macOS you can do this via the display-properties dialog. On linux you will have to edit your /etc/X11/XF86Config-4 file either by hand or (if your system supports it) via an appropriate editor (yes, nowadays there are some). Now create your gem-window on the second screen: it should have the same dimensions as the 2nd screen (e.g: "[dimen 800 600("). to place it at the second screen use the offset (e.g: if your primary sreen (the one you want for patch-editing) has the dimension 1024x768 use "[offset 1024 0(", which will create the gem-window 1024 pixels right of the upper-left corner of the total screen (and 0 pixels below it), which is exactly the upper-left corner of the 2nd screen. You most probably want to turn off the borders with "[border 0(".
Note: some grafix-card have openGL-hardware-acceleration only on the 1st screen (so you should create the gem-window on the 1st screen and move your patches to the 2nd screen)

If you are using an XServer for displaying (under linux) you can also use another computer for rendering. You can specify the place where the gem-window should be created with something like "create <render.host>:0.0"

If you are doing audio with graphics, the only solution to prevent clicking (question 5.1) is to run 2 computers and have them communicate with netsend/netreceive. We are working on making Pd/GEM multi-processor friendly, so if you have a multi-processor system, you can run everything on one machine eventually.


USING GEM


2.1)How do I (???)

Many of the general usage questions are probably answered in the manual or release notes. The pd mailing list is also a good place to find answers as well.

2.2) How do I make GEM run?

GEM is not an executable. It requires Pd to work and is loaded in at run time. For example, I have an alias on the SGI which does

/usr/people/mdanks/pd/bin/pd -lib /usr/people/mdanks/pd/gem/Gem

and on WinNT

\pdDir\pd\bin\pd -lib /pdDir/pd/gem/Gem

on UNIX-systems you will probably want to use a .pdrc file, where you can put the command-line arguments for pd that you "always" need.

If you don't see a startup message from GEM, then something went wrong.

Most people use use the command shell to start Pd. It is not very difficult to configure Pd to run from double-clicking on the icon.

----
2.3) Why doesn't GEM run?

Notice that the -lib flag always requires Unix styles slashes. This is the case even on Windows.

You may also want to use the -nosound flag. For instance, my PC has problems using audio (it leaks memory), so I just turn off the audio part of Pd. However, other people can't get GEM to work if the -nosound is used (on Win95). You can also try the -dac or -adc flags (for digital-analog-conversion only and analog-digital-conversion only).

----
2.4) I've got it running. Now what?

Try out the manual. It will step you through the basics.
You will also want to look at the example files. Assuming that everything is installed correctly, you can get to the examples by going to the Help menu in Pd and selecting examples. A bunch of the patches should start with gem<something>. The best one is gem/01.basic/01.redSquare.pd It puts a red square up on the screen and allows you to rotate it. gemImage.pd shows how to load in a TIFF file. gem/03.lighting/04.moveSpheres.pd moves two spheres around the screen. Try the other ones.
Most of the GEM objects have test patches which give some information about the various controls for the object.

----
2.5) On IRIX 5.3, why does GEM dump with an rld error?

GEM only works under IRIX 6.2+. The rld error is probably something about not having glBindTextureEXT (or something). OpenGL 1.0 has some extensions to speed up texture mapping (which are an integral part of OpenGL 1.1). However, these don't exist on IRIX 5.3. If you recompile GEM (see the next question), things should work fine.
I don't have access to an IRIX machine, so don't expect any builds from me. Upgrading to IRIX 6.2+ is worth it.

----
2.6) Why can't I compile GEM on IRIX 5.3?

There was probably an error saying that the compiler couldn't find the file "dmedia/vl_vino.h" in pix_videoSGI.cpp. IRIX 6.2+ adds new functionality to the media libraries which makes life much easier. You cannot compile pix_video or pix_indycam as is under 5.3. You can remove them from the Pix/Makefile and from the linker part of the global Makefile. You will also need to recompile the Td and Tiff libraries.

There shouldn't be any problems doing this. I haven't tried any of this, so if it works for someone, please let me know.

----
2.7) Why is GEM slow in general?

Examine what you are doing. If you are constantly changing textures, then this is probably your problem. If you have models with a million triangles, then this is probably the problem. Compare what you are doing with realistic specs on your system. Some systems slow down when they have to draw very large polygons (slow fill rate).
You can also turn on profiling to see how long it takes to render a frame. Send a profile message to the gemwin object. The number that is printed is the number of milliseconds one frame takes to render. 50 milliseconds is 20 frames per second. 'profile 2' is good if you want to see how long the image processing is taking.
profile 0 - turn off profiling
profile 1 - turn on profiling
profile 2 - turn on profiling and don't cache pixes

----
2.8) Why is GEM slow on IRIX?

If you are having major slowdowns, then please let me know. I have gotten very good performance on most machines (Indy, O2, Impact, Onyx2).

----
2.9) Why is GEM slow on WinNT/Win95?

You probably don't have hardware acceleration. You can use software rendering, but it basically useless except for extremely basic patches. You can get a good graphics accelerator for really cheap these days. I recommend a card based on nVidia's chipsets, such as the TNT2 or GeForce, but there are other companies such as 3dfx and Matrox. Make sure that you are running the latest drivers for your card. The basic drivers that come with the cards are usually very bad.
Also, PCs don't deal with lots of texture maps very well (they are bus limited, at least until AGP), so if you are trying to use lots of constantly changing texture maps (especially with [pix_multiimage], [pix_video] or [pix_film]), that will cause problems.

----
2.10) Why is GEM slow on Linux?

It is because you have to use Mesa, which might be running iin software. Mesa (http://www.mesa.org) is an awesome package by Brian Paul (brianp@avid.com) which "emulates" OpenGL. Basically, it is a fully compliant OpenGL package, but it isn't officially sanctioned by the OpenGL ARB, such, it is doesn't have the OpenGL name. There is an acceleration package for the many graphics card, but I don't know anything about it.
nVidia is being very supportive of Linux: their TNT2 and GeForce cards work under Linux with hardware-acceleration of openGL. (but the drivers are proprietary)
radeon cards should also be supported very well under linux (even with open-source drivers)

----
2.11) If I resize the window, everything looks strange.

GEM doesn't trap resize events in IRIX or Linux (this is not a problem in WinNT). This means that OpenGL doesn't have the correct information to render properly. If you want to resize the window, send a 'dimen x y' message to gemwin before you create the window.

----
2.12) Can GEM run on a 3Dfx Voodoo card?

I (this is: Mark Danks) have a Voodoo2 card, which runs fine under WinNT. I use the OpenGL beta driver from 3Dfx at work all the time without any problems and, except that the Voodoo takes over the full screen, it seems to work fine. You will need to download the OpenGL Beta driver from 3Dfx's web site at http://www.3dfx.com and put the OpenGL32.dll into the same directory as pd.exe (NOT gem.dll). Debugging patches is much easier if you have two monitors, one for the 3-D card and one for the 2-D card.

IMPORTANT: You MUST set the environment variable

GEM_SINGLE_CONTEXT = 1

to make the Voodoo card work. It will make a window 640x480 (which is the correct size for TV video out on my Canopus V2 card). On WinNT, right click "My Computer" and go to "Properties". On the "Environment" tab, you need to add the variable "GEM_SINGLE_CONTEXT" with a value of 1.
Resizing the GEM window with a Voodoo card is not a great idea. The Voodoo card can only display certain window sizes and will clip the graphics.

For the tech heads in the audience...I create an OpenGL context at startup and never actually display its associated window. This means that GEM objects can create display lists, call OpenGL commands, etc. in their constructors, even if no window is actually being displayed. However, with the Voodoo card, there can only be one OpenGL context. So, instead of creating one context and just holding onto it in the background, I create the normal GEM window and associate the OpenGL context with it...and the user can never destroy or close that window.

----
2.13) Will GEM support hardware transform and lighting (T & L)?

Absolutely! Unlike some other APIs, OpenGL will automatically use hardware accelerated transform and lighting if the card has it. GEM gets great performance from cards like nVidia's GeForce.

2.14) I get an error "GEM needs Truecolor visual support".

This error means that your X display is running with paletted colors, which is the result of limited color depth. If you start the X display with

startx -- -bpp 16

or some higher number, then it should work fine. 32-bit color is the best.


VIEWING OBJECTS


3.1)Why does everything seem dim?<

You probably turned on lighting but don't have any lights in the world. Either add a light with world_light or light or turn lighting off by sending a message 'lighting 0' to the gemwin. You can also send a reset message to gemwin to set it back to the startup state (which doesn't have any lighting).

----
3.2) Why does everything seem dark?

See question 3.1.
If you are using view in your patch to change the viewpoint, you may not be pointing in the correct direction. You also might have translated everything outside of the current viewport.
Also, if you have been using single buffering ('buffer 1' message to gemwin), then you might still be in that mode. Either send a 'buffer 2' message or a 'reset' message to gemwin. Then, destroy and create your window.


TEXTURE MAPPING


4.1) My image doesn't appear. What is going on?

Normally images have to be texture-mapped onto Geos. You have to use [pix_texture] to map the current image onto a Geo. "Current" means that any pix-manipulation that is done after texturing will not be displayed.

Any Geo has a color (which is initially set to white). If you have set the color to black, your Geo (including the image) might be very dark. If you are using alpha-blending, make sure that the Geo is not invisible.

Normally images that want to be texture mapped with openGL should have dimensions that are a power of 2 in both height and width. Now [pix_texture] will make this totally transparent to you (so normally you don't have to care about the size of the image). However with non-power-of-2 images pix_coordinate might not behave as expected, because these images need absolute texture-coordinates rather than normalized ones (as are used with power-of-2 images): so if the texture-coordinates are set to "(0,0) (1,0) (1,1) (0,1)" you might see only the first pixel of the image (which might be black).

Also, make sure that GEM can find your image (ie, that the path name is correct).

----
4.2) My image looks strange. What is going on?

GEM supports gray8, YUV, and RGBA images. If it sees that the number of bits per channel and the number of channels is something that it should be able to handle, it tries to load the raw data. If you have compressed or stored the pixel data in some "strange" format, then GEM will probably not read the information correctly.
Also, if it is an RGBA image, then make sure that the alpha channel is something useful (this only matters if you are using the alpha channel, like in the alpha object or pix_mask).

----
4.3) Why does GEM say that it can't handle a gray image?

This error message occurs whenever a pix object receives a gray8 image and the implementor hasn't provided a way to deal with that format of image. (Implementors often only provide functions for GEM's native color-format RGBA. Any other color-format (like BGR) will try to call the function for gray8 images, which might not be supported.) If you do not want to change the image format with some extern image-programm (like Photoshop or the Gimp) you might want to try pix_rgba or harass whoever made the object to add the functionality.

----
4.4) What image formats can GEM handle?

GEM can read in TIFF, JPEG, and SGI images. These can be in any color format. Gray scale images are loaded in as gray scale (ie, one byte per pixel). Everything else is loaded in or converted to an RGBA image (ie, four bytes per pixel). If there is an alpha channel, then it will be respected. Otherwise, the alpha channel will be set to fully opaque (alpha == 255).

GEM can write TIFF and JPEG images. TIFF-images will be full RGBA-images, wheras JPEG-files only support (compressed) RGB.

----
4.5) What movie formats can GEM handle?

The movie formats GEM can handle (still) depend on the platform you are using.

On Windoze you can read all AVI-files you have codecs for

On linux the readable formats depend on the libraries you had installed when you compiled GEM. Currently there is (optional) support for AVI, quicktime (*.MOV) and MPEG (*.MPG) files. Not all quicktime-formats are supported. This is unfortunate but is due to linux restrictions. I highly recommend that you install the mpeg3-library from Heroine because it is much more stable than mpeg1 (which comes with many linux-distributions). If you have compiled in support for libavifile, you will be able to open Micro$oft-AVI-files. If you have installed the proper codecs (libavifile supports a mechanism for loading codecs from windows-DLLs) you should be able to open almost any format. If you have serious problems, mail them to me. (Be ready to upload the movie-file that won't work)

----
4.6) Why is pix_draw so slow?

pix_draw is almost never hardware accelerated on PCs graphics accelerator. This means that it runs extremely slowly. Always use pix_texture, even if you are just displaying an image.


WORKING WITH PD


5.1) Why do I get clicks in the audio?

If you are getting a constant stream of clicks in your audio, then it is probably because you are trying to do graphics and audio in the same process. Rendering a graphics frame usually takes longer than the size of the audio buffer, which is why you get clicks (the clicks are usually at 20Hz...the typical frame rate).
One way around this is to use two computers, one for graphics and one for audio. If you have enough processing power (or dual processors), then you can run two versions of Pd, one for graphics and one for audio. Just use netsend and netreceive to have the two versions of Pd talk to each other.

----
5.2) How do I get audio data to GEM?

One simple way to get raw audio values right now is to use snapshot~. Just set up a metro which bangs snapshot~ and use the floating point value. If you want "musical" information, then use objects such as env~. You might also have a look at the pix_sig2pix~ which interprets audio-data as pixels and its counterpart pix_pix2sig~

----
5.3) Why can't GEM find an image/model file?

This means that GEM can't locate the file. If you use an absolute path (with / for instance), then GEM will look there. Otherwise, GEM will look in the directory of where the patch is. Then pd/GEM will search the paths you specified at startup with the -path flag.

Check the following:

1) Does the file exist?
2) Did you make a typo in the filename?
3) Is the file in the search-path ?

----
5.4) How can I optimize my patches?

One of the biggest performance hits is having UI elements in your patch which have to be updated. The biggest performance hog is the number box. While the number box is great for debugging, make sure that they are all gone from your "release" patch. If you run a performance meter, you will see that whenever Tcl/Tk has to update the user interface, it sucks the entire processor. Another examples of this is when you move a lot of objects at once, everything jerks and slides across the screen. There are probably ways to improve this...
Another problem is doing unneccessary calculations. When you are throwing lots of numbers around, especially packing/unpacking, doing vector math, etc., they add up. If the calculations are going unused (for instance, that part of the patch is turned off), then do not trigger the math objects. Use spigot or gate and block the events early. This is especially important with objects that send a lot of numbers, like ~ objects or line/tripleLine.


WRITING NEW GEM OBJECTS


6.1) How do I write a new GEM object?

For the time being, you have to look at the code. It is fairly well documented and straight forward (if you know C++ and OOP). Start with an object which is similar to what you want and derive a new class. The biggest issue right now is how to load in GEM as a DSO/DLL. For SGIs, you will need to setenv LD_LIBRARY_PATH. On NT, you will need to have your path include the directory with GEM.

----
6.2) What are the default OpenGL states?

GemMan (and by association, gemwin) disables alpha testing, alpha blending, culling, and lighting. Lighting defaults to two sided, with GL_COLOR_MATERIAL enabled. The viewport is set to

float xDivy = (float)m_width / (float)m_height;
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
glFrustum(-xDivy, xDivy, -1.0, 1.0, 1.0, 20.0);
gluLookAt(0.0, 0.0, 4.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0);
glMatrixMode(GL_MODELVIEW);
glViewport(0, 0, m_width, m_height);

which gives a range of about -4 to 4 in X and Y at the origin. This is a small range, but changing it now would break a lot of patches.

The specific functions to look at are:

GemMan::windowInit()
GemMan::resetValues()
gemhead::renderGL()


OBJECT SPECIFIC


7.1) Why doesn't <object> exist on <platform>?

Usually, this is because I don't have the resources to get the object running on that platform. If an object that you want doesn't exist on your platform, then ask for it! However, if it is tied to hardware, then it is much less likely that I will be able to do anything about it (unless someone donates the hardware to me...)

----
7.2) Why doesn't gemtablet work?

gemtablet only works on WinNT. I don't have drivers for IRIX or Linux (also, see question 7.4)
If GEM can find the tablet, then it will print a message at window creation time. If you don't see a message, then GEM doesn't think that you have a tablet.
The tablet is mapped to the size of the GEM graphics window.

---
7.3) I don't want GEM to take over my tablet. How do I stop it?

Set the environment variable

GEM_NO_TABLET = 1

----
7.4) Why doesn't gemmouse work in IRIX?

Basically, I don't have physical access to an SGI machine. This makes it hard to do some of the OS specific work. It should be straightforward to do the event handling, so if someone gets it working, I would love to include it (and give you credit). All you have to do is call the correct event functions from GemEvent.h and everything should just start to work (ie, gemmouse doesn't have any OS specific code in it).

----
7.5) Why doesn't gemorb work?

You need to make sure that your SpaceOrb is hooked up correctly. I am using a library which isn't supported by SpaceTec so there can be problems, although I have not had any.
<RANT> When will companies wake up and actually provide drivers and support for their products under WinNT? </RANT>

----
7.6) What is wrong with pix_video in WinNT?

I haven't completely figured out how to get access to the video stream in WinNT. I'm using Video for Windows with a Connectix QuickCam, as well as an Intel Video Capture Card, and it seems to assume that you are only writing to a file or previewing into a window. Windows tries to take over the system and doesn't really provide any stable hooks (unlike IRIX). If anyone knows how to deal with this, please let me know.

[return]

11. List of GEM objects

an incomplete list of Gem objects
List of GEM objects

List of GEM objects

Controls
Manipulators
Geos
Particles
Nongeos
Pixes
TV
MarkEx


Controls
gemhead - the start of rendering chain
gemwin - the window manager
gemmouse - outputs the mouse position and buttons in the GEM window
gemkeyboard - outputs the keycode of a key pressed when you are in the GEM window (there might be different keycodes in Windows/Linux)
gemkeyname - outputs a symbolic description of a key pressed when you are in the GEM window (there might be different symbols in Windows/Linux)
gemorb - outputs the position, rotation, and buttons for a Space Orb
gemtablet - outputs the pen position, pressure, and buttons in the GEM window



Manipulators
accumrotate - accumulate a rotation
alpha - enable/disable alpha blending
ambient - set the ambient color with a vector
ambientRGB - set the ambient color with 3 discrete values
camera -
color - set the color with a vector
colorRGB - set the color with 3 discrete values
depth - enable/disable depth testing
diffuse - set the diffuse color with a vector
diffuseRGB - set the diffuse color with 3 discrete values
emission - set the emissive color with a vector
emissionRGB - set the emissive color with 3 discrete values
linear_path - generate a path from an array of points
ortho - change the view to orthogonal, with the viewport the size of the window
polygon_smooth - turn on anti-aliasing for the objects below
rotate - rotate with an angle and vector
rotateXYZ - rotate with 3 discrete values
scale - scale with a vector
scaleXYZ - scale with 3 discrete values
separator - push the OpenGL state for the rest of the chain and pop when done
shininess - set the shininess of an object
specular - set the specular color with a vector
specularRGB - set the specular color with 3 discrete values
spline_path - generate a spline from an array of knots
translate - translate with a vector
translateXYZ - translate with 3 discrete values

Geos
circle - render a circle
colorSquare - render a colored square (evtl. with color gradients)
cone - render a cone
cube - render a cube
cuboid - render a box
curve - render a Bezier curve
curve3d - render a surface
cylinder - render a cylinder
disk - render a disk
imageVert - make pixel colors to a height field map
model - render an Alias|Wavefront model
multimodel - render a series of Alias|Wavefront models, render by number
newWave - render a wave (that is evolving over time)
polygon - render a polygon
primTri - a triangle primitive
rectangle - render a rectangle
ripple - a rectangle with distorted (over time) texture-coordinates
rubber - a grid where you can move one of the grid-points
slideSquare - render a number of sliding squares
sphere - render a sphere
square - render a square
teapot - render a teapot
text2d - render 2-D text (a bitmap)
text3d - render 3-D text (polygonal)
textextruded - render an extruded 3D-text
textoutline - render outlined text (polygonal)
triangle - render a triangle

Particles
part_head - The start of a particle group
part_color - Set the range of colors for the new particles
part_damp - set the damping for particles
part_draw - Apply the actions and render the particles.  Accepts a message "draw line" or "draw point" to change the drawing style.
part_follow - Particles will follow each other like a snake
part_gravity - Have the particles accelerate in a direction
part_info - get the information (position, color, size,...) of each particle
part_killold - Remove particles past a certain age
part_killslow - Remove particles below a certain speed
part_orbitpoint - Orbit the particles around a specified point
part_render - render the remaining gem-tree as particles.
part_size - Set the size of new particles
part_source - Generate particles
part_targetcolor - Change color of the particles toward the specified color
part_targetsize - Change size of the particles toward the specified size
part_velocity - Set the velocity domain (distribution like CONE and the appropriate arguments)
part_vertex - emit a single particle

Nongeos
light - make a point light
world_light - make a light at infinity

Pixes
pix_2grey - convert rgb pixels to grey (still an RGBA image)
pix_a_2grey - convert rgb pixels to grey based on alpha channel
pix_add - add two pixes together
pix_aging - super8-like aging effect
pix_alpha - set the alpha value of a pix
pix_background - let through only pixels that differ from a static "background" image
pix_backlight - a backlight photo effect
pix_biquad - 2p2z-filter for subsequent images
pix_bitmask - apply a bitmask to a pix
pix_blob - get center of gravity
pix_buf - buffer a pix
pix_buffer - storage room for pixes (like [table] for floats)
pix_buffer_read/pix_buffer_write - put/get pixes into/from a pix_buffer
pix_chroma_key - color keying (like "blue-box")
pix_coloralpha - set the alpha-channel of a pix as a mean-value of the color-components
pix_colormatrix - recombine the RGBA-channels with matrix-operation
pix_color - set the color of a pix (leaving alpha alone)
pix_colorreduce - reduce the number of colors (statistically)
pix_composite - composite two pixes together
pix_convolve - convolve a pix with a kernal
pix_coordinate - set the texture coordinates
pix_crop - get a sub-image of a pix
pix_curve - apply color-curves onto a pix
pix_data - get pixel data information
pix_delay - frame-wise delay
pix_diff - get absolute difference of two pixes
pix_dot - rasterize a pix with big dots
pix_draw - draw a pix
pix_dump - dump the pixel-data as a long list of floats
pix_duotone - reduce the number of colors by thresholding
pix_film - use a movie file as a pix source for image-processing
pix_flip - flip the pixels of a pix
pix_gain - apply a gain to a pix
pix_grey - convert any pix into greyscale colorspace
pix_halftone - rasterize a pix like it was printed in a newspaper
pix_histo - get the histogram of a pix
pix_hsv2rgb - transform a pix from HSV-colorspace into RGB-colorspace
pix_image - load in an image file
pix_imageInPlace - load a series of image files directly into texture-buffer, display by number
pix_info - get information about the pix (like dimension, colorspace,...)
pix_invert - invert a pix
pix_kaleidoscope - as if you were looking at the pix through a kaleidoscope
pix_levels - level adjustment
pix_lumaoffset - y-offset pixels depending on their luminance
pix_mask - mask a pix based on another pix
pix_metaimage - recompose an image out of smaller versions of itself
pix_mix - mix to pixes together
pix_motionblur - motionblur an image
pix_movie - use a movie file as a pix source and load it immediately into the texture-buffer
pix_movement - set the alpha-channel with respect to the change between two frames
pix_multiply - multiply two pixes
pix_multiimage - load in a series of image files, display by number
pix_normalize - normalize a pix
pix_offset - add an offset to a pix (wrapping instead of clipping)
pix_pix2sig~ - interpret a pix as 4 (RGBA) audio-signals
pix_posterize - posterization photo effect
pix_puzzle - shuffle an image
pix_rds - generate a Random Dot Stereogram out of the image (aka: Magic Eye (tm))
pix_rectangle - generate a rectangle in a pix buffer
pix_refraction - break up an image into coloured "glass-bricks"
pix_resize - resize a pix to next power of 2
pix_rgb2hsv - transform a pix from RGB-colorspace into HSV-colorspace
pix_rgba - transform a pix of any format into RGBA
pix_roll - (sc)roll through an image (wrapping)
pix_rtx - swap time-axis and x-axis
pix_scanline - take every nth line of the original image
pix_set - set the pixel-data with a long list of floats
pix_sig2pix~ - interpret 4 audio-signals as (RGBA) image-data
pix_snap - capture the render window into a pix
pix_snap2tex - capture the render window directly as a texture
pix_subtract - subtract two pixes
pix_tIIR - time-base Infinite-Impulse-Response filter (for motion-bluring,...) with settable number of poles/zeros
pix_takealpha - take the alpha channel of one pix and put it into another pix
pix_texture - use a pix as a texture map
pix_threshold - apply a threshold to a pix
pix_video - use a video camera as a pix source
pix_write - capture the render window to disk
pix_zoom - zoom into a pix (using OpenGL)

openGL there are more than 250 objects that form a complete wrapper around the openGL set of functions (as defined in the openGL-1.2 standard).
each openGL-function is prefixed with "GEM", eg: [GEMglVertex3f] is wrapped around glVertex3f.

MarkEx
alternate - alternate between two outlets
average - average a sequence of numbers
change - only output on change
counter - count bangs
invert - non-zero numbers to zero, zero to 1
multiselect/multisel - a select object which accepts a list in the right inlet
oneshot - send a bang, then block until reset
randomF / randF - floating point random numbers
strcat - string concatentation
tripleLine - do a line with three numbers
tripleRand - random with three numbers
vector+ / v+ - add a scalar to a vector
vector- / v- - subtract a scalar from a vector
vector* / v* - multiply a vector by a scalar
vector/ / v/ - divide a vector by a scalar
vectorpack / vpack - attach a scalar to the end of a vector
rgb2hsv - convert a list of three floats from RGB to an HSV value
hsv2rgb - convert a list of three floats from HSV to an RGB value
abs~ - absolute value of a signal
reson~ - resonant filter

[return]
 
 

12. Glossary

What does ... mean?
Glossary/Index

Glossary

Alpha - The amount of opacity.  An alpha equal to 1.0 means completely opaque.  An alpha equal to 0.0 means completely transparent.

Controls - GEM objects which access the low levels of GEM, such as window managers.

Geos - GEM objects which have a shape of some kind, such as a cube.

Manips - GEM objects which manipulate the geos.

MarkEx - A collection of objects which help with data manipulation, especially for usage in GEM.

Nongeos - GEM objects which do not have an explicit shape, yet affect the rendering in some way.

OpenGL - A graphics API which exists on many different platforms.
Gem can also be used as a wrapper for openGL, allowing to program openGL without having to compile

Particles - GEM objects which involve the particle system.

Pd - A visual programming language for audio processing.  This is the host application for GEM.

Pixes - Image processing objects in GEM

Texture mapping - Applying an image to a geometric object.

[return]
 


Powered by IEM Powered by Plone Section 508 WCAG Valid XHTML Valid CSS Usable in any browser