Here are my notes from the Procedural Textures Croquet/OpenGL/GLSL project.
 I had this in a personal wiki - it loses something as plain text.  I
haven't come across the code anywhere, and I fear it was lost a couple of
years ago when I had a hard-drive failure.  Time to redo it in WebGL anyway!

Cheers,
Dave


ProceduralTextures
DaveFaught, 10 August 2009 (created 3 May 2006)
tags:
Croquet
This is an open community project to procedurally build some cool textures
for use in the Croquet environment. Plans for this project are still fluid.
There are other active projects in the Croquet rendering area that may at
some point present code conflicts with this project. We will attempt to
make use of the best alternatives available.

The inspiration for the actual procedural textures is largely due to Jerzy
Karczmarczuk's Clastic tutorial paper, and so this project is intended to
be a library of general functions that can be combined in various ways,
rather than a finished set of textures.

The Plan
The current Croquet SDK code base will be used for the OpenGL support and
Tweak 2D UI support. The framework will put the project strictly in the
local processing arena, outside of Croquet's replication and collaboration
model, at least for now.

Current Status
The current code for this project is in the PublicContribs repository at:
http://hedgehog.software.umn.edu:8888/PublicContribs in the Tweak-OpenGL
and ((OpenGL-OpenGL packages. It is necessary to manually do "OGLExtManager
initialize" after loading the OpenGL-OpenGL package. Andreas also has an
updated OpenGL package in the Hedgehog repository that needs to be merged
in.)) <- I believe that this can be all replaced by UMN's new OpenGL
packages in the Jabberwocky repository. This also depends on a fix for
Mantis #6553, which can temporarily be a pretty trivial do-it-yourself kind
of thing. Just change VectorColor#asVector4 so that it returns a Vector4
instance instead of a Vector3.

The current Tweak-OpenGL package contains:
Two, yes count them, two(2) different classes for a basic OpenGL display in
Tweak. One is a subclass of CPlayer and is based on a hack of Bert
Freudenberg's Croquet Jasmine-based COpenGLPlayer, and the other is a
subclass of CPanel and was donated by Andreas Raab. The CPanel version can
be launched directly from Morphic, and the CPlayer version needs to be
opened in an already-open Tweak environment. There are a few other
differences.
Two examples of the translated NeHe Tutorial #5 (the first one that has 3D
objects), one for each of the basic Tweak OpenGL classes.
The beginnings of a GPUinfo query class to display the level of OpenGL
support that the video card and drivers supply. Currently this is primarily
used to verify GLSL and FramebufferObject support as required by the
Wrinkle application.
Two versions of the Wrinkle application, the main product of this project.
The first version, Wrinkle1, uses a single fragment(pixel) shader program
at a time and displays the resulting texture dynamically. The second
version, Wrinkle2, is more advanced through combining two dynamic
procedural textures and allows saving a snapshot of the result to a .png or
.jpg file by use of an OpenGL FramebufferObject.

There are short video clips at http://www.youtube.com/watch?v=Eo22Gdi_b9Mand
http://www.youtube.com/watch?v=qBHbMxplM5c

It has taken about 2 years to get the simple method working (calendar time,
not working time :-) A similar page to this one is on the Consortium wiki
at http://croquetconsortium.org/index.php/Procedural_Texturing

Procedural Code
The procedural code is targetted to run on the Graphics Processor Unit
(GPU) of the 3D graphics video card, using the OpenGL Shading Language
(GLSL or glslang). Tobias Germer supplied the GLSL support for Croquet
Jasmine, Joshua Gargus supplied the NullTerminatedStringArray class, and
Dave Faught supplied the FrameBufferObject support.

The former simple method, used by Wrinkle1, is:
- Compile the shaders that will generate the texture.
- Enable the shaders.
- Render an OpenGL geometry with proper texture coordinates to dynamically
display the generated texture.
- Interactively pass parameters to the running shaders from the Tweak UI.

The current method, used by Wrinkle2, is:
- Create an FBO with an attached texture.
- Compile the shaders that will generate the texture.
- Every render cycle:
— Enable the shaders.
— Interactively pass parameters to the running shaders from the Tweak UI.
— Render the shaders into the FBO with texture coordinates.
— Disable the shaders and FBO.
— Render an OpenGL geometry to the display with texture coordinates mapped
to the FBO's texture in the normal manner.
— Allow a snapshot of the FBO texture to be saved to a bitmap file, such as
png. To get the texture back from the GPU. This step uses glReadPixels(),
which is syncronous.

The input x and y values to the procedure are supplied by the OpenGL
geometry texture coordinates, and so are each in the interval 0.0 to 1.0.
This is different than many other implementations where the resulting
bitmap pixel coordinates, for example between 0 and 127, are used. The
results of the functions are assumed to be in the interval -1.0 to 1.0, and
may be forced to that by using the fract() function. The function results
are then generally mapped to an RGB color space, where each of the
components are in the interval 0.0 to 1.0.

Some of this process is purposely vague in order to promote
experimentation, but there are limitations imposed on the process by the
GPU hardware.

One general procedure might therefore look like this:

/* convert the input x and y from a 0.0 : 1.0 interval to a -1.0 : 1.0
interval */
myX = (x * 2.0) - 1.0;
myY = (y * 2.0) - 1.0;

/* do some interesting function or combination of functions */
myZ = myX * myY;
...

/* ensure that the result in in the -1.0 : 1.0 interval */
myZ = fract(myZ);

/* convert the result to a 0.0 : 1.0 interval */
myZ = (myZ + 1.0) / 2.0;

/* map the result into an RGB color space */
color = mix (cAqua, cGold, myZ);

UI Design
There is some dynamic interaction between the Tweak UI (sliders and
buttons) and the compiled shader programs, although the UI is not expected
to ever get to the level that is achieved by the ArtOfIllusion Texture
Editor. There will have to be a tradeoff between the code restrictions of
smaller GPUs and the desire for procedural function flexibility. The desire
to accomodate smaller GPUs pretty much rules out the AOI TexEd toolkit
approach, so a more fixed stack with sliders for each stage would be
appropriate, and by definition more limited. How much of a stack can be
accommodated or is desirable?

There are a few different classes of functions that may feed this, for
example ones that take a single parameter z=f(x) and those that take two
parameters z=f(x,y). The fact that they (all?) produce a single output
means that they cannot simply be linked in series.

Tweak Considerations (orig. from the mailing list)
Yes, I too have had problems when switching back out of a Tweak Project to
Morphic, and I have for the moment elected not to deep dive into assisting
with the debugging in this area. The result is that I generally don't use a
Tweak Project at all.

Note that there is an environmental difference in opening a "Tweak Project"
and opening a "Tweak Project Window" from the open menu. I have not had the
same problems with a Tweak Project Window, and in fact it generally seems
to cooperate with Morphic more cleanly. It is
actually much easier to develop and test code using a Tweak Project Window
because of this cleaner interaction with Morphic.

So the way I go about developing with the OpenGL player in Tweak is to make
my code changes with the Morphic class browser and other nice Morphic code
tools, and test out the code with a Tweak Project Window (which can remain
open or minimized while going back to the Morphic browser for more
changes). When I think the code is relatively clean, then I save the image.

Andreas Raab has provided an alternative OpenGL Tweak panel which can be
launched directly from Squeak. This panel may be a better alternative for
some projects.

Background
Here are a few resources for general procedural texturing:
http://users.info.unicaen.fr/~karczma/Work/Clastic_distr/clastic.html
http://mrl.nyu.edu/~perlin/
http://graphics.cs.uiuc.edu/~jch/papers/
http://conal.net/Pan/
"Texturing and Modeling, A Procedural Approach" by Ebert, et al. is an
excellent book on this subject.

Also, check out the procedural texture editor in the Open Source 3D modeler
Art Of Illusion. You can use the VRML export feature of AOI and select
"Create Image Files for Textures" to export 2D procedural textures that can
be imported to Croquet with the VRMLImporter. Take a look at the AOI
tutorial, Procedural Texture and Material Editor, Chapter 9, "Example
Textures" for an idea what the GUI looks like. Only AOI's textures can be
exported to VRML, and end up being both textures and materials in Croquet.
In AOI, materials are more like OpenGL shader programs, and cannot be
expressed in VRML.
http://www.artofillusion.org
http://www.artofillusion.org/docs/texedtut/chapter9

Besides Clastic and the ArtOfIllusion Texture Editor, a couple of other
free tools for creating procedural textures are .werkkzeug 3 TE, and mental
mill Artist Edition that comes with the NVidia's FX Composer 2 beta 4.
http://www.werkkzeug.com/
http://developer.nvidia.com/object/fx_composer_home.html
http://www.mentalimages.com/2_4_mentalmill/index.html

Michael Kleiber did some nice extensions to Croquet Jasmine to use OGL
shader programs for his thesis.
http://www.kleibi.com/index?page=univ

Other shader languages that might be considered at some point are
Stanford's BrookGPU or UW's Sh.
http://www.cs.lth.se/home/Calle_Lejdfors/pygpu/
http://www.graphics.stanford.edu/~hanrahan/talks/gp2/walk001.html
http://graphics.stanford.edu/projects/brookgpu/
http://merrimac.stanford.edu/
http://merrimac.stanford.edu/brook/
http://libsh.org/
http://www.shadertech.com
http://developer.3dlabs.com
http://conal.net/Vertigo/

In order to have the GPU run the code for doing procedural textures, a
higher level of GPU functionality is required than that needed to run
normal Croquet. A recommended minimum GPU chipset is the NVidia GeforceFX
series, or the ATI Radeon 9500.
http://developer.nvidia.com/object/General_FAQ.html#s2
http://www.ati.com/developer/atiopengl.pdf

More recent NVidia (Forceware 60 or newer) and ATI (Catalyst 4.5 or newer)
drivers include the GLSL compiler within the drivers for Windows and Linux.
Apple takes a much larger role in writing drivers for OS X (the Windows and
Linux drivers are substantially the same code base, at least for NVIDIA),
but finally did finish their glslang support.
http://www.opengl.org/documentation/oglsl.html
http://www.3dshaders.com
http://www.lighthouse3d.com/opengl/glsl/

The new OpenGL FrameBuffer object is a new extension that is
platform-independent and is more flexible than PBuffers, but just became
generally available in the NVidia Forceware 75 drivers released in June
2005. Not sure exactly which release of ATI drivers contains FrameBuffer
object support, but there is support in the October 2005 ATI Radeon SDK.
http://www.opengl.org/documentation/extensions/EXT_framebuffer_object.txt
http://download.nvidia.com/developer/presentations/2005/GDC/OpenGL_Day/OpenGL_FrameBuffer_Object.pdf
http://www.nvnews.net/vbulletin/showthread.php?t=49157
http://www.ati.com/developer/

The class comment of OGLExtManager explains what needs to be done to add
new OpenGL extensions. OGLExtManager does not support platform-specific
extensions.

Some related code
Here is some simple Croquet/Squeak code to generate a one-shot texture and
display it. Based on example 1 in Clastic (see above). Does not require any
special graphics support beyond running Squeak. This gives an idea of where
we're headed, but this is not the method we will use.

Define the class AFirstTry as a subclass of Form, and then enter this
method in AFirstTry:

plot1
"plot a function"
" (AFirstTry extent: 128 at 128 depth: 32) plot1 asMorph openInWorld "
| xmin xmax xincr ymin ymax yincr x y z scaled |
xmin := -1.
xmax := 1.
xincr := (xmax - xmin) / (self width).
ymin := -1.
ymax := 1.
yincr := (ymax - ymin) / (self height).
scaled := 0.1.

1 to: self width do: [ :xpixel |
x := (xmin + (xincr * xpixel)) / scaled.
1 to: self height do: [ :ypixel |
y := (ymin + (yincr * ypixel)) / scaled.

" the main mapping function "
z := ((x sin + y) sin + x) sin - (((y sin + x) sin + y) sin).
self colorAt: xpixel at ypixel put: (Color r: z g: 0 b: (1-z)).

].
].

"xfun1 (V2 x y) = sin(x+sin(y+sin x)) - sin(y+sin(x+sin y))
xfun2 (V2 x y) = floor(sin(x+sin(y+sin x))) -
floor(sin(y+sin(x+sin y)))
"

To execute the code, highlight the code in the comment and DoIt.

NVidia GPU Shader Capabilities
The table uses DirectX nomenclature (i.e., Vertex Shader and Pixel Shader
version numbers). For GeForce FX and newer GPUs that are capable of flow
control, the static instruction count represents the number of instructions
in a program as it is compiled. The dynamic instruction count represents
the number of instructions actually executed. In practice, the dynamic
count can be vastly higher than the static count due to looping and
subroutine calls.

GPU DX Vertex Shading Capability Pixel Shading Capability
GeForce FX 2.0+ 256 static instructions 512 static instructions
65,535 dynamic instructions 512 dynamic instructions
GeForce 6 Series 3.0 512 static instructions 2,048 static instructions
65,535 dynamic instructions 65,535 dynamic instructions

A personal note from daf:
Using the GPU to do the processing for procedural textures through vertex
and pixel/fragment shaders is a current and exciting area of
experimentation for many developers inside and outside of the Croquet
community. While I am very interested in exploring this area, I am also a
bit disappointed that I have to code the actual shader programs in a
variant of C, instead of Smalltalk. Maybe someday this can be remedied by
an industrious Croquet developer as an embedded language extension. Look at
PyGPU, Vertigo or UW's Sh in the links above for ideas.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20130223/95fd517d/attachment.htm