Footprints in the snow

Develop a system that automatically captures a characters footfalls on the ground and leaves displaced imprints in the snow (or whatever soggy medium in which he might find himself). This covers: Depthmaps Rendering reference passes External image processing Displacement mapping

Project files

The idea is we render a prepass which we can process and use as a displacement map on our ground plane, to displace footprints, or any arbitrary penetration into our surface. The prepass we are going to set up is a depthmap pass. The depthmap pass (same as a shadow map) encodes the distance of each visible surface point -from the point of view of a camera. In the movie below, where the objects go red they are penetrating the ground plane and need to displace it downward. This distance is what we want to capture with our depthmap pass, then turn into displacement.

Scene Setup To start, we are going to set up a test scene: 1. Create a ground plane at y= 0 (although the Y value does not need to be 0) and a number of objects moving in and over this plane. 2. Create a second plane just below this one, which we shall use as our collision plane (if you use rigid bodies, for instance, this will be the passive rigid body). Make sure to turn off all the render stats. 3. Create a Maya set of all the impacting objects you want to capture in your reference depth pass. 4. Create an orthographic camera and call it zDepthCam, place it below the ground plane looking up, then adjust the orthographic width to match the size of your ground plane. 5. In render settings->passes, create a new shadow pass.

6.

7.

In the shadow pass settings, we already have z declared, but we want to overwrite this one with a new one. So add a new output of type custom, and enter z into the field. This will create a new shadow output, in addition to the default.

8. Set the filetype to tiff32, and create a sensible pass name format (here, z-depth). 9. Under pass settings for our shadowPass, right click in the camera field. Select your camera, here called zdepthCam.

10. Also right-click in the sets field and select your impact object set. 11. Finally, in the pass settings, set the depth filter to min. This records the minimum depth between the camera and the object being shaded; exactly what we want in the map. Now we are ready to render out our shadow maps, for post-processing.

12. In the common tab, set your render resolution to 2x2 pixels and your frame padding to 3. We have no interest in the final render here, make it quick as possible. We only want to render things through for the shadow maps. 13. Click Render. This gives us a cache of floating point (32bit) TIFFs on our drive. So to talk about the rest of the tools, we need to accomplish our snowy mission. Next, the magic displacement shader. Shader Writing Here is where we dig into the advanced topics that make this whole thing work. Conceptually, our depth maps contain world space positions of our scene from the point of view of the shadow camera. Also, we have our current shading point, P, also living somewhere in the world, with respect to a coordinate system called current. What we want to do is get our current point P into the same coordinate system as the points in our depth maps (ie. zDepthCam space).

The camera that rendered the pass. The mechanism we are going to use is the shading language function textureinfo(). We get the camera viewing matrix and the camera projection matrix. See the RenderMan documentation for more information. uniform matrix camViewSpace, camProjectionSpace; textureinfo(refMapForCamMatrix, "viewingmatrix", camViewSpace); textureinfo(refMapForCamMatrix, "projectionmatrix", camProjectionSpace); Shadow maps in the RenderMan texture format (tex) and TIFF format store this information in the header of the image format. point PshadCam = transform(camViewSpace, P); float Pzdepth = PshadCam[2]; This bit of code simply converts the shading point on our ground plane into our depth cam coordinate system. But as our camera is directly below our scene pointing straight ahead, the useful component for extracting our displacement value is the z component (or PshadCam[2]) of our transformed point. Moving on: point PshadNdc = transform(camProjectionSpace, P); We ALSO transform our point P into zDepthCam NDC space (i.e. the flat, projected image space used to write the depth maps in the first place). This is what we need to look up values from our shadow map! float ss = (1.0+xcomp(PshadNdc)) * 0.5; float tt = (1.0-ycomp(PshadNdc)) * 0.5; This simply remaps our point in NDC from -1,1 to 0,1 (ie. well-formed texture coordinates) for texture lookup. float displacementDist = 0; if( (ss<0 || ss>1) || (tt < 0) || (tt > 1)){ displacementDist = 0; } Declare our displacement distance float variable. Then test our ss, tt texture coordinates and make sure they lie inside 0,1 in both directions; we are not going to displace anything because our current shading point is outside the depth maps generated. else { float zDepthObject = 0; zDepthObject = texture(depthmap, ss, tt, "samples", samples); ....

If we HAVE good ss, tt coordinates, then look up the depth from our depth map. Remember: this value represents the z distance from our depth camera to the objects in the depth map, not the ground plane. We also have the z distance from our depth cam to the ground plane, which we obtained earlier. That is all we need to get our displacement value, just subtract the two. .... if (zDepthObject > maxDistance) zDepthObject = Pzdepth; displacementDist = Pzdepth - zDepthObject; if (displacementDist < 0) displacementDist = 0; } We do the subtraction in this bit of code. We also need to do a couple of extra tests to make sure everything is ok. We first test if the z-depth we lookup in the map is BIG (ie. it is at the far clipping plane), then set our z-depth to that of our ground plane. The other conditional makes sure our object is below the ground plane, otherwise set the displacement to zero (note: this gives very sharp edges which may not anti-alias well). normal Nn = normalize(N); P = P - displacementDist*Nn*dispMultiplier; N = calculatenormal(P); Finally, the business of actually pushing points. Here we move along the normal direction. But equally, we could displace explicitly along the world y axis with code something like this: vector Vworld = vector "world" (0,-1,0); vector Vcurrent = normalize(vtransform("world", "current", Vworld)); P = P + displacementDist*Vcurrent*dispMultiplier; N = calculatenormal(P);

Accumulating the Zdepth We need to do some simple image-processing to accumulate the depth map and give frame to frame persistence to the footprints. The script is quite simple, and has nothing specifically to do with RenderMan. But it is important for executing this effect. It will be explained here in generic pseudo-code. The final python script relies on the PIL module (Python Imaging Library), and is attached. depthMaps[] = getAllDepthMaps(path); Image map1 = depthMaps[0].open(); int currentFrame = 2; for currentMap in depthMaps: Image map2 = currentMap.open(); Image comp = Image.lighten(map1, map2); //which pixel is lighter comp.save(outname+currentFrame); // save composite image system('txmake -float -format tiff';) // conform our TIFF to RenderMan format map1 = comp; //roll the composite forward, a sum of all previous frames currentFrame ++; Note 1: Due to the extensible nature of the TIFF format, many image importers have a difficult time opening files with custom tags - like our RenderMan depth maps (remember the camera matrices included in the tags). To work around this, we use a PhotoshopCS2 batch process to open and resave all depth maps; this removes the tags that create loading problems in PIL, but took with it the camera matrix information. You will see the following: system('txmake -float -format tiff';) Note 2: We make a system call to the txmake utility (included in the Studio distribution) to conform our output TIFF to Pixars texture format (even though we still save it as a tiff file). This allows us to directly reference the texture files from the shader, otherwise RenderMan will complain that file is not a RenderMan texture; then throw an error.

Finishing the Setup and Rendering 1. Compile our shader. 2. Create a RenderMan displacement node. 3. Import shader and point paths to your files: depth maps should be of the form: /myDepthMaps/zComp.$F1.tif reference map is one of your original z-depth maps that still has the camera matricies in the TIFF tags. Should be a /myDepthMaps/zDepth.001.tif 4. Add displacement bounds to your RenderMan node. They should be bound to the maximum depth in the surface your objects penetrate to. 5. Add any fancy lighting you want. Remember if you are ray tracing, you will need to add the trace displacement attribute to the groundplane.