Combining 2D and 3D

This tutorial demonstrates advanced techniques for adding 3D objects into a real photographed scene. It shows various tricks for creating a seamless interaction between 2D and 3D worlds and makes frequent use of VSL. It is therefore intended for advanced users who understand the basics of programming languages.

Level: Advanced

Sample files: 'tutorprojects/material/lake/boat', 'tutorprojects/material/lake/Lake'

We will use the image lake.jpg found in the textures folder of the program installation. Go to the Images tab of the Select window, select New from the popup menu and double click the new image object to open the property window. Use the browse button of the Image file gadget ("..." button) to pick the lake.jpg file. The property window displays the image shown at the right.

Our main objective of this exercise is to add a boat sailing across the lake. We want to see the reflection of the boat on the water (among other things) so that the animation looks convincing. We do not have exact information of the position of the camera or the dimensions of the lake. Therefore, we have to estimate their coordinates roughly to make things work well.

A lake scene

We start by setting up the view camera. Close the image property window. Reset the view window to the default front view (by clicking N button of view controls). Then open the view property window from "windows/view property window". Select the first tab Camera, which contains the information of view's internal camera. Select Perspective projection from it if not already selected.

We may think that the front bank of the lake is at the world origin, and the camera is approximately 10 meters behind it. The camera is held at a typical eye height - say, 1.5 meters above the ground. Therefore, enter the values '0 1,5 10' to the Position field. The camera is directed quite horizontally, therefore enter the values '0 1,5 0' to the Aimpoint field. The last important setting is Focal length, set 50 mm to that gadget. Now the camera is set and you may close the view property window.

We need to store the defined camera orientation more permanently. Therefore, go to the Creation tab of the toolbar and click Camera tool icon, and select Accept. This creates a camera object that has the same settings as the view camera.

The next step is to create a backdrop plane that shows the photograph in rendering. Make sure that the created camera object is selected in the object hierarchy. The tool control bar automatically displays the camera related tools. Look for Backdrop. Click the button, and new controls are shown. The first, Distance, defines the distance between the camera position and the backdrop plane. Set 100 meters, which is approximately the distance across the (quite small) lake. Leave the next gadget, Material, empty; we have not defined any material for the backdrop. The third gadget New image can be used for defining a texture map for the backdrop. Click the browse button beside it ("...") and select the image lake.jpg from the Textures folder. The last Color gadget won't be used, because the color comes from a texture map. Click Accept.

Creating a backdrop

After these steps, a backdrop plane and a texture map to a suitable material becomes inserted into the camera object's sub hierarchy. Render the view and you will see the lake image filling the view. The backdrop plane has such a special property that it automatically expands itself to fill the whole view of its parent camera object.

Next we will add a plane representing the water surface of the lake. The backdrop plane already shows the water, but it is in a wrong place (100 meters away), and is oriented vertically, so it cannot interact with objects close to the camera. Therefore, take a top view. Switch to Parallel mode by disabling View/Camera/Perspective from the view popup menu, for example. Then zoom right out until you see the backdrop plane. (Drag the root directory to the view window will set clipping planes to show all objects.) It may be wise to open the numerical window (View/NumericWindow in the main menu bar) if not already opened. Select the Analytic rectangle from the rectangle tool's shortcut menu in the Creation toolbar section, and draw a large rectangle covering the view. Note that the edge of the rectangle should go behind the backdrop plane (z coordinate over -100 meters).

When the rectangle is ready, close the numeric window. You may rename the rectangle as Lake for easier identification. Then select the camera object and pick the menu View/Camera/Object view (Right click on the view window) from the view's popup. This sets the view window to show the orientation stored in the camera object. Render the view. The created ground plane hides the backdrop's lower half. Optimally, the rectangle surface should become visible about at the waterline behind the lake. If this is not the case (for example, no water is visible at all), select the camera object, open the property window, and go to the Spec tab. This shows the camera specific object attributes. Change the Aimpoint from '0 1,5 0' to '0 1,7 0', which directs the camera slightly upwards, therefore lowering the ground level visible through it.

Water surface inserted

Update possible camera object changes to the view by selecting View/Camera/Object view, and render (Dragging the camera from the Select window to the view will achieve the same result). When the adjustment is suitable (in the image above, some water is slightly visible at the right edge), close the property window.

Now we have a rough model of the water. The next step is to make a material for it. Go to the Materials tab of the Select window. You may get rid of extra materials by selecting Materials/Purge Material Library from the main menu. The only material that is left is called backdrop0, which was created by the backdrop tool. Select it and select Duplicate from the popup menu. Double click the icon of the copy to open the property window. Rename the copied material as lakesurface in the Name field. We do not call it water, because the created material will not have the optical properties of water, but it will include quite a special combination of shading effects.

Enable the Advanced option and maximise the VSL Objects frame of the property window. Drop a Constant VSL object onto the Surface properties shader. The inserted Color=constant(0,0,0) removes diffuse shading from the water. This is necessary because the lake photograph already includes such shading, and we do not want to shade it again in the 3D model. The first version of lakesurface is now ready and you may close the property window. Later, we will improve the material step by step to achieve better and better integration between the 2D photo and inserted 3D objects.

The first version of the material for the water surface

Go to the object hierarchy tab of the Select window and select the rectangle representing the lake. Select the Materials tab of the main toolbar and click the Default map tool icon. Select lakesurface into the displayed Material gadget in the tool controls, and set Mapping type to Camera. Click Accept. If you now render the view, the ground rectangle has disappeared and all you see is the lake photo. Anyway, now there is a real 3D ground plane in the scene and the color information for the water area comes from a sensible distance. Save the project as lake1 to a suitable temporary location using File/Save as menu to backup the work done so far.

Path animated boat

Next we add a small boat to the scene. Therefore, reset the view and select parallel projection. You may model the boat yourself, or load the included simple boat object, which was freeform modeled. Use Paste from file of the Select window's popup menu for this purpose. Remember to disable the replace option if you use File/Load menu for loading the boat, otherwise you loose the already modeled scene. The boat should have sensible dimensions: the length can be about 3-4 meters. You can use the numeric window and the scale tool to adjust the boat size to the right magnitude. Using the front view, move the boat so that its bottom is slightly below the waterline (y coordinate zero; world origin is at that level as well).

Take the top view and zoom right out, to see the whole lake rectangle (alternatively, drag the lake rectangle from the Select window into the view to zoom the object). Move the boat to the top left corner of the lake rectangle. Then select the curve creation tool from the toolbar and draw a cubic curve starting from the boat, going to the right and downwards, then left and down. In other words, in the camera view, the boat will appear from the left near the back shore of the lake, make a turn at the right and then pass the camera at a close distance disappearing to the left, like in the image below. Click Accept to finish the curve tool.

Select the boat and the nurbs curve, in this order. Go to the Lattice mapping section of the toolbar and select the Path tool. The default options produce a directed path, which moves and rotates the boat, which is exactly what we want in this example, so just click Accept.

The path for the boat

Open the AnimationWindow from the main menu bar's View menu, if not already open. Play the animation to see how the boat travels along the path. If camera tracking is on, the view window changes immediately as the animation playback starts to show the camera object's view. You can control this using the view property window's Camera/Tracking section. You can also use the View window's popup menu item Camera/Object view to take the camera view.

After the playback has stopped, edit and fine tune the nurbs curve from the top view if necessary. Then return to the camera view, move the time slider to a position where the boat is relatively close to the camera, and render. The bottom of the boat is partly hidden by the lake photo. This is the first step towards the 2D-3D blending effect we want. Currently, the boat seems to be hanging in the air.

This typical problem can be solved by adding a suitable shadow, or in this case, a reflection of the boat to the water. This task is a bit tricky because only the boat should be reflected. Other reflections are already included in the scene. Therefore, we need to use VSL for customized reflections.

The boat does not appear to float in the water because reflections are missing

Selective reflections

Switch to the Channels section of the Select window. Select New/Float channel from the popup menu. Rename the created channel object as boatchannel. Then make the object hierarchy tab of the Select window visible and double click the boat to select it and to open the property window. Select the Col tab of the property window, and pick the item boatchannel from the Surface properties/Attribute list. Enter the value 1.0 to the field below it. In other words, we mark the boat with the boatchannel property value of 1, other objects, like the backdrop plane, have the default value 0 and will be excluded.

Go to the Materials tab of the Select window, and select the lakesurface material. Drop Constant VSL object to the surface properties shader, use the popup menu to change Output to Surface:Reflection, and change the black constant value (0,0,0) to light gray (0.7 0.7 0.7). Render the image. Note that we must be rendering with at least "Reasonable quality" rendering mode to see reflections. If this is not the case, go to the rendering tab and drag "Reasonable quality" onto the view window and render again.

There is a reflection of the boat on the water, but things are not quite right:

  • The water is too light and reflects the mountains twice

  • The boat reflection contains color from the scene behind the boat, just like the boat was translucent in reflections

Incorrect reflections

Let's continue editing the lakesurface material on the property window. Drop a new Shader object from VSL Objects just under the Material node. Change the Type to Secondary ray. This shader controls traced rays - just what we need. Drop an If object into Secondary ray shader. Change Input 0 from the popup menu to Traced ray:boatchannel . The If object gadget at the bottom of the screen displays three conditions. Change the topmost X field to If inside and enter Lower=0.1, Upper=1.1 to the two numeric fields below it. Leave other conditions unmodified. In other words, we test here if the reflected ray contains nonzero boatchannel information.

Now drop a new Constant object into the If object. (Place it underneath and inside.) As explained above, this If block handles the boat reflection area, because it is executed only if the boatchannel value is between 0.1 and 1, and only the boat object has currently such a value (its boatchannel property was set to 1). The problem area is that the reflection also showed the background scenery. The reason is the original camera projection mapped illumination. Therefore, set output channel of the new constant shader to Surface:Illumination. The value zero is exactly what we want: we reset the illumination mapped from lake.jpg to zero before adding the reflection from traced ray to it.

Another problem was that outside the boat reflection, the lake reflected the scene behind it twice. The first correct reflection was already in the scene photograph. The second incorrect reflection comes from the added reflection property: rays reflect from the lake rectangle and hit the backdrop plane. This effect can be eliminated easily.

Select the If object and then Copy from the popup menu. Select the Secondary ray node from the VSL hierarchy and Paste from the popup menu. Then change the first condition of the new If object to If not inside. In other words, this VSL block will handle those reflections that don't have boatchannel value set.

Select the Constant object inside the If object and change its output to Traced ray:Illumination. This way we reset unnecessary reflections (not showing the boat) to zero. Test render to verify the result. This is also a suitable point to save your work again.

Correct reflections


The next effect to be added is waves around the boat. The main difficulty is that the lake photo already contains reflections of the scene in the background without any waves. Therefore, we have to somehow modify the existing reflections.

First we create a suitable procedural bumpmap for the waves. Go to the Materials tab of the Select window, and create a new VSL material. Open the property window (if not already open), and rename the material as waves. Open the advanced properties. Drop a Shader object onto the Material node, and change the type of created shader to Surface geometry. This shader is suitable for bump mapping purposes. Drop a Variable object to Surface geometry shader, change its name to tmp and type to Vector. Then drop another variable to the same shader, rename it as 'r' and set type to float. Then insert an Operation object to the shader, and select dot product from the long list in the Type gadget. Set Output to r variable using the popup. Both Map coords inputs are correct. This VSL object computes the square of the radial distance from the zero point of the mapping. We will use the radial distance to fade out the waves within a circular area around the boat.

Drop an If object to the shader, and set its Input 0 to float variable r with the popup menu. Change the first condition X from Always to If inside, and define Lower=0, Upper=1. The included code will be executed if the mapping coordinate measured distance (which can be adjusted by scaling the mapping object) will be less than one.

Drop a Constant object into the If object. Set its output to r, value to 1, and change Operator to Subtract from the popup menu. Then drop a new Operation object into the If object. Select multiply from the long Type list, Set Input 0 and Input 1 both to r, and set Output = Bump height. The purpose of these tedious and technical computations is to define a nice bell-like curve that fades out the wave smoothly. The nice and smooth curve used here is :

bump height = (r^2 -1)^2

We need a couple of VSL objects more. First add a Copy object to the end of the If sub hierarchy. Set Source to Surface:Time, and Destination to vector variable tmp. The Copy object's gadget shows three selectors which bind the sub channels. Change the first two to the None state, but set Sub channel 3 to Time. Then drop another Copy object to the If node. The default source Map coords is OK, but change Destination to tmp and Operator to Add from the popup. Now we have computed a set of modified mapping coordinates by adding the animation time to the third coordinate component. The purpose is to animate waves by sliding a noise field through the water surface. Sounds strange - but it works well! The last object to be added is Noise: Drop one to the end of the If hierarchy. Set output to Bump height and Input 0 to tmp. Set the Operator to Multiply from the popup menu. Then adjust the noise properties at the bottom of the property window: Set Octaves to 3 (it defines the level of detail in the waves), Amplitude = 0.1 (height of waves = 0.1 meters), Smooth and 1D enabled. The material is ready. Its structure is shown on the right.

The VSL structure of the wave material

Close the property window and rewind the animation to the beginning. Take a parallel top view and select the boat from the object hierarchy section of the Select window. Then select Autofocus to find the boat (Autofocus is in the view's popup menu and in the view control icons). Zoom out a bit so that you see a reasonable area around the boat. Select only the lake+lakesurface level from the Select window. Then go to the Materials tab of the main toolbar, and activate the Parallel map tool. Select the waves material from the material list in the tool controls.

Click first time just behind the boat (where the waves are supposedly highest), and move the mouse downwards and to the right (the "shift" modifier key held down) to get a uniform rectangle. Click a second time to finish the parallel map. The edge of the parallel map should be 1-2 times the length of the boat. You may resize the parallel map later to adjust the extent diameter of the waving area.

We have to animate the wave parallel map exactly the same way as we animated the boat. Make sure that the parallel mapping is selected. Move the mouse over the middle point of the object space handle on the view window, and drag the handle so that it is in the top left corner of the parallel mapping (just behind the boat). We do this to ensure that the maximally waving area follows the path, maintaining the originally defined 'behind the boat' status. The path tools fit the object space origin to the path. Press the Shift key down and multiselect the nurbs curve, which was used in animating the boat. Go to the Lattice Mapping toolbar section and use the Path tool to animate the parallel map with the default options.

Take the camera view and start animation play. You should see the parallel map following the boat across the lake. Stop the playback when the boat is quite close to the camera, and test render. You should see the boat reflection nicely twisted because of the waves. The strength of this effect can be adjusted by editing the Amplitude slider of the Noise object in the waves material hierarchy.

At this point, the original reflections of the photograph do not follow the defined local waves. Let's fix that next. Activate lakesurface from the material tab of the Select window and open the property window.

Drop a new Variable object to the top of the object list inside the Surface properties shader. Rename it as tmp and set the type to Vector. Then drop an Operation object just above Illumination=Texture object. Set Output to tmp, Input 0 to Surface:Bump height and Input 1 to Surface:Bump normal. Select Multiply from the Type gadget at the bottom of the property window.

Drop a Copy object just below the previously added multiply operation. Change Destination to tmp and Operator to Add from the popup. Then select the already existing Illumination=Texture(Map coords) object from the VSL hierarchy and change its input from Map coords to tmp. The material is now ready. The subtle effect of backdrop reflection waves is achieved by adding the bump vector to the texture coordinates directly! This is just a fake but better than static reflections.

The final version of lakesurface material

Foreground layers

The last part of this tutorial explains how to add foreground mask objects. Namely, the photograph includes a tree at the right edge of the screen, and also the front bank of the lake is visible. It may happen that the boat meets these areas while travelling on the lake and hides them, which certainly looks strange. Also, the front bank should not reflect the boat.

Close the property window and take the camera view. At this point we have to pay some attention to the image aspect. We will draw the mask objects over a backdrop image on the view window, and therefore the view must have exactly the same image width/height ratio as the camera object being used for the final rendering of the animation. The camera object was created from the view settings, so they may already match. The numerical window and other windows appearing at the bottom of the screen may have changed the view size, so you may have to close some of them to restore the original situation.

Another possibility is to use the Spec tab of the property window to edit the camera object's properties. In the Image aspect section, there are some predefined settings to choose from, or you may enter the desired custom value. The most suitable value depends on the source of the backdrop image. If the lake image was digitized from a PAL video signal, then we should use the same image aspect in the 3D model as well, otherwise the backdrop will stretch (spheres will look like eggs, etc.). Let's assume that the backdrop is a PAL image. Therefore, choose PAL, then close the object property window, open a view property window, select the leftmost Camera tab and open the Image Frame section. Then enable the Fixed size option and enter the same aspect value (0.75 for PAL) to the Image aspect field. The height of the view is adjusted accordingly. If the height grows, the bottom area of docked views becomes invisible, because the view cannot stretch. If that happens, reduce the width of the view by 50 pixels and enter the aspect again. Repeat this until the view is capable of displaying the image with correct proportions.

After these preparations, copy the camera view onto the view window by selecting the camera object and using the View/Camera/Object view popup menu function. If the preparations were successful, the wire frame of the camera is visible as a thin rectangular border matching the view edges. If the camera view rectangle is not visible, zoom out a bit. If the side edges of the camera view rectangle are visible but the top and bottom edges are not, or if the top and bottom edges are far away from view borders, the view - camera aspect matching has failed and you should recheck the aspect ratios.

After taking the camera view successfully, close the object property window and the view property window. Select the Images tab from the Select window. Drag and drop the lake image object onto the view. This activates view's continuous backdrop drawing. Select the curve tool from the toolbar. Set the Type field of the tool controls to Analytic, which option creates a filled polygon.

Start drawing a polygon from the bottom left edge of the view, so that the polygon follows the outline of the front bank. After the rightmost point of the bank, click one point at bottom right corner of the view, then at the bottom left corner, and then Accept to finish the tool (see the image). Draw another polygon to cover the tree at the right edge of the image.

Foreground polygons

When the polygons are ready, open the view property window, go to the Draw tab and clear the Backdrop/Image field displaying the name of the image object, which was dropped onto the view. The backdrop disappears. Close the view property window.

Select the two polygons, and use the Drop to a level tool of the Select window's popup menu. Select and rename the created level as frontlayers. Open the property window and change the color of the frontlayers level to black. This eliminates diffuse shading (black diffuse color was added to the lakesurface material, but we will now use Backdrop0 material which does not eliminate diffuse color). Close the property window.

Make sure that frontlayers is still selected and activate the Default mapping tool from the Materials section of the toolbar. Select the Backdrop0 material, set Mapping type to Camera and click Accept. A default mapping object texturing the polygons becomes inserted to the hierarchy.

The long tutorial example is now ready. You can save and test render it, for example, to an .avi file, to see the created effects animated.

The final hierarchy and a sample image