The post processing system has an important role when preparing rendered images for a compositing system (software/hardware that reads image sequences and blends them together using alpha channel and other information). Especially the VSL effect is useful in this context. The user can define any necessary channel modification using it.

In this section, three compositing cases are examined:

Tutorial level: Medium/advanced

High quality rendering requires use of anti-aliasing. In a compositing situation, anti-aliasing has one undesirable side effect: The edge pixels of objects become blended with the background color. For example, if you render the scene over a blue background, some pixels will inevitably include color both from the rendered object and from the background.

The leftmost square is a magnification of a pixel, which is split by an edge of a red object. After rendering, the color of the pixel is the average of all colors included in the pixel - dark purple shown in the second square. If the rendered scene is then 'superimposed' over a new background, the purple pixels become an undesirable glow or color error. The third square representing a 4*4 pixel array shows this situation on a white background.

A suitable VSL effect can remove this problem. The next example demonstrates this.

Example file: 'tutorprojects/postprocessing/nobackgroundantialiasing'.

1. Model a simple test scene: add a couple of objects with different colors and add a light source or two.

2. Compositing usually requires use of the alpha channel. Select the 'root' level of the scene, open the property window and go to the 'Col' tab. The 'Surface Properties' frame includes 'Attribute' list - select 'Alpha' from it and set the value slider below to one. In most compositing systems alpha level one means full non-transparency. If the scene contains glass (or otherwise transparent) objects, you set an alpha value of 0.5 - 0.8 for them.

Alpha=1.0 makes objects non-transparent in compositing

3. Go to the Post Image Effects tab of the select window. Select 'New/Post Image' from the popup menu. Then select the 'New/VSL effect from the popup. Two new objects appear to the end of effect list.

4. The property window shows the latest created effect - the VSL effect. Change its name to a more suitable one, 'removebackgroundaa'. The effect is currently empty and does nothing. Check the Advanced option to open the VSL editor, and open the VSL Objects frame.

5. Drag and drop an If object from the VSL Objects into the Image processing node. Open the popup menu of the VSL hierarchy gadget and set Input 0 to Image:Alpha. The lower part of the property window shows If specific properties. Select If Inside from the gadget labeled with Sub 1. Set Lower to 0.001 and Upper to 1.001. Now the If object tests if the Alpha channel value is non-zero - in other words, if the pixel is not a purely background colored pixel. Other sub channels labeled as Sub 2 and Sub 3 need not to be tested, because the alpha channel has no sub channels (unlike the Color channel with R, G and B sub channels).

6. Drag and drop a 'Copy' VSL object into the 'If' node of the VSL tree. Using the popup menu, set 'Source' to 'Image:Alpha' and 'Operator' to 'Divide'. Set all the three sub channel selectors on the bottom of the property window to 'Alpha'.

Now the VSL code divides color of all pixels that have non-zero alpha level with the alpha level. This compensates the blending with a black background, because alpha measures the amount of non-transparency included in a pixel.

There may be some isolated pixels where the alpha level is very small ('If' operator accepted alpha level greater than 0.001). We may clean these up as follows:

7. Drop a new 'If' object into the 'Image processing' node. Set exactly the same 'Input 0', 'Lower' and 'Upper' as in the step 5 above, but this time select 'If not inside' to the 'Sub 1' sub channel test selector.

8. Drop a 'Constant' object into the new 'If' object. Select 'Output' = 'Image:Alpha' from the popup menu. In other words, we reset very low alpha channel values to zero. This is in most cases only a 'cosmetic' operation, because alpha values below 0.001 are truncated to zero by all common file formats (such as Targa). The VSL effect is now ready.

9. Select the 'Post Image' object which was created in the beginning of the tutorial, from the select window. We configure its contents next using the property window. If the scene contains particles, drag a 'ParticleEffect' object from the 'Available Effects' to the 'Active Effects' list. Then drag & drop the created VSL effect to the 'Active Effects' list. Finally rename the post image in a suitable way, for example 'No-background-aa'. Close the property window.

10. The post processing configuration is now ready for use. We can try it by rendering the view window. Drag and drop the post image (No-background-aa defined above) from the select window to the view window. Then start ray tracing.

The final structure of the VSL effect

You will see the usual shaded rendering appearing to the window, with anti-aliasing that makes object edges appear smooth and nice. After a while, the post processing replaces it with a new image where edges against the background look jagged and ugly! Do not worry, this is exactly the correct result. The alpha channel contains the information for making the edges smooth again when the image is merged over a new background.

A part of rendering magnified. Note, how interior pixels are nicely anti-aliased, but pixels against the background are not

[Note] Note
The above mentioned post processing configuration works only when rendering over a black background. Do not use a backdrop image with it.

Preparing images for compositing using a second rendering pass

When using motion blur, the background anti-aliasing elimination presented above cannot be used. The motion blur system assembles images together after post processing and the background color becomes blended with the object colors again. It is better to render the animation frames in the usual way and make another rendering pass to fix the background anti-aliasing.

Example project: 'tutorprojects/postprocessing/preparecompositing'.

1. Load in the scene modeled in the previous tutorial ('tutorprojects/postprocessing/nobackgroundantialiasing'), unless you continue directly from it.

2. We will need a short and simple test animation. Open the animation window (from 'View/AnimationWindow' menu) and set frame count to, say, 5 frames only. Select one of the scene objects, move the time slider to the last frame, activate animation recording button and move the object to a new place. Turn recording off and rewind the animation.

3. Prepare motion blur rendering (see the rendering tutorial 'Motion Blur' for details): Activate 'Motion Blur' using the 'Windows/Animation Settings' menu. Four samples are enough for the basic test. Set anti-aliasing of 'Reasonable Quality' rendering settings to 'Stochastic' and anti-aliasing level to zero. Make sure that the Output objects tab of the select window includes a 'Targa RGBA' object - if not, create one. Add a camera object to the scene.

4. Open the file rendering window. Define a suitable file name (e.g. 'aa_test'). Add 'Targa RGBA' to 'Active File Formats' and remove possible other active formats. Select 'Reasonable Quality' to the 'Rendering Settings' gadget. Set 'Image Effect' to 'Default Effects' or leave it empty.

[Note] Note
Do not select the No-background-AA configuration yet. Set the resolution to a suitable test value, say 300*300 pixels. Hit the 'Render Animation' button and wait until rendering is finished.

So far we have not done anything new - the above mentioned briefly described steps only render a motion blurred test animation to a sequence of Targa files. If you take a look at one of the rendered images, it looks as in the example image on the right.

The image shows that the red moving rectangle is blurred with the black background. This will be a serious problem if the sphere and rectangle are composited over a white background. The second rendering pass fixes the Targa image sequence. This is the actual issue of this tutorial and is described next.

5. The scene is no longer needed, so delete all objects. Open the Animation Settings window from the 'Windows' menu and turn off 'Motion Blur'. The second rendering pass manipulates already blurred images and the motion blur should be switched off this time. Close the Animation Settings window.

6. Go to the rendering settings tab and create a new settings object. Open the property window to edit the rendering settings. Rename it as 'Compositing settings'. The purpose is to prepare a rendering settings object, which renders the empty scene as quickly as possible - the actual work is done in post processing. Set 'Undersampling/Horizontal & Vertical' both to the level 4 and clear the 'Interpolate' option. Turn off 'Box rendering' from the 'Distrib' tab. You should not use network rendering during the second rendering pass, because the simple configuration is probably faster to compute locally instead of sending image data across the network.

7. Switch to the Post Image Effects tab of the Select window, and select the VSL effect 'removebackgroundaa'. This time it should not process ray traced data (from the empty scene), but Targa disk files. Therefore, we have to modify the effect a bit. So, open the property window (if not open yet).

8. Check the 'Advanced' options of the property window if not checked. The VSL effect already contains the two 'If' blocks which were inserted in the previous tutorial. Drag a 'Texture' object from 'VSL objects', move the mouse pointer over the top edge of the upper 'If' icon before releasing the left mouse button (the cursor changes to an up arrow when insertion point is correct) and drop the texture object there. The texture should be read before 'if' tests, therefore it should be first object in the VSL tree.

9. At the top of the property window you can see a gadget labeled as 'Image Coordinate Channel' and it is set to 'UV coords'. Therefore, using the popup menu of the VSL tree gadget, set 'Input 0' of the inserted texture object to 'UV coords'.

Color channel loader inserted

10. The inserted Texture object should load images of the motion blurred Targa sequence. Therefore, we have to define properties of an animated texture. Select the Animation tab from the bottom of the property window. Set Animation Type to Automatic for automatic frame indexing.

11. Switch to the Texture tab and enter the name of the rendered images without index but with the format postfix to the File Name gadget. For example 'AA_test.tga'.

Options of an animated texture

12. Make sure that the Texture object is still selected and choose Copy from the popup menu of the VSL tree gadget. Then select Paste from the same menu. A copy of the Texture object appears to the end of the VSL tree. Drag and drop it just below the first Texture object (above the If objects). Change the Output to Image:Alpha using the popup menu. Close the property window.

The finished VSL structure

The VSL code for the effect is now ready. Instead of processing rendered image data, the effect removes background anti-aliasing from a sequence of disk files.

13. Open the file rendering window. Change the file name to a new one, e.g. 'secondpass'. The active output list should again include only 'Targa RGBA'. Select the fast-to-render 'Compositing Settings' to the 'Rendering Settings' gadget. Set image effect to Nobackgroundaa (=the post image including the prepared VSL effect). Use the same resolution as in the first motion blur rendering. Hit Render Animation to start rendering.

File rendering settings for the second rendering pass
Background color blending eliminated

Attached is a processed sample image. There is no motion blurring left at all and the anti-aliasing is poor. Alpha channel includes the necessary information for blending the image over a new background, as the next tutorial shows.

Compositing using the alpha channel

The Targa image sequence, which was processed in the previous tutorial, includes alpha channel information and is suitable for use in various composition systems. For example, a video hardware device can insert the computer generated graphics over a live video signal.

It is also possible to use Realsoft 3D itself for doing the compositing job. Thanks to VSL, the user can freely define any kind of keying algorithm, which blends two or more image sequences. The next example shows how to achieve this.

Example file: 'tutorprojects\postprocessing\compositing'.

1. Go to the Post Image Effects tab of the Select window. Select 'New/Post Image' from the popup. Then select 'New/VSL Effect' from the popup.

2. Open the property window. Check the 'Advanced' button to edit the new empty VSL effect. Rename the effect as 'Compositing by VSL'.

3. Drag and drop three 'Variable' objects from 'VSL Objects' frame into the 'Image processing' node of the VSL tree. Set the names and types of variables using the controls at the bottom of the property window as follows:

4. Drag and drop a 'Texture' object to the VSL tree. Set 'Input 0' = 'Image:UV Coords' and 'Output' = 'foreground'. Define the properties of the animated texture sequence exactly as in the previous tutorial, but this time use the second, processed sequence: ('Animation Type' = ''Automatic', 'Frames' = 5, 'File Name' = 'secondpass.tga').

5. Copy and paste the texture object. Change 'Output' of the pasted texture object to 'key' using the popup menu.

6. Drag and drop a third 'Texture' object to the end of the VSL tree. Set 'Input 0' to 'Image:UV Coords' and 'Output' to background. Select a suitable background image or image sequence. For example, use the good old 'textures/lake.jpg' image.

7. Now we have inserted three texture loaders. The remaining step is to combine their information. Drop an 'Operation' object to the end of the VSL tree. Set the 'Type' of operation (using the gadget at the bottom of the property window) to the long formula at the end of type list: '(1-p1)*p2 + p1*p3'. Mathematically oriented users may notice that this is the formula of linear interpolation. Using the popup menu, set 'Input 0' to 'key', 'Input 1' to background and 'Input 2' to 'foreground'. The default output is already 'Image:Color' which is the correct target channel.

The VSL effect for image composition

8. The VSL effect is now ready. Create a new post image effect object from the select window. Rename it as Compositing effects. Insert only one effect to it, namely Compositing by VSL. Close the property window.

9. Open the file rendering window. Define a new file name, for example 'thirdpass'. The output can again be a Targa file, but you can also use other formats such as 'Windows BMP'. The VSL effect above does not maintain the alpha channel (because it is already used once for compositing). Set 'Compositing settings' to the 'Rendering Settings' gadget and 'Compositing effects' to the 'Image Effect' gadget. The resolution should be the same as in the previous rendering passes. Start file rendering by hitting the 'Render Animation' button.

The sample image above shows how the blurring appears again, but now with a new background photograph. Smooth anti-aliasing is included.


The detailed tutorial examples as outlined above, constructed all the necessary compositing tools and objects from the very beginning. In a real situation, you can reuse these example projects:

Of course, you can include the necessary post processing, rendering settings and file rendering configurations to your startup project. Then all the necessary tools would be available all the time.