Povray Gedöns - Scripts for Parallel Rendering of Animations

Back to Overview

Scripts for Parallel Rendering of Animations

Rendering a complex animation with Povray can easily take weeks if only a single processor core is at hand. Even with Povray 3.7, which supports multiple cores, the power of one multi-core system may not be enough to finish an animation in acceptable time. The obivious solution would be to let several machines render the frames in parallel. Povray doesn't have native support for this, but it can be done by the help of scripts. In this article, I'm going to present such scripts for parallel processing of Povray animations under Windows and Linux.

The scripts in the article are tailored for the following example situation:
We have an animation described by a scene file named landscape.pov and an .ini file landscape.ini such that, if Povray was started with landscape.ini as input, it would render all frames of the animation in one go. Also, we assume that the order in which the frames are rendered does not matter, ie. there is no data transfer from one frame to the next frame.
The animation shall be 458 seconds long at 30 frames per second. That is, landscape.ini looks similar to this:

;
; landscape.ini
;
Input_File_Name=landscape.pov
Output_File_Type=N ; image format is png
Jitter=Off
Antialias=Off
Width=1920
Height=1080
Initial_Clock=0
Final_Clock=458
Initial_Frame=1
;
; Final_Frame = (Final_Clock - Initial_Clock) * FPS + Initial_Frame
; = 458 * 30 + 1 = 13741
;
Final_Frame=13741

Started with this .ini file, Povray would render the images landscape00001.png till landscape13741.png. Now, to make rendering faster, we'd like to have several instances of Povray rendering in parallel. This can be achieved by applying the following scripts.

Windows

The first script presented, builder.bat, runs on Windows. It should be placed in the same folder as the .ini and .pov file and works like this:

1. The script searches the current folder for the image files landscape00001.png till landscape13741.png.
2. When it detects that an image file does not exist, it starts an instance of Povray to render that image file (only that file, none else).
3. When Povray finishes, the script looks for the next non-existing image file, starts Povray to render it, and so forth.

To achieve parallel rendering, you simply start builder.bat several times. For instance, when using Povray 3.6 on a machine with four CPU cores, you'd click builder.bat four times. Or you could run several Povray instances on different machines, working on the same folder shared through the network.

To prepare the script to your needs, you just have to define some variables at its beginning:

@rem builder.bat
@rem
@rem Variables to be defined:
@rem povray - The path of the Povray executable.
@rem povname - The name of the scene to render, without extension.
@rem digits - The number of digits in the index of the image files.
@rem suffix - The extension of the image files, eg. png.
@rem s       - Number of the initial frame.
@rem e       - Number of the final frame.
@rem
@rem Author: Burkhard Reike, 2011-10-29

@echo off
setLocal EnableDelayedExpansion

set povray="C:\Users\Burkhard\AppData\Roaming\POV-Ray\v3.6\bin\pvengine-sse2.exe"
set povname=landscape
set digits=5
set suffix=png
set s=1
set e=13741

echo Starting: %computername% %date% %time% >> render.log
set lockdir=%povname%.lck
for /L %%c in (%s%,1,%e%) do (
set padcntr=0000000000%%c
set filename=!povname!!padcntr:~-%digits%!
call :Lock
if not exist !filename!.%suffix% (
    type NUL > !filename!.%suffix%
    call :Unlock
    call :GetTime
    set starttime=!cseconds!
    start /WAIT /min "Render" %povray% -d +SF%%c +EF%%c /exit +I %povname%.ini
    call :GetTime
    set endtime=!cseconds!
    if !endtime! LSS !starttime! set /a endtime+=8640000
    set /a elapsed=!endtime!-!starttime!
    set /a elapsed/=100
    echo !filename!.%suffix% !elapsed! %computername% >> render.log
) else (
    call :Unlock
)
)

echo Finished: %computername% %date% %time% >> render.log
goto:eof

:GetTime
for /f "usebackq tokens=1-4 delims=:., " %%f in (`echo.%time%`) do (
set temp=%%f %%g %%h %%i
)
for /f "usebackq tokens=1-4" %%f in (`echo %temp: 0= %`) do (
set /a cseconds=%%f*360000+%%g*6000+%%h*100+%%i
)
goto:eof

:Lock
md %lockdir% 2>NUL
if errorlevel 1 (
   rem ping seems the only "sleep" command availabe on all Windows versions
   ping -n 2 127.0.0.1 >NUL
   goto Lock
)
goto:eof

:Unlock
rd %lockdir%
goto:eof

Along with the image files, the script generates the file render.log with information about the rendering process, eg. when the scripts were started, finished, and how many seconds each image file took to render.
The script also provides a lock mechanism such that a process has exclusive access to the "if not exist !filename!" - "type NUL > !filename!"statements. This prevents two processes from rendering the same image. The locking is achieved by temporarily generating a .lck directory.
When a script is started, a DOS shell window will open, and it starts Povray instances one after another. Each time a Povray instance starts, you will see a splash screen. This cannot be avoided. Still, the Povray window does not pop up owing to the "-d" option.
If you want to stop rendering for a longer time, you just have to kill the dos shell windows. Each currently running Povray instance will stay alive till it has finished rendering the current frame.

If, for some reason, you have killed Povray instances, the images they had been rendering will remain unfinished. Starting new instances of builder.bat will not rerender them unless the files have been removed. To identify such image files, the following script, broken_files.bat, comes in handy.
It prints the names of all image files which exist in the current folder but which are not mentioned in the log file. Typically, those are broken image files whose Povray instances were killed prematurely. broken_files.bat should be placed in the same folder as the .ini, .pov, and builder.bat file.

@rem broken_files.bat
@rem
@rem Variables to be defined:
@rem suffix - The extension of the image files, eg. png.
@rem
@rem Author: Burkhard Reike, 2011-10-26

@echo off
setlocal EnableDelayedExpansion

set suffix=png

for %%f in (*.%suffix%) do set allfiles=!allfiles! %%f
for /f "tokens=1 delims= " %%f in (render.log) do set logged=!logged! %%f
for %%g in (!allfiles!) do (
    set found=0
    for %%h in (!logged!) do if "%%g"=="%%h" set found=1
    if !found! EQU 0 echo %%g
)
pause

Linux (Ubuntu)

There are several good reasons to use Linux for Povray rendering. Linux requires few disk space, few RAM, and renting a Linux server is usually cheaper than renting a Windows server since you don't have to pay for a license. The following scripts were created for the Ubuntu 8.04 Linux distribution. The scripts builder.sh and broken_files.sh are equivalent to the Windows scripts described above. Additionally, the script add_render_process.sh will be presented which can automatically mount a folder from a remote machine through SSHFS such that you can start Povray instances on several machines, all instances rendering into the same shared folder. All operations can be executed from a login shell, you don't need a desktop environment like KDE or so.

For sharing a remote render folder, you need to have SSH/SFTP support installed (most Linux distributions, even the minimal ones, come with it already installed), and the SSH port of each machine used should be open such that remote mounting can be applied. SSH is actually the only connection you will need since it's also the connection of choice for remote login and file transfer (eg. with the Windows programs PuTTY and WinSCP), since it's encrypted. Apart from SSH/SFTP support, you should have the Screen tool installed. Screen will enable you to logout from a machine without hanging up the render and mount processes. You just have to execute Screen to open a new shell, start the scripts, detach from Screen, and logout. Screen will stay alive, and so will the sshfs and render processes started from it.
To install Screen in Ubuntu, type:

$ sudo apt-get install screen

Let's assume that you have a user account called "john" on each machine. Then you should organize the render folders and scripts as follows. On each machine, put the add_render_process.sh script into a certain folder, eg. a folder inside your home directory called "povray":

/home/john/povray/add_render_process.sh

One machine, which we call master, must also contain the render folder. The render folder contains the .ini, .pov, and remaining .sh files. Besides, it is the folder into which the image files will be rendered:

/home/john/povray/render/builder.sh
/home/john/povray/render/broken_files.sh
/home/john/povray/render/landscape.ini
/home/john/povray/render/landscape.pov

All other machines, which we call clients, do not have a render folder preinstalled. Instead, they will mount it from the master machine by the help of the add_render_process.sh script.
For this purpose you will need to know the IP-address of the master machine. You can find it out while logged in on the master machine with
$ ifconfig eth0
or
$ ifconfig eth1
depending on the hardware configuration. Inspect the output of the ifconfig command. The number right of the "inet address:" entry is your IP-address. Possibly you have two IPs, one public IP, and one private IP to be used for inter-server communication. If so, you should pick the private IP of the master machine.

So much for the setup. Now for starting the render processes.

You don't necessarily have to start render processes on the master machine since its main purpose is to keep the render folder. But if you also want to use it for rendering, proceed like this. First, login on the master machine (as "john") and go to the povray folder. Then run Screen. Let's assume that each machine has four cores, so run add_render_process.sh four times (with the render folder path and the builder script name as arguments). By this, four builder.sh scripts will be started, which again start Povray instances to render image files. Finally, and detach from Screen. Then you can logout. The render processes will keep running in the Screen shell:

$ cd povray
$ screen
$ ./add_render_process.sh /home/john/povray/render builder.sh
$ ./add_render_process.sh /home/john/povray/render builder.sh
$ ./add_render_process.sh /home/john/povray/render builder.sh
$ ./add_render_process.sh /home/john/povray/render builder.sh
[Now press "Ctrl+A" and then "D" to detach from Screen.]
$ logout

Now for starting render processes on the client machines. On each client machine, you basically perform the same steps as on the master machine. The only difference is that the calls of the add_render_process.sh will have two additional arguments, namely the name of the user on the master machine who owns the render folder, and the IP-address of the master machine. Let's say the IP-address is 12.34.56.789. Then the commands to be executed on each client machine look like this (again, let's assume we want to run four Povray instances in parallel):

$ cd povray
$ screen
$ ./add_render_process.sh /home/john/povray/render builder.sh john 12.34.56.789
$ ./add_render_process.sh /home/john/povray/render builder.sh john 12.34.56.789
$ ./add_render_process.sh /home/john/povray/render builder.sh john 12.34.56.789
$ ./add_render_process.sh /home/john/povray/render builder.sh john 12.34.56.789
[Now press "Ctrl+A" and then "D" to detach from Screen.]
$ logout

The add_render_process.sh script will automatically install the SSHFS package, add the user to the fuse group such that he can start remote connections, mount the render folder and start the builder.sh script.
You will probaly be asked two times for passwords. The first time occurs when the first "sudo" command in the add_render_process.sh script is executed. Then you will have to enter the root password of the client machine. The second time you will have to enter the password of user "john" on the master machine, in order to establish the remote connection through sshfs. When connecting for the first time to the master machine from the client machine, you'll also have to confirm that you want to establish the connection.
Here are the scripts:

#!/bin/bash
# add_render_process.sh <full-path-of-render-directory> <name-of-renderscript>
#                       [<remote-user-name> <remote-host-address>]
#
# Author: Burkhard Reike, 2011-10-21

if [ $# -ne 2 -a $# -ne 4 ]; then
    echo "Usage: $0 <full-path-of-render-directory> <name-of-renderscript>"
    echo "       [<remote-user-name> <remote-host-address>]"
    exit 1
else
    RENDERDIR=$1
    if [ $RENDERDIR != */ ]; then
        RENDERDIR="$RENDERDIR"/
    fi
    RENDERSCRIPT=$2
    if [ $# -eq 4 ]; then
        USER=$3
        HOST=$4
        ISMASTER=false
    else
        USER=''
        HOST=''
        ISMASTER=true
    fi
fi

if [ -d "$RENDERDIR" -a -n "$(ls -A $RENDERDIR 2> /dev/null)" ]; then
    # Render directory already exists and is non-empty, so don't mount it.
    if [ !$ISMASTER ]; then
        echo "$0: Render directory seems already mounted. Mounting omitted."
    fi
else
    # Render directory doesn't exist or it's empty. Try to mount it.
    if [ !$ISMASTER ]; then
        if [ ! -d "$RENDERDIR" ]; then
            mkdir $RENDERDIR
        fi

        #
        # Mounting the render directory via sshfs
        #
        sudo chown `whoami`:`whoami` $RENDERDIR # Fix the render dir ownership
        sudo apt-get install sshfs # Install the sshfs package. This may have
                                   # to be adjusted, depending on your Linux
                                   # distribution.
        sudo modprobe fuse                      # Load the fuse module
        sudo adduser `whoami` fuse              # Add us to the fuser group
        sudo chown root:fuse /dev/fuse          # Fix /dev/fuse ownership
        sshfs -o ServerAliveInterval=15 $USER@$HOST:$RENDERDIR $RENDERDIR

        # Wait till the render directory is non-empty
        echo "$0: Waiting for content in the render directory..."
        until [ -d "$RENDERDIR" -a -n "$(ls -A $RENDERDIR 2> /dev/null)" ]; do
            sleep 2
        done
        echo "$0: Render directory was mounted sucessfully."

    else
        # The master process needs a non-empty render directory,
        # but could not find it.
        echo "$0: Could not find script $RENDERDIR$RENDERSCRIPT. Exiting."
        exit 1
    fi
fi

echo "$0: Starting background render script $RENDERDIR$RENDERSCRIPT"
cd $RENDERDIR
./$RENDERSCRIPT &>/dev/null &

#!/bin/bash
# builder.sh
#
# Variables to be defined:
# POVNAME - The name of the scene to render, without extension.
# DIGITS - The number of digits in the index of the image files.
# SUFFIX - The extension of the image files, eg. png.
# S       - Number of the initial frame.
# E       - Number of the final frame.
#
# Author: Burkhard Reike, 2011-10-29

echo "Starting: `hostname` `date`" >> render.log

POVNAME=landscape
DIGITS=5
SUFFIX=png
S=1
E=13741

LOCKDIR=$POVNAME.lck
lock() {
   while ! mkdir $LOCKDIR 2> /dev/null
   do
     sleep 1
   done
}
unlock() {
   rm -fr $LOCKDIR
}

while [ $S -le $E ]; do
    INDEX=`printf "%0"$DIGITS"d" ${S}`
    FILENAME=$POVNAME$INDEX.$SUFFIX
    lock
    if [ ! -f $FILENAME ]; then
        touch $FILENAME
        unlock
        BEFORE="$(date +%s)"
        povray +SF$S +EF$S -d +I $POVNAME.ini &> /dev/null
        AFTER="$(date +%s)"
        ELAPSED="$(expr $AFTER - $BEFORE)"
        echo "$FILENAME $ELAPSED `hostname`" >> render.log
    else
        unlock
    fi
    S=$(($S+1))
done
echo "Finished: `hostname` `date`" >> render.log

#!/bin/bash
# broken_files.sh
#
# Author: Burkhard Reike, 2011-10-21

for F in *.png
do
    GREPOUTPUT="$(grep $F render.log)"
    if [ -z "$GREPOUTPUT" ]; then
        echo $F
    fi
done

Here you can download all files in one package: povray_parallelrender.zip

That's it! I hope you will find all this useful.
Thanks to Dave for the hint on the locking mechanism on Linux.

Cheers,
Burkhard

Back to Overview