Render farm

Render farm, also called Compute Cluster, is a software that is designed to queue up and execute resource (CPU and/or GPU) intensive long running tasks on a group of computers.

It is commonly used in post production for image processing, offloading heavy workloads from workstations to a set of dedicated render nodes.

accsyn has built in render farm functionality, providing execution of render applications through Python script based engines. By combining file transfers with render jobs, accsyn supports building a render farm that spans multiple physical locations such at on-prem and cloud rendering.

The accsyn render farm feature is API centric and is designed to be integrated into other Python enabled applications such as DCC (Digital Content Creation) softwares (Maya, Unreal, Houdini etc), although render jobs can be submitted using the accsyn Desktop app to some extent.

Note: Reach out to the accsyn support if you need the desktop app extended to fully support render job submission.

accsyn provides queue management out of the box, as part of the file transfer mechanism. Render jobs are just a different type of jobs that instead of transferring files, execute applications through "engines". 

Render jobs can have file transfer dependencies, and these are set up if a job is submitted from another site or if the render job should render on a remote site. This makes accsyn the only render manager to fully support multi-site/cloud render workflows natively.

Engines

Each render application, e.g. Houdini, FFmpeg, Unreal Engine and so on, is defined by an Engine in accsyn.

An engine consist of a freely customisable Python script, that is synchronised and executed in runtime at render nodes.

There is the mandatory Common engine, that must be installed before any other engines can be installed. The common engine provides the base Python class for engines, and has shared functions and interfaces that is used by the engine.

Lanes

To be able to provide parallellism, accsyn provide something called a "lane". A lane is a virtual render server on a physical server, each server can have one ore more lanes.

Each lane can have one or more engines assigned, enabling basic planning of your render farm - define which machines are allowed to execute which applications.

Filters

Filters are how you define the rules for each job, for example only render at machines having at least 64GB of RAM or on a certain pool (see below).

Pools

To provide more advanced render farm scheduling, accsyn supports something called "pools". A pool is a group of render servers, and each render job can use a pool in different ways through filtering:

• Include; Only run at servers part of the pool.

• Exclude; Never run on servers part of the pool.

• If unused; Run on servers if no other job is including them.

• Dedicated; Alwas run on these servers, even if other jobs higher up in queue are using them.

The accsyn Render farm feature is licensed per configured render server, were a configure render server is a server with at least on engine assigned and enabled. There is no restrictions on number of engines or amount of rendered jobs or frames.

The amount of active render servers are measured each night at 00:00 CET and the monthly top notation is used as reference for next billing period invoice.

For more information, please visit https://accsyn.com/pricing. To check your current render farm usage, visit your workspace billing page @ https://accsyn.io/signup.

accsyn provides a set of default engine scripts, available as open source on Github:

• An active BYOS accsyn workspace and an active administrator login.

• One or more dedicated render computers, with the render application installed and licensed.

In this guide we will showcase how to setup render of Houdini 20.5 mantra scenes on a bunch of Windows machines, using the already available compute scripts provided on our Github.

As a first step, we need to enable compute:

1. Go to admin.io/admin/settings and Compute tab.

2. Check Enable compute.

Common engine

Before we can install any engines, we need to install the common engine - the base:

1. Logon as an administrator at accsyn.io/admin/engines and click Create engine.

2. Enter the engine name common.

3. Open the common script on Github as raw, link: https://raw.githubusercontent.com/accsyn/compute-scripts/refs/heads/main/source/common.py

4. Copy the script and paste in Python script entry.

5. (Optional) Set description and vendor [accsyn]. Color has no effect here.

6. Click Create & Publish.

You now have the base setup.

Application engine

• Logon as an administrator at accsyn.io/admin/engines and click Create engine.

• Enter the application engine name, we recommend giving the exact same name as the python script is named, without the .py extension [mantra-20.5]

• Open the common script on Github as raw, link: https://raw.githubusercontent.com/accsyn/compute-scripts/refs/heads/main/source/mantra-20.5.py

• Copy the script and paste in Python script entry.

• (Optional) Set description 

• (Optional) Set vendor (SideFX). 

• (Optional) Set the color, used in farm view to distinguish applications.

• Click Create & Publish.

You now have a configured render farm and are ready to install nodes.

To be able to execute engine scripts, you will need a server:

1. Go to Workspace menu>Administrate>Servers (https://accsyn.io/admin/servers) and click INSTALL SERVER.

2. Choose Render server role.

3. Conclude the server installation by installing the daemon and authenticate it using the code displayed.

4. Edit the server and go to Lanes & Engines tab.

5. One lane should be displayed, to change the number of lanes - go to Attributes tab.

6. Right click on the lane, choose the engine [Mantra 20.5] and choose Available.

The render server is now setup and ready to run jobs, reload the web admin pages to have the Farm menu option appear on left hand side - use it to monitor your render servers.

Prerequisites

Prepare the file to render and save it with dependencies to an accsyn volume. To be able to render, the input file(s) and dependencies has to reside on an accsyn volume. Also the generated output file(s) needs to be written back to a volume.

In this guide we need an exported .ifd 100 frames file sequence from Houdini ready to be rendered with Mantra in command line mode.

Finally, the submitting user account needs access to the volume. Standard users can be given explicit access to submit render jobs (see Manage render farm section).

Submit using the accsyn Desktop app

Notes/hints: 

• Submitting render jobs from the accsyn web UI is not supported yet.

• Newly added engines, or your custom engines, are not automatically available/supported to submit with desktop app. Please reach our to support making a implementation request.

• To view the resulting render job submit API payload, click the JSON button next to RENDER button - it is very helpful when designing your own API based submit logic.

1. Download and install the accsyn Desktop app.

2. Login using with a user that has permission to access the input files at the volume, and submit render jobs.

3. Open the Render tab.

4. Drag and drop the input file [render.01001.ifd] on the area or click Storage button and browse to the file.

5. Eventual input file sequence will be detected.

6. Check Parse input for dependencies to have accsyn parse ascii input file(s) for dependencies when engine is selected, and track them - enables proper cross-site rendering.

7. Select the engine [Mantra 20.5]

8. Enter the frame range to render, either as a single continuous range or a set of ranges [1001-1100]. See examples below

9. (Optional) Enter one or more frames or ranges to render before the rest, it has to be within the main frame range above.

10. (Optional) Adjust the job attributes as needed, see below for descriptions.

11. Click RENDER in bottom right corner to submit the job to the farm.

Render job attributes/settings:

• Split mode (for engine supporting items); choose if should render the entire job on a single machine without splitting (Single task) or if it should be split up in buckets(default: 5 frames per machine) across render servers. 

• Filter:Estimated RAM usage; Only run on servers having at least the ram amount chosen.

• Filter:Select which site(s) to render at (cross-site render only); Choos the sites to render at, default is to render on all available server across all sites.

• Manual filter input; Manually enter filters, comma(,) separated list of filter expressions that all must be fulfilled before render is dispatched to a server. Syntax:

  • RAM spec; 

    • ram:>60GB only run at machines having at least 60GB ram. 

    • ram:<60GB only run at machines having less than 60GB ram.

  • Cores spec; 

    • cores:=12 only run at machines having exactly 12 cores (threads). < and > operators works aswell.

  • Site constraints; 

    • site:dupp+sthlm only run at the sites "dupp" and "sthlm". 

    • site:-dupp exclude site "dupp".

  • Hostname constraints; 

    • hostname:ws01+ren01: only run at servers "ws01" & "ren01". Define site & hostname: 

    • hostname:dupp/pc01. hostname:*render*: only render at servers having "render" in their hostname.

  • Pool constraints (each server can be member of one or more pools):

    • pool:+mypool  only run a servers member of "mypool";

    • pool:-mypool avoid servers member of pool "mypool".

    • pool:~mypool use pool "mypool" if it is free - no other job includes the pool.

    • pool:@mypool have dedicated access to pool "mypool" servers, but also utilise other servers if they are available.

  • Dependencies; Enter path to each dependency one entry per row, either a local path on the form "D:\picture.png" or an accsyn path on the form "volume=projects/picture.png". Will be uploaded to workspace volume in the same manner as a local input file would, and distributed to remote rendering site in a cross-site rendering setup.

  • Upload dependencies to <workspace name>; If selected, the dependencies will be uploaded as part of the render job. De-select this if you already have sorted the synk by other means.

  • Output; Browse/create folder on accsyn storage were the result should be written by the engine/render application.

  • Clear output directory; Define if the output directory should be cleared before render is started on a new site, mitigates stray files present from previous renders to the same folder.

  • Download output from <workspace name> on finished items(s)/tasks(s); Decide if output should be continously synked back to submitting machine when a engine completes execution on a render server.

  • Additional render parameters (advanced); Define default DCC render command line parameters and other advanced attributes.

  • Common and platform environment variables; Enter environment variables, one entry per row, on the form "FLEXLM_DIAGNOSTICS=2".

  • Bucket size; Define how many items/task should be collected and dispatched to each render server. Requires engines to support items - e.g. each input file can be used for rendering multiple images defined by sub frame ranges (Maya, Nuke etc).

Submit using the accsyn Python API

The accsyn Python API allows for integrating render job submission inside the DCC application, or from another external tool. As mentioned above, you can use the JSON button in desktop app submitter to inspect the REST payload as it would have been submitted to accsyn - very handy when developing your own submitter.

Detailed information about the Python API and how to submit render jobs, are available within our Python API documentation:

Normally, all renders are executed on server computes physically located at the same premises as the file server were the accsyn storage server daemon is running, and there is no need to render elsewhere.

But in some scenarios, one would want to distribute the render of a job across multiple physical locations. This could be a remote office, a single power workstation located at an employee home office or at cloud infrastructure like GCE or AWS.

accsyn supports this, and it is called cross-site rendering.

Prerequisites

• At least one active remote site, manage sites at https://accsyn.io/admin/sites.

• A site server running at the remote site, serving the volume(s) were render input files and dependencies are located.

• Verified working file transfers between main site (hq) and the remote site.

Setting up remote render

As a first step, you will need to install at least one render server at the remote site. Follow the same instructions as you would for a render server at main premises - install & license DCC render applications and then install a new server @ https://accsyn.io/admin/servers. Remember to choose the remote rendering site when creating the server.

You are now basically all set to render, existing render jobs cannot utilise the new site but new render jobs will automatically have sync tasks created for the new tasks that will be activated as soon as an available render server is to be utilised.

How it works

1. Before the first item/task is launched, the download sync task for site is kicked into gear once. The synk task has sub tasks, one or more for the input file(s) and then one for each dependency. This to make sure that all data required by the DCC render application is available on the site before launched.

2. Whenever an item/task finishes, the upload sync task for site is retried unless already running. This to make sure the result is "streamed" back to the main premises and available for review.

Considerations

Cross-site rendering can be complex if the render process uses licenses or other network assets that are not available on all sites. Make sure to have these dependencies synced properly each time they are update, or add them as a sync dependency with the job API submit payload to be sure everything is read to go.

Also, syncing assets during render can cause extra overhead especially if the dependencies are huge and/or the site network bandwidth is limited. Always lock down the scenarios were you allow users to submit renders to remote sites, to avoid congestion and long delays in production.

As mentioned earlier, accsyn farm feature is API first meaning that it is primarily designed to be integrated into other software or built into your own production toolset.

In this example we build a minimal Python (PySide) based submitter designed to be launched as a standalone desktop application, source code.

accsyn-submitter.py:

Most likely, the default engines provided by accsyn does not cover your needs and you will need to create your own engine script. This guide walks through the basics, we recommend you take a look at the existing engine scripts to get inspiration.

Developer guidelines/prerequisites

• A code editor (IDE) - Visual Studio Code, PyCharm or similar.

• Python 3 

• (Recommended) Source code control - Git(hub)/Perforce/CVS.

Script structure

Engine scripts must adhere the following base structure:

Breakdown of the engine script:

• Engine config; defines the settings for engine, for example if the application supports items or the default command line arguments to pass on to app.

  • items (boolean); True means each input file can the source of multiple output files, for example the case for Maya, Nuke, Houdini. Some renderers like Houdini Mantra and Arnold takes a file sequence as input, still it will be executed as numbered items defined by a sub frame range on each render server. ffmpeg on the other hand does not support items - each input file is executed as a task and generates exactly one or more output files.

  • multiple_inputs (boolean) True means the the engine script supports multiple inputs, this is false for Maya, Nuke etc but true for ffmpeg.

  • filename_extensions (string);  Comma separated list of input filename extensions associated with the underlaying (DCC) application, for example ".ma,.mb" for Maya.

  • binary_filename_extensions (string); Comma separated list of filename extensions that denotes binary file format, this tells accsyn which input files can be parsed during submit with the desktop app or now.

  • binary (boolean); Tells accsyn that all input files are binary.

  • default_range (string); The default frame range to suggest in desktop app submitter.

  • default_bucketsize (number); The default bucket size to suggest in desktop app submitter.

  • max_bucketsize (number); The maximum bucket size render application supports.

  • default_output_path (string); Suggest this default output path.

  • type; The type of engine, default is "compute" for DCC rendering applications. The rest is special engines not covered by this documentation.

• Init; instantiate the engine, and also define additional class variables.

• Get executable; Evaluate and return the path to application binary executable, will be the first element of command line.

• Get envs; (Optional) Build a dictionary holding environment variables to pass on to application.

• Get commandline; Build the full command line to launch.