Tutorial - Automised production outsourcing

Rev 1.005, 2019.10.28

How to read this tutorial


Summary


This tutorial is targeted :


Wishing to:


This tutorial is a complete walkthrough covering:

Scenario


Your task would be to have large package of material being sent smoothly to a remote subcontractor "jennifer@gda.com" when task is assigned in your production management system, for easy sync and pick up of work project files.  You also want to implement a "button" that freelancer can press that sends back the result into the correct folders at headquarters, with a post publish step that updates your internal project tracking tool.

IMPORTANT NOTE: The terms mentioned/configuration steps outlined below are described in detail in our Accsyn Admin Manual, for instructions on how to use the Accsyn desktop app - have a look at the Accsyn User Manual. Always consult the manuals before your reach out to support@accsyn.com.

Installing Accsyn


The following guide is a short summary of the installation process described in detail here: Accsyn Admin Manual


By now you have a fully working file transfer solution that can be used by external user to receive and send back large file packages at high speed (i.e. an accelerated encrypted FTP server replacement). We are now going to continue and make file transfers submitted by Python scripts to enable an automised workflow.

Configuring Accsyn


API User

This step is optional, you can run the Python API as your own user but we recommend you create a separate user for this purpose:


Installing API

The Accsyn Python API has the same role as the web admin interface - it only controls Accsyn and cannot perform actual file transfers. A file transfer happens p2p between your daemon(server) and a remote client (desktop app or background daemon). This means you can choose any machine that supports Python for running your scripts. In this tutorial we have chosen the file server "alpha" for this purpose, to keep things simple:

ACCSYN_DOMAIN=geoscan

ACCSYN_API_USER=workflow@geoscan.org

ACCSYN_API_KEY=...

Note: You can skip these environment variables and instead enter them when creating the API session later.


Using the API

In your Python code, to use the Accsyn API, prepare a session instance that will be used throughout rest of your scripts to access Accsyn:


import accsyn_api


session = accsyn_api.Session()

 

Note: For detailed information about how to install, setup and use the Python API, refer to: Accsyn Python API Documentation


You are now all setup to start using the Accsyn API within your Python scripts.


Inviting the subcontractor, sharing initial data and sending package


Within your organization, when a subcontractor is assigned to a data analysis task, a Python script is triggered that shares the dataset folder and sends a starter file package so they can start working.


Prerequisites

Let's assume the following:


Inviting the user to Accsyn

If the freelancer have not been using Accsyn before at domain "geoscan", they have to be invited:

u = session.find_one("user WHERE code=jennifer@gda.com")

if u is None: u = session.create("user",{"code":"jennifer@gda.com","clearance":"user"})

Note: We assign 'user' clearance meaning that she will only have access to their default home share (@<default share>/accsyn/<E-mail>/ folder) and additional shares granted access to. Assign 'employee' access to grant read&write access to entire projects share.


Creating/loading the share

A share in Accsyn is similar to a FTP account on a FTP server - a directory on a root share/volume that one or more specific users have access to through ACLs. We put the share at root of film project folder, this way we can easily share further shots/assets later using the same workarea to the same freelancer or other freelancers.

Note: Even if share resides in the project root, it does NOT mean that user have access to entire project. Access is later granted by defining ACL:s on subfolders within the share.


The share "proj002-outsourcing" might already exist, so we attempt to first load it and then create if it does not exist:

share = session.find_one("share WHERE code=proj002-outsourcing")

if share is None: share = session.create("share", {"code":"proj002-outsourcing","parent":"production","path":"proj002"})

A dict containing share data is returned, we use the 'id' key when referencing the share from now on.   Note how share paths are relative to "production" root share location, no need to give an absolute path.


Next we need to give subcontractor read access to the dataset, information about the task and upload folder. For each  check if an ACL exists and creates otherwise:

p_dataset="DATASET_001_1203"

acl_source = session.find_one("acl WHERE (ident=user:{0} AND target=share:{1} AND path={2})".format(u['id'], share['id'], p_dataset))

if acl_source is None: acl_source = session.create("acl",{"ident":"user:{0}".format(u['id']),"target":"share:{0}".format(share['id']),"path":p_dataset,"read":True,"write":False})


p_taskref="TO_VENDORS"

acl_ref = session.find_one("acl WHERE (ident=user:{0} AND target=share:{1} AND path={2})".format(u['id'], share['id'], p_taskref))

if acl_ref is None: acl_ref = session.create("acl",{"ident":"user:{0}".format(u['id']),"target":"share:{0}".format(share['id']),"path":p_taskref,"read":True,"write":False})




Finally we give read & write access to "FROM_VENDORS/jennifer@gda.com" work folder:

p_work="FROM_VENDORS\jennifer@gda.com"

acl_work = session.find_one("acl WHERE (ident=user:{0} AND target=share:{1} AND path={2})".format(u['id'], share['id'], p_work))

if acl_work is None: acl_work = session.create("acl",{"ident":"user:{0}".format(u['id']),"target":"share:{0}".format(share['id']),"path":p_work,"read":True,"write":True})


From this point and on, the subcontractor can download dataset and task information using Accsyn desktop app (or browser for <5GB files). She can also upload material back to "FROM_VENDORS\jennifer@gda.com" folder.


(Optional) Configure subcontractor to automatically receive and store files with folder structure preserved

To further enhance the workflow, Accsyn supports setting up the receiving end so all files lands with the same folder structure as on server - a local share mapping:

From now on, packages sent to subcontractor will start immediately saving files to, given the example in this tutorial: "E:\WORK\GEOSCAN\Production\proj002\DATASET_001_1203".


Sending dataset to subcontractor

In the next step we collect all files needed for subcontractor to do their work, this enables subcontractor to immediately receive the material with needing to ask for it or start manual download.

Compiling and sending file package:

job = session.create("job",{

   "code":"proj002_DATASET_001_1203_outsource_jennifer@gda.com",

   "tasks":{

      "0":{

         "source":"share=proj002-outsourcing/DATASET_001_1203",

         "dest":"jennifer@vfxhouse.com:proj002\DATASET_001_1203"

      },

      "1":{

         "source":"share=proj002-outsourcing/TO_VENDORS",

         "dest":"jennifer@gda.com:TO_VENDORS"

      },

   }

})

Notice how also supply a relative path at receiver end, this way files gets delivered with the same folder structure and we can easily remap paths when project files arrives back. (Will be ignored if the setup the optional local share mapping above).

The freelancer, Jennifer, will receive an E-mail that a package is pending and can download it using the Accsyn desktop app. We assume she downloads it to the following local work path:

E:\WORK\GEOSCAN\Production

Resulting in the following files:

E:\WORK\GEOSCAN\Production\proj002\DATASET_001_1203

E:\WORK\GEOSCAN\Production\proj002\TO_VENDORS


Have freelancer send back results


Summary

The standard way of doing this is to have subcontractor launch Accsyn desktop app and upload result back to share "proj002-outsourcing" folder "FROM_VENDORS/jennifer@gda.com". In this tutorial we are going to be a bit more advanced, allowing subcontractor to build a Python script that collects the data for upload and submits the transfer job by a push of a button within app.


Hint: If you do not wish to do Python scripting at client end, you can instead utilise the Accsyn Publish workflow - allows users to publish results back into correct directories at production servers with workflow integrations.


We assume  the subcontractor is familiar to the Accsyn Python API and have set the proper environment variables - ACCSYN_DOMAIN, ACCSYN_API_USER and ACCSYN_API_KEY ( Can also be provided inline when creating the Accsyn API session).


Subcontractor task would be to write a small Python script that collects and sends back the work done to GeoScan, that is followed by a post process call at the server that publishes result to production management software.


Compile and send the job

First, we detect and store all dependencies into an list, together with the path prefix of project directory. How that is done in analysis app is left out:

project_dir="E:\\WORK\\GEOSCAN\\Production\\proj002"

share_name="proj002-outsourcing"

task_name="DATASET_001_1203"

files=[

   "FROM_VENDORS\\jennifer@gda.com\\cleaned_set.dat",                           # An asset

   "FROM_VENDORS\\jennifer@gda.com\\proj002_DATASET_001_1203.analyze",          # The project file

   "FROM_VENDORS\\jennifer@gda.com\\output\\proj002_DATASET_001_1203_exported.dat",      # The output

]

Note: if we were sending files back into other share folders, like for DATASET_001_1203/, job submit below will fail as she does not have write access to these folders.


Now we have all we need to send back the files:

import accsyn_api

session = accsyn_api.Session()


tasks = {}


n = 0

for p in files:

   tasks[str(n)] = {"source":"%s\\%s"%(project_dir,p),"destination":"acmefilme:share=%s:%s"%(share_name,p)}

   n += 1


job = session.create("job",{

   "code":"%s_output"%(task_name),

   "tasks":tasks

})


The job will automatically start if Jennifer has Accsyn desktop client running and enabled.


Creating the post process script at server for ingestion

As a last step, we want to have the production management system updated with the results - change status of task. To achieve this we first configure a hook in Accsyn:

\\alpha\production\workflow\accsyn_job_done.py ${PATH_JSON_INPUT}


The Accsyn server (for default share, "projects" in this case) will now run this script with path to job JSON data when all files have arrived. 

accsyn_job_done.py:

#!/usr/local/bin/python3

# Accsyn Hook example Python 2/3 script for post-processing a subcontractor upload 


import json,sys,os


import accsyn_api


def generic_print(s):

    try:

        if ((3, 0) < sys.version_info):

            # Python 3 code in this block

            expr = """print(s)"""

        else:

            # Python 2 code in this block

            expr = """print s"""

        eval(expr)

    except:

        pass



def publish(user=None, comment=None, filename=None):

    ''' Do something with data uploaded by user. '''

    generic_print("User {0} published '{1}' ,comment: {2}".format(user, filename, comment))


if __name__ == '__main__':


    p_input = sys.argv[1]

    data = json.load(open(p_input, "r"))

    generic_print("Publish hook incoming data from user %s: %s"%(data['user_hr'], json.dumps(data, indent=3)))


    # Find the output amongst files

    for p in [d["source"]["path_abs"] for d in data["tasks"].values()]:

        if p.lower().endswith("_exported.dat"):

            publish(user=data["source_hr"].split(":")[1], comment="Subcontractor publish", filename=p)


Note: How you write the "publish" function is depending on which project management system you have, we leave that to you.


Wrapping it all up

These scriptlets are just rough guidelines, you can implement this in any way you want. Our goal with this tutorial has been to show how much more powerful an API scriptable file delivery system can be compared to an traditional FTP/Dropbox/WeTransfer solution combined with manual efforts of keeping an remote subcontract in sync with workflow