Job JSON specification

This document provides accsyn job JSON  (JavaScript Object Notification) payload examples and describes best practices for using the accsyn API or CLI.

Job JSON specification

When is the job JSON used?

accsyn jobs are internally submitted in JSON format, from desktop app, web and CLI.

When submitting a job through the API our using a file with CLI, the correct job JSON must be provided as specified throughout the rest of this document.

Hint: the job JSON payload can be inspected from within the accsyn Desktop app, by clicking the "JSON" button beneath the green submit button in NC mode:

Structure of transfer job JSON payload

The general structure:

{

    .. job attributes ..,

    "tasks": .. list or JSON ..,

    "settings": .. JSON ..  # Optional

    "metadata": .. JSON ..  # Optional

}

• Job attributes; (optional) Typically name of job, queue, initial status and so on. For detailed information, please refer to the accsyn Python API documentation.

• Tasks: List or dictitionary of tasks - files and directories to transfer, see party and path notation below.

• Settings; (optional) Additional job settings, see accsyn Settings documentation for a complete listing 

• Metadata; (optional) Additional job metadat

Transfer tasks

A task is a file or folder to transfer, a job can contain multiple tasks but must at least contain one. accsyn support different methods of supplying tasks with in the payload:

A) Relaxed structure of transfer job payload (single file):

{

    ..,

    "source": "<party>:<path>",

    "destination": "<party>:<path>",

}

B) A list of one or more tasks:

{

    ..,

    "tasks":{[

        {

            .. task attributes ..,

            "source": "<party>:<path>",

            "destination": "<party>:<path>"

        },

        ..

    ]}

}

C) Or as a dictionary (the internal accsyn notation):

{

    ..,

    "tasks":{

        "0":{

            ..,

            "source": "<party>:<path>",

            "destination": "<party>:<path>",

        ], 

        "1": {

            ..

        },

        ..

    }

}

Notes: 

• Nested tasks are allowed, these are defined by a "tasks" sub key. It is used with compute jobs, see examples below.

• "0" and "1" above are called task "uri":s and must be unique (within the tasks locally at that level).  Tasks are also assigned an unique ID that can be used to further modify the job.

Source and Destination notation

An accsyn source/destination should be passed on as a combination of a party and a path similar to standard rsync and scp notation, using this template:

<party>:<path>

• Party; Identifies the sending or receiving endpoint entity.

• Path; The path to the file or directory to transfer, either on accsyn path notation or an absolute path.

For a detailed breakdown of parties and path, please refer to the accsyn Python API documentation. 

Example party definitions:

• myworkspace; Denotes the workspace side as source or destination - the main site (hq), resolves to the Server hosting the file (through volume) identified by the path.

• john@user.com; Denotes a user as a source or destination, they must have a (running) client (Desktop App or User Server instance) that can be resolved by accsyn 

• site=london; Denotes the site with (unique) API "code" identifier "london".

• client=6611fbca3f8c4d3e7a3b678a; Denotes an explicit client, for example if a user are running multiple clients.

Path examples:

• C:\Users\john\Desktop\image.png; A local path at user end.

• ~/Document.pdf; A relative path on user's Home share at workspace.

• /Volumes/projects/reference.tif; An absolute path on a volume, can be used with workspace/site parties if they match configured volume paths.

• volume=projects/reference.tif ; Same notation,  but references the volume "projects" by its unique API "code" attribute and leaving accsyn to resolve the absolute path using configured prefix for server platform. This is also called the accsyn path notation.

• volume=(default)/reference.tif ; Same notatation but specifying the default volume directly.

• share=myproject/assets.zip ; Relaxed share definition - unspecified share type, resolves to the volume, shared folder, home or collection "myproject" by its unique API "code" attribute. 

• folder=6734b3ca8c3592a922bdb0de/TO_ACME/source.rar; File is located in shared folder identified by the explicit it, in subfolder "TO_ACME".

• home=john@user.com/UPLOAD/test.abc; File is located at Home share john@user.com , in subfolder "UPLOAD".

• myproject/reference.jpg ; Assume file paths being relative default volume, equivalent to share=projects/myproject/reference.jpg.  

• racing2019_grade/test.abc; Used in conjunction with user as target, deliver file into relative folder "racing2019_grade" at user end.

Notes:

• The "default volume" is a the volume having the default attribute set to true, and is assigned the first volume created for a workspace. At least one default volume must be assigned within an accsyn workspace.

• • A folder cannot be given as destination unless a "/" (or "\" for Windows) is added. For example downloading a file "x.jpeg" to destination "/Volumes/nas/TEMP" will store the file as "TEMP", not inside folder TEMP. Correct destination notation in this case are: "/Volumes/nas/TEMP/", or even better: "/Volumes/nas/TEMP/x.jpeg". 

  • Destination paths can be left out if other party is a site or a user's locally mapped share, this is called "path mirroring", and is suitable for keeping servers and/or workstations in sync when it comes to file structure.

  • If party is omitted, accsyn interprets this as the workspace party - file is to be sent to or from hq.

Client resolve

When submitting a job, accsyn tries to resolve a client-server combo based on the party and path given. The rules vary depending on the type of party:

• Workspace; Here accsyn tries to resolve the server that serves the volume pointed out by the path, or indirectly by paths on a shared folder or collection.

• User; accsyn tries to find a file transfer client belonging to the user, this might be the default client created when using the desktop app or a user server. For more information, see Hosts.

• Site; accsyn tries to find a site server, that serves the volume pointed out by the path at the remote site, or indirectly by paths on a shared folder or collection.

If no server or client can be resolved, an error will be given with appropriate feedback. Once resolved, the resolve remains static which means that if a new server endpoint is deployed or the user launches another client, the mapping is NOT updated - a new transfer job must be submitted.

Note:  The accsyn Python API does not provide a built-in p2p ASC client - it can only be used to control transfers.

User permissions - upload and download

A user is only allowed to access files and folders on an accsyn site (workspace/main hq premises or remote site) given explicit access through ACLs.

When downloading or uploading a file, only the user owning the API/CLI session are allowed to specify local absolute paths (e.g. C:\Users\ John\Downloads), to send a file to user you must create a Delivery were the user then chooses were to download (or what to upload if it is an upload request). For more information, see Delivery.

There is an exception to this: if the user has mapped a share locally and has given explicit write permissions to it, an elevated user are allowed to push files to the user's client using mirror path option or explicit path notation (see examples below). Same goes for upload - an elevated user can pull files from a remote user's computer and upload them to a workspace volume if the user has given explicit read access to the corresponding locally mapped share. For more information on how to setup locally maped shares, see Hosts.

User permissions - site transfers

When transferring files between sites, the user has to have the admin role or employee with full access to the involved volume(s).

Example transfer job snippets

Workspace drop-off

Prerequisites:

• The user needs to have a Home share and write access to it (entire share or a subfolder)

• The user needs to have a registered accsyn client (Desktop App or User Server) instance.

Drop off the file "Prototype.zip", short simplified single task notation:

{

    "source":"/home/Adrian/pitches/Prototype.zip",

    "destination":"~"

}

• "~" resolves to the home share.

• No destination path were given, leave for accsyn to resolve destination folder. This includes adding eventual delivery dropoff date subfolder (default on the form: YYYYMMDD)

With target subfolder and providing a name, letting accsyn append the source filename to destination path:

{

    "source":"/home/Adrian/pitches/Prototype.zip",

    "destination":"~/preproduction/",

    "name": "Prototype send"

}

With a different filename for destination:

{

    "source":"/home/Adrian/pitches/Prototype.zip",

    "destination":"~/Prototype_260703.zip"

}

User share download

Prerequisites:

• The user needs to have a Home share and read access to it (entire share or a subfolder)

• The user needs to have a registered accsyn client (Desktop App or User Server) instance.

Download folder "Material" from workspace home share:

{

    "source":"~/Material",

    "destination":"/Users/Adrian/Downloads/"

}

• The destination party client will resolve to the most recently online client.

Download two files, with an explicit client specified:

{

    tasks:[

        {

            "source":"~/Material",

            "destination":"client=6a479e243bed6009ac9d6763:/Users/Adrian/Downloads/"

        },

        {

            "source":"~/Legal",

            "destination":"client=6a479e243bed6009ac9d6763:/Users/Adrian/Downloads/"

        },

    ]

}

Specify target endpoint by hostname

{

    "source":"~/SketchesFinal",

    "destination":"emma@compers.com@PCLocal-001:/Users/Emma/Downloads/"

}

• Be aware of ambiguity in case two computers share the same name!

Operator* upload

Sync the folder "deployment" to a subfolder on accsyn workspace storage:

{

    "source":"D:/dev/pipeline/build/deployment",

    "destination":"_PIPEINE/live/deployment"

}

Notes:

• A relative path is given as destination, this resolves to the default* volume on workspace.

• The operator needs to have write access to the volume

Operator folder download

Download the file "Fireflies_001" from volume "asset" going into High priority queue, storing locally:

{

    "source":"volume=assets/Flies/Fireflies_001",

    "destination":"D:/work/ASSETS/Fireflies_001",

    "queue":"High"

}

Notes:

• Term operator means an elevated accsyn user - administrator or employee with read access to (default) volume

• The workspace party is omitted here, this is allowed since the other user party is clearly stated and no ambiguity exists when it comes to the source party.

• Neither the source volume is given here as source, just a relative path. When no source volume or share is given, the default volume is assumed being the source.

Corresponding full expanded syntax for reference, assuming the workspace code/name is "thecompany":

{

    "source":"thecompany:volume=assets/Flies/Fireflies_001",

    "destination":"john@thecompany.com:D:/work/ASSETS/Fireflies_001"

}

User share upload

Upload folder "/Users/john/Desktop/delivery" to Shared Folder "thefilm" into subfolder "from_john/20180413":

{

    "source":"/Users/john/Desktop/new_scans", 

    "destination":"folder=thefilm/from_john/20180413/"

}

Note: Shares are identified wither by their unique ID, or by their unique API "code" identifier.

Operator download to site

Sync a folder on volume "projects" @ main site(default: "hq") to site "berlin":

 {

    "source":"share=projects/thefilm/SCENES", 

    "destination":"site=berlin"

} 

• The operator needs to have read and write permissions to the volume

• No destination path is given, accsyn will mirror the path structure on receiving end.

Operator transfer between sites

Corresponding transfer of a folder from site "cloud" to site "berlin":

{

    "source":"site=cloud:share=render/got/sc01/sh01/render/got_sc01_sh01_comp_v012",

    "destination":"site=berlin"

}

Operator push

Push(download) a folder to locally mapped share "thefilm" at remote user, storing in a different folder:

{

    "source":"share=thefilm/_OUTSOURCING/Paint_and_cleanup-Compers-260703",

    "destination":"emma@compers.com:share=thefilm/_FROM_THECOMPANY/"

}

• The user must have mapped the share locally (configured through app or through ACCSYN_*_PATH envs) , write access enabled.

Operator pull

Pull a file from remote locally mapped share "thefilm" back to workspace storage:

{

    "source":"emma@compers.com:share=thefilm/DELIVERY/WIP-260703.zip",

}

• No destination is given, accsyn will resolve to workspace default volume and also apply mirrored paths.

Metadata

Sync a folder on share "projects"  from site "berlin" back to hq, deleting files that does not existing on receiving end. Supply metadata that can be picked up by hooks:

{"tasks":[{

     "source":"site=berlin:projects/racing2019_grade/davinci_files",

     "destination":"myorg",

     "metadata":{"app":"davinci_resolve"}

}],

    "metadata":{"artist":"malcolm"},

    "settings":{"transfer_mode":"onewaysync"}

}

• Settings are always supplied as strings, please refer to Settings documentation.

Skip existing files

Upload file "final_export.mov", at share "projects"  from user to share "racing2019_grade", but not overwriting it if exists and size or modification date differ:

{"tasks":[{

    "source":"E:\racing2019_grade\davinci_files\final_export.mov",

    "destination":"mycompany:share=racing2019_grade/FROM_EDIT/final_export.mov",

    "settings":{"transfer_ignore_existing":"file"}

}]}

Exclude files

Upload a large folder, excluding all files ending with "tmp" and is only numbers:

{"tasks":[{

    "source":"F:\BIGGIE",

    "destination":"mycompany:share=projects\__UPLOADS\BIGGIE",

    "settings":{"transfer_exclude":"*tmp\,re('[0-9]')"}

}]}

Note: multiple exclude statements are separated by an escaped comma - \, . This means that an escaped comma cannot be used in exclude expressions.

Priorities

Tasks (files) can have different priorities, enabling pre-delivery of some important files. Here is an example of downloading two files with the PDF prioritised:

{"tasks":[{

    "source":"share=bidding/LFM/brief_v001.pdf",

    "destination":"lisa@"mail.com:/Volumes/media/_TO_BID,

    "priority":999

},{

    "source":"share=bidding/LFM/material.rar",

    "destination":"lisa@"mail.com:/Volumes/media/_TO_BID,

}]}

In this case, the file bried_v001.pdf will be sent first, then material.rar. accsyn priorities range from 1000 (highest) to 1 (lowest).

File sequences

accsyn supports sending a subset of a numbered file sequence, this is handy when you do not want to send an entire directory with a huge amount of files but instead want to do a selection.

Transfer a part of a file sequence from one site to another, mirrored paths (requires VPN direct connection or both server setup with NAT port forwarding of accsyn protocol ports):

{"tasks":[{

    "source":"site=berlin:myproj/images/movie.%04d.jpg[100-167]",

    "destination":"site=london",

}]}

Placeholder jobs

In some situations, a job needs to be created in beforehand, to enable tasks to be added shortly after. To achieve this, a placeholder job can be submitted:

{"tasks":[{

    "source":"site=cloud:share=projects/nofile",

    "destination":"site=hq",

    "status":"excluded"

}]}

The job will immediately be set to done, with no files actually transferred.

Note: This example relies on the main site name is the default - "hq", the expression can also be replaced with your workspace code/domain (mycompany in these examples).

Best practices and limitations

Source and destination party

accsyn is a p2p file transfer protocol which means that each transfer job can only have one unique source and one unique destination party.

A source or destination party can be:

• The workspace(organization); on-prem servers at main site (hq), running the accsyn daemon app in server mode. 

• A user; Identified by the Email, running the accsyn desktop app or daemon app in user server mode.

• A site; Identified by a unique name, running the accsyn daemon app in server mode. 

• Web browser; For downloads and uploads using web browser.

For the workspace party, source or destination files can reside on multiple volumes within the same job:

{

  "code":"My download",

  "tasks":[

     {

       "source":"share=share1/file.001",

       "destination":"john@user.com/X:/download"

     },{

       "source":"share=share2/file.002",

       "destination":"john@user.com/X:/download"

     },

  ]

}

Example: Transfer one file from share1 and another from share2, that can reside on different servers.

Path specification

Job count limit

Although now limits on the amount of jobs are enforced by accsyn, when amount reaches 500+ a substantial degrade of performance occurs and UI will become immeasurable. 

The recommended approach is to reduce the amount of jobs and instead have multiple tasks within each job. 

Giving an example were the API in an automatic workflow submits a sync job for each file updated and everyday, if thousands of files are updated accsyn will soon reach its limits and the job listing in desktop app /web app will be overflowed.

A better approach would be to let the API load a daily sync job for each source-destination pair (create if not exists) and add tasks to that job instead. This would reduce the amount of jobs drastically, and will make job listing much more readable when it comes to find other important jobs that otherwise would totally drown.

Large jobs with a lot of files

accsyn transfers files using the same algorithm as *NIX rsync which means that a list of files with size and modification dates are sent to receiving end in order to determine which files need to be sent.

For very large file transfers containing a lot of smaller files in deep lengthy folder structures, accsyn might run out of RAM during file transfer init and in those cases it is recommended to split a job in multiple tasks. 

For example when doing project backup with accsyn, instead of sending the entire root share or directory - send each project directory as individual tasks and set "task_bucketsize" setting to "1":