Transfer job specification

Rev 1.003, 20.07.21

Summary

This document describes best practices and JSON (JavaScript Object Notification) specifications for using the Accsyn API or CLI to submit a transfer job.

Best practices


Source and destination party

Accsyn is a p2p file transfer protocol which means that each transfer job can only have one unique source and one unique destination.

A source or destination can be:

  • The organization; on-prem servers at main site (hq), running the Accsyn daemon app.

  • A user; Identified by the Email, running the Accsyn desktop app or daemon app.

  • A site; Identified by a unique name, running the Accsyn daemon app.

  • Web browser; For downloads and uploads using web browser.


For the organization party, source or destination files can reside on multiple (root) shares within the same job:

{

"code":"My download",

"tasks":[

{

"source":"share=share1/file.001",

"destination":"john@user.com"

},{

"source":"share=share2/file.002",

"destination":"john@user.com"

},

]

}

Example: Transfer one file from share1 and another from share2, that can reside on different servers.


Job count limit

Although now limits on the amount of jobs are enforced by Accsyn, when amount reaches 500+ a substantial degrade of performance occurs and UI will become immeasurable.

The recommended approach is to reduce the amount of jobs and instead have multiple tasks within each job.

Giving an example were the API in an automatic workflow submits a sync job for each file updated and everyday, if thousands of files are updated Accsyn will soon reach its limits and the job listing in desktop app /web app will be overflowed.

A better approach would be to let the API load a daily sync job for each source-destination pair (create if not exists) and add tasks to that job instead. This would reduce the amount of jobs drastically, and will make job listing much more readable when it comes to find other important jobs that otherwise would totally drown.


Large jobs with a lot of files

Accsyn transfers files using the same algorithm as *NIX rsync which means that a list of files with size and modification dates are sent to receiving end in order to determine which files need to be sent.

For very large file transfers containing a lot of smaller files in deep lengthy folder structures, Accsyn might run out of RAM during file transfer init and in those cases it is recommended to split a job in multiple tasks.

For example when doing project backup with Accsyn, instead of sending the entire root share or directory - send each project directory as individual tasks and set "task_bucketsize" setting to "1":



{

"code":"Daily backup",

"source":"share=raid01/projects",

"destination":"site=backup"

}




=>

{

"code":"Daily backup",

"tasks":[

{

"source":"share=raid01/projects/PROJ001",

"destination":"site=backup"

},{

"source":"share=raid01/projects/PROJ002",

"destination":"site=backup"

},

],

"settings":{"task_bucketsize":"1"}

}

Job JSON specification


When is the job JSON used?


Accsyn jobs are internally submitted in JSON format, even from the desktop app.

When submitting a job through the API our using a file with CLI, the correct job JSON must be provided as specified throughout the rest of this document.


Structure of job JSON

The general structure of job JSON:

{

.. job attributes ..,

"tasks": .. list or JSON ..,

"settings": .. JSON ..

"metadata": .. JSON ..

}


  • Job attributes; (optional) Typically name of job (attribute name: "code"). The name of job is auto generated using filename of first task if not given.

  • Tasks: List of tasks - files and directories to transfer, see party and path notation below.

  • Settings; (optional) Additional job settings, see below for a complete listing.

  • Metadata; (optional) Additional job settings.

Note: tasks can be omitted if just a single file, and be replaced with job attributes "source" and "destination".


Accsyn party and path notation


The Accsyn job JSON requires "tasks" entry containing on ore more source & destination definitions, Accsyn have adopted the rsync notation on how to define a target and a path:

<party>:<path>

Where party can be:

  • <domain>; Main premises/hq servers, for example domain "myorg" with Accsyn URL https://myorg.accsyn.com.

  • <username>; A user, for example "john@user.com".

  • <site=id|code>; A remote site server, for example "stockholm".

And path can be:

  • C:\Users\john\Desktop\image.png; A local path at user end.

  • /Volumes/projects/reference.tif; An absolute path on a root share.

  • share=projects/reference.tif ; Same notation, but references the share "projects" and leaving Accsyn to complete path with configured prefix for server platform. This is also called the Accsyn path notation.

  • share=john@user.com/UPLOAD/test.abc; File is located at home user share john@user.com , in subfolder "UPLOAD".

  • myproject/reference.jpg ; Assume file paths being relative default root share, equivalent to share=projects/myproject/reference.jpg.

  • racing2019_grade/test.abc; Used in conjunction with user as target, deliver file into relative folder "racing2019_grade" at user end.


Notes:

    • Destination paths can be left out if other party is a site or an employee, this is called "path mirroring", and is suitable for keeping servers and/or workstations in sync when it comes to file structure.

    • If party is omitted, Accsyn interprets this as the domain party - file is to be sent to or from hq.



Some job JSON examples


Send a file to a user

Deliver the file "/projects/lfm/114/290/lfm_114_290.jpg" (requires a share to be configured /projects) to user "john@user.com", storing at relative path "projects/lfm/114/290/". Short format, single task:

{

"source":"myorg:/projects/lfm/114/290/lfm_114_290.jpg",

"destination":"john@user.com:projects/lfm/114/290/lfm_114_290.jpg"

}


Upload to home share

Upload folders "/Users/john/Desktop/delivery" & "quicktimes" to user john@user.com's home share, have job reside in queue "low_prio", with task identifiers:

{

"tasks":{

"0":{

"source":"/Users/john/Desktop/delivery",

"destination":"share=john@acmefilm.co.uk"

},

"1":{

"source":"/Users/john/Desktop/quicktimes",

"destination":"share=john@acmefilm.co.uk"

}

},

"queue":"low_prio"

}


Download from user share

Download file "20180413/script_v001.pdf" from user share to local folder "C:\Users\John\Download\Accsyn":

{"tasks":[{

"source":"myorg:20180413/script_v001.pdf",

"destination":"C:\Users\John\Download\accsyn\"

}]}

Notes:

    • Leaving out the trailing slash will copy the file to file "Accsyn" on the receiving side.

    • Tasks can be supplied as a list, leaving to Accsyn to apply task identifiers.


Upload to share

Upload folder "/Users/john/Desktop/delivery" to project share "thefilm" into subfolder "from_john/20180413":

{"source":"/Users/john/Desktop/new_scans", "destination":"share=thefilm/from_john/20180413/"}


Push to site

Sync folder "thefilm/010" on share "projects" to site "stockholm01", paths mirrored (default when site are involved):

{ "source":"share=projects/thefilm/010", "destination":"site=stockholm" }


Upload from site

Download rendered folder "projects/got/sc01/sh01/render/got_sc01_sh01_comp_v012" from site "google" to site "stockholm":

{"source":"site=google:share=projects/got/sc01/sh01/render/got_sc01_sh01_comp_v012","destination":"site=stockholm"}


Metadata

Upload folder "racing2019_grade/davinci_files", at share "projects" from site "stockholm" back to hq, with metadata (can be picked up by post process hooks):

{"tasks":[{

"source":"site=stockholm:projects/racing2019_grade/davinci_files",

"destination":"myorg",

"metadata":{"app":"davinci_resolve"}

}],

"metadata":{"artist":"malcolm"}

}


Skip existing files

Upload file "final_export.mov", at share "projects" from user to share "racing2019_grade", but not overwriting it if exists and size or modification date differ:

{"tasks":[{

"source":"E:\racing2019_grade\davinci_files\final_export.mov",

"destination":"myorg:share=racing2019_grade/FROM_EDIT/final_export.mov",

"settings":{"transfer_ignore_existing":"file"}

}]}


Exclude files

Upload a large folder, exluding all files ending with "tmp" and is only numbers:

{"tasks":[{

"source":"F:\BIGGIE",

"destination":"myorg:share=projects\__UPLOADS\BIGGIE",

"settings":{"transfer_exclude":"*tmp\,re('[0-9]')"}

}]}

Note: multiple exclude statements are separated by an escaped comma - \, . This means that an escaped comma cannot be used in exclude expressions.




Notes:

    • The prefix "share=" can be omitted if the party is not a user, i.e. sending to/from the organization hq/site.

    • Source/destination can be specified without tasks, for single file/directory transfers. Tasks can be a list instead of a JSON, each transfer task while then be numbered by the order they appear in list.

    • The following scenarios can also be submitted directly from the commandline, for example: "accsyn job create myorg:/projects/lfm/114/290/lfm_114_290.jpg john@user.com:projects/lfm/114/290/lfm_114_290.jpg".

    • If destination exists and is a directory, Accsyn behaves as unix "rsync" - the file ends up beneath the folder.

    • If file exists were Accsyn were supposed to save a folder, transfer will fail.


Job settings


The following job settings can be provided on submit: