accsyn transfer job specification
Summary
This document describes best practices and JSON (JavaScript Object Notification) specifications for using the accsyn API or CLI to submit a transfer job.
Best practices
Source and destination party
accsyn is a p2p file transfer protocol which means that each transfer job can only have one unique source and one unique destination.
A source or destination can be:
The workspace(organization); on-prem servers at main site (hq), running the accsyn daemon app in server mode.
A user; Identified by the Email, running the accsyn desktop app or daemon app in user server mode.
A site; Identified by a unique name, running the accsyn daemon app in server mode.
Web browser; For downloads and uploads using web browser.
For the workspace party, source or destination files can reside on multiple (root) shares within the same job:
{
"code":"My download",
"tasks":[
{
"source":"share=share1/file.001",
"destination":"john@user.com"
},{
"source":"share=share2/file.002",
"destination":"john@user.com"
},
]
}
Example: Transfer one file from share1 and another from share2, that can reside on different servers.
Job count limit
Although now limits on the amount of jobs are enforced by accsyn, when amount reaches 500+ a substantial degrade of performance occurs and UI will become immeasurable.
The recommended approach is to reduce the amount of jobs and instead have multiple tasks within each job.
Giving an example were the API in an automatic workflow submits a sync job for each file updated and everyday, if thousands of files are updated accsyn will soon reach its limits and the job listing in desktop app /web app will be overflowed.
A better approach would be to let the API load a daily sync job for each source-destination pair (create if not exists) and add tasks to that job instead. This would reduce the amount of jobs drastically, and will make job listing much more readable when it comes to find other important jobs that otherwise would totally drown.
Large jobs with a lot of files
accsyn transfers files using the same algorithm as *NIX rsync which means that a list of files with size and modification dates are sent to receiving end in order to determine which files need to be sent.
For very large file transfers containing a lot of smaller files in deep lengthy folder structures, accsyn might run out of RAM during file transfer init and in those cases it is recommended to split a job in multiple tasks.
For example when doing project backup with accsyn, instead of sending the entire root share or directory - send each project directory as individual tasks and set "task_bucketsize" setting to "1":
{
"code":"Daily backup",
"source":"share=raid01/projects",
"destination":"site=backup"
}
=>
{
"code":"Daily backup",
"tasks":[
{
"source":"share=raid01/projects/PROJ001",
"destination":"site=backup"
},{
"source":"share=raid01/projects/PROJ002",
"destination":"site=backup"
},
],
"settings":{"task_bucketsize":"1"}
}
Job JSON specification
When is the job JSON used?
accsyn jobs are internally submitted in JSON format, even from the desktop app.
When submitting a job through the API our using a file with CLI, the correct job JSON must be provided as specified throughout the rest of this document.
Structure of job JSON
The general structure of job JSON:
{
.. job attributes ..,
"tasks": .. list or JSON ..,
"settings": .. JSON ..
"metadata": .. JSON ..
}
Job attributes; (optional) Typically name of job (attribute name: "code"). The name of job is auto generated using filename of first task if not given.
Tasks: List of tasks - files and directories to transfer, see party and path notation below.
Settings; (optional) Additional job settings, see below for a complete listing.
Metadata; (optional) Additional job settings.
Note: tasks can be omitted if just a single file, and be replaced with job attributes "source" and "destination".
accsyn party and path notation
The accsyn job JSON requires "tasks" entry containing on ore more source & destination definitions, accsyn have adopted the rsync notation on how to define a target and a path:
<party>:<path>
Where party can be:
<workspace>; Main premises/hq servers, for example domain "workspace" with accsyn URL https://workspace.accsyn.com.
<username>; A user, for example "john@user.com".
<site=id|code>; A remote site server, for example "berlin".
And path can be:
C:\Users\john\Desktop\image.png; A local path at user end.
/Volumes/projects/reference.tif; An absolute path on a root share.
share=projects/reference.tif ; Same notation, but references the share "projects" and leaving accsyn to complete path with configured prefix for server platform. This is also called the accsyn path notation.
share=john@user.com/UPLOAD/test.abc; File is located at home user share john@user.com , in subfolder "UPLOAD".
myproject/reference.jpg ; Assume file paths being relative default root share, equivalent to share=projects/myproject/reference.jpg.
racing2019_grade/test.abc; Used in conjunction with user as target, deliver file into relative folder "racing2019_grade" at user end.
Notes:
Destination paths can be left out if other party is a site or an employee, this is called "path mirroring", and is suitable for keeping servers and/or workstations in sync when it comes to file structure.
If party is omitted, accsyn interprets this as the domain party - file is to be sent to or from hq.
Following, is a collection of sample job JSON snippets as examples.
Send a file to a user
Deliver the file "/projects/lfm/114/290/lfm_114_290.jpg" (requires a share to be configured /projects) to user "john@user.com", storing at relative path "projects/lfm/114/290/". Short format, single task:
{
"source":"myworkspace:/projects/lfm/114/290/lfm_114_290.jpg",
"destination":"john@user.com:projects/lfm/114/290/lfm_114_290.jpg"
}
Send two files to multiple users
Send out a small preview, that will be written to "preview" subfolder at the user download location, and a large movie written to "movie" subfolder, destined two different users that will be invited if not exists:
{
"tasks":[
{
"source":"myworkspace:/projects/lfm/114/290/lfm_114_290.jpg",
"destination":"preview/lfm_114_290.jpg"
}, {
"source":"myworkspace:/projects/lfm/114/290/lfm_114_290.mov",
"destination":"movie/lfm_114_290.mov"
}
],
"recipients":["john@user.com","mia@"user.com],
"invite" : true
}
Note: Omit the destination if no destination subfolder is required.
Upload to home share
Upload folders "/Users/john/Desktop/delivery" & "quicktimes" to user john@user.com's home share, have job reside in queue "low_prio", with task identifiers:
{
"tasks":{
"0":{
"source":"/Users/john/Desktop/delivery",
"destination":"share=john@acmefilm.co.uk"
},
"1":{
"source":"/Users/john/Desktop/quicktimes",
"destination":"share=john@acmefilm.co.uk"
}
},
"queue":"low_prio"
}
Download from user share
Download file "20180413/script_v001.pdf" from user share to local folder "C:\Users\John\Download\accsyn":
{"tasks":[{
"source":"myworkspace:20180413/script_v001.pdf",
"destination":"C:\Users\John\Download\accsyn\"
}]}
Notes:
Leaving out the trailing slash will copy the file to file "accsyn" on the receiving side.
Tasks can be supplied as a list, leaving to accsyn to apply task identifiers.
Upload to share
Upload folder "/Users/john/Desktop/delivery" to project share "thefilm" into subfolder "from_john/20180413":
{"source":"/Users/john/Desktop/new_scans", "destination":"share=thefilm/from_john/20180413/"}
Download to site
Sync a folder on share "projects" @ main site(default: "hq") to site "berlin", paths mirrored (default when site are involved):
{ "source":"share=projects/thefilm/010", "destination":"site=berlin" }
Note: Site can have explicit permissions that controls which file operations are allowed.
Upload from site
Upload a folder from site "cloud" to main site(default: "hq")
{"source":"site=google:share=projects/got/sc01/sh01/render/got_sc01_sh01_comp_v012","destination":"myworkspace"}
Transfer between sites
Upload a folder from site "cloud" to site "berlin":
{"source":"site=cloud:share=projects/got/sc01/sh01/render/got_sc01_sh01_comp_v012",
"destination":"site=berlin"}
Metadata
Upload a folder on share "projects" from site "berlin" back to hq, with metadata that can be picked up by hooks:
{"tasks":[{
"source":"site=berlin:projects/racing2019_grade/davinci_files",
"destination":"myorg",
"metadata":{"app":"davinci_resolve"}
}],
"metadata":{"artist":"malcolm"}
}
Skip existing files
Upload file "final_export.mov", at share "projects" from user to share "racing2019_grade", but not overwriting it if exists and size or modification date differ:
{"tasks":[{
"source":"E:\racing2019_grade\davinci_files\final_export.mov",
"destination":"myworkspace:share=racing2019_grade/FROM_EDIT/final_export.mov",
"settings":{"transfer_ignore_existing":"file"}
}]}
Exclude files
Upload a large folder, excluding all files ending with "tmp" and is only numbers:
{"tasks":[{
"source":"F:\BIGGIE",
"destination":"myworkspace:share=projects\__UPLOADS\BIGGIE",
"settings":{"transfer_exclude":"*tmp\,re('[0-9]')"}
}]}
Note: multiple exclude statements are separated by an escaped comma - \, . This means that an escaped comma cannot be used in exclude expressions.
Priorities
Tasks (files) can have different priorities, enabling pre-delivery of some important files:
{"tasks":[{
"source":"/media/disk/brief_v001.pdf",
"destination":"lisa@"mail.com,
"priority":999
},{
"source":"/media/disk/material.rar",
"destination":"lisa@"mail.com,
}]}
In this case, the file bried_v001.pdf will be sent first, then material.rar. accsyn priorities range from 1000 (highest) to 1 (lowest).
File sequences
accsyn supports sending a subset of a numbered file sequence, this is handy when you do not want to send an entire directory with a huge amount of files but instead want to do a selection.
Transfer a part of a file sequence from one site to another, mirrored paths (requires VPN direct connection or both server setup with NAT port forwarding of accsyn protocol ports):
{"tasks":[{
"source":"site=berlin:/Volumes/NAS/myproj/images/movie.%04d.jpg[100-167]",
"destination":"site=london",
}]}
Notes:
The prefix "share=" can be omitted if the party is not a user, i.e. sending to/from the workspace hq/site.
Source/destination can be specified without tasks, for single file/directory transfers. Tasks can be a list instead of a JSON, each transfer task while then be numbered by the order they appear in list.
The following scenarios can also be submitted directly from the commandline, for example: "accsyn job create myworkspace:/projects/lfm/114/290/lfm_114_290.jpg john@user.com:projects/lfm/114/290/lfm_114_290.jpg".
If destination exists and is a directory, accsyn behaves as unix "rsync" - the file ends up beneath the folder.
If file exists were accsyn were supposed to save a folder, transfer will fail.
Placeholder jobs
In some situations, a job needs to be created in beforehand, to enable tasks to be added shortly after. To achieve this, a placeholder job can be submitted:
{"tasks":[{
"source":"site=cloud:share=projects/nofile",
"destination":"site=hq",
"status":"excluded"
}]}
The job will immediately be set to done, with no files actually transferred.
Job settings
The following job settings can be provided on submit: