Datasource sync types and statuses
Learn about the various Datasource sync types and status available in Onna
# Datasource synchronization types
Onna Datasources have various attributes and properties: owners, authentication credentials, and how the data is synchronized.
All cloud based Datasources, such as Dropbox or Slack, have three sync types available:
- auto: continuously sync
Auto-sync means that Onna will perform a full sync first and will keep the Datasource and Onna in mirrored sync. Any deletions from the Datasource will be deleted in Onna, as well. - one: one-time sync
One-time is a one-way sync that collects information only once. - arch: archive, like auto but without deletions
Auto-sync and Archive means that Onna will perform a full sync first and will continuously add any new files generated at the Datasource. The sync type does not delete files deleted from the Datasource
Folder Datasources only support one-time syncs.
# Datasource synchronization statuses
You keep track of the state of a Datasource's sync status with the following values:
- pending: sent to processing pipeline. The Datasource is waiting to be processed.
- created: after initial creation, not configured yet
- syncing: synchronization is in progress
- synced: synchronization is finished
- error*: synchronization errored and will not be retried
- failed: synchronization failed and will be retried
- paused: synchronization is paused
- resuming: synchronization is picking up from where it was paused
- invited: Invited, sharing a Datasource with another user
A Datasource with an auto sync type will be scheduled to run periodically, every 20 to 30 minutes.
# Endpoints for Datasource-specific tasks
For the endpoints listed below, you can call them by appending to the Datasource URL.
@frontsearch: performs a search operation using the Datasource as the context
@sendToCompute: sends a resource to the processing pipeline.
You can GET @sendToCompute to force a resource in a Datasource to enter the processing pipeline.
f"https://enterpise.onna.com/api/{container}/{account}/workspaces/{workspace_name}/{datasource_id}/{resource_id}/@sendToCompute?force=True"
This HTTP GET will force the Datasource to collect data
f"https://enterpise.onna.com/api/{container}/{account}/workspaces/{workspace_name}/{datasource_id}/@sendToSpyder?force=True"
@refreshServiceCredentials: tries to refresh credentials that are stored in your "wallet" and associated with this Datasource.
This POST will try to refresh the credentials of a Datasource. Your auth token, and specific headers are part of the payload:
f"https://enterprise.onna.com/api/{container}/{account}/workspaces/{workspace_name}/{datasource_id}/@refreshServiceCredentials?force=True"
data=json.dumps({"X-Auth-Secret": ..., "Authorization": "Bearer ..."})
2
@getAllChildren: gets all children of a Datasource.
f"https://enterpise.onna.com/api/{container}/{account}/workspaces/{workspace_name}/{datasource_id}/@getAllChildren"
Because Datasources can be large, you also provide a way to control the page size, and using a cursor for the results:
f"https://enterpise.onna.com/api/{container}/{account}/workspaces/{workspace_name}/{datasource_id}/@getAllChildren",
data=json.dumps({"page_size": 7})
2
f"https://enterpise.onna.com/api/{container}/{account}/workspaces/{workspace_name}/{datasource_id}/@getAllChildren",
json.dumps({"page_size": 7, "scroll": resp["cursor"]}),
2