Platform Sync - Data Source Connector

Platform Sync (previously known as Hosted GET) lets you set up a data source connector for any external database on any external system for which we do not have a live connector. While it is a bit more technical to configure, it does allow you to get around certain platform limitations, like our 50,000-row limit for system data sources, as Platform Sync data sources can have any number of rows. In this article, we'll take a deep dive to give you all the details on Platform Sync, such as when to use it, the pros and cons, and how to set it up. This article assumes that you understand data sources, database systems, and API endpoints, as these are all concepts we will refer to extensively.

Article CONTENTS

• Overview
• Setting up a Platform Sync Data Source Connector
• GET Request Essentials
• GET Request Syntax
• GET Response Syntax
• GET Response Details
• Example Code
• Troubleshooting / Testing

Overview

Below is a summary of some of the key benefits of using Platform Sync data sources in your projects:

• The key advantage of the Platform Sync connector is that you can control the data rows exposed to the app, letting you filter rows down to the individual app user level as desired. This is important for ERP, CRM, and other enterprise systems as the security/permissions around "who gets to see what" is critical.  
• Your data does not need to be pushed to our platform, so this connector resolves any data sovereignty concerns you may have, as no data is copied to our platform servers as with other data source connectors.  
• Data Sources that run off of Platform Sync connectors are not subject to the standard 50,000-row limit since the rows are delivered directly to the user's device.
• Unlike other data source connectors that run on a scheduled time interval (e.g., every 15 minutes), with Platform Sync data sources, rows are fetched each time the App syncs with the platform, giving you access to your data as needed. This might come at the expense of large amounts of data usage or slow app response times for large data sources, so experimentation and testing are advised when attempting a Platform Sync implementation on 

As a rule of thumb, performance is affected by the overall file size of the data source more than the number of rows.

For example, 5,000 rows containing 100kb of text data per row can be expected to run slower than 50,000 rows with a total of 1kb of data per row.

Additionally, using "heavy" data sources on many screens that can be navigated sequentially by the user will decrease compounding performance, as those big data sources can get progressively accumulated in device memory.

With Platform Sync, you will need to test the app to find the best row count and screen mix for your specific data set and usage requirements.

Regarding implementing a Platform Sync data source, due to the technical complexity and requirements, it is assumed that you have software development resources to create the web service the Platform Sync connector needs.

Setting up a Platform Sync Data Source Connector

For test purposes, we recommend you use a REST API development service like - https://www.mockable.io

1. Create a Data Source as normal or select an existing one for use.
   
2. On the Rows page of your Data Source, manually define and organize your Data Source's columns to match the output of your web service.
   
   The order of the columns must match up with what your web service returns in order for any column references to work.
   
   If you change column orders on your web service response at any stage, you will need to update the columns on the Rows page to match.
   
3. On the Settings page of your Data Source, make sure that you set the External Id field with some value for identifying this Data Source.
   
   Remember this External ID value since you will need it when you work through the GET web service details later in this document.
   
4. Also, on the Settings page, add a Platform Sync connector via the "Add Connector" button.

Specify the web address of your web service endpoint in the provided Target GET URL field.

Note that an External ID is required when using a Platform Sync Data Source Connector.

    5. You can also set a global Data Source Sync URL for your organization (defined in 

        Organization Setup > Global Data Source URL) and specify that the connector uses

        it. This allows the reuse of the same service endpoint across multiple data sources.

GET Request Essentials

Most of the settings for the integration with your Platform Sync API can be obtained from the Organization Setup screen. This includes the Integration Key (Private Token) and Provider ID (Company ID).

We also support gzip and deflate compression on responses. This is recommended for large data sets since text data compresses very well, thus conserving mobile data allocations and massively reducing download time.

Our support for compression is identified through the "Accept-Encoding" header set by the app on its REST requests.

 Your web service/server should encode the entire response stream using gzip or deflate, which will then be recognized by our app code.

This is usually an automatic behavior at the webserver level, though you should check that your server configuration has enabled compression for JSON content.

 This is beyond our support scope regarding how to apply response encoding - you'll need to confirm this with your software developers.

If you use Microsoft IIS as your web server, this article may help check your compression settings.

This service may also help identify if your server response is indeed being compressed.

GET Request Syntax

The app issues GET requests to your service, passing the values below as query string parameters.

This is the format of a typical request that is made by the app to your Platform Sync endpoint, with user@somewhere.com as the logged-in user.

GET https://yourdomain.com/xx/rowsearch?email=user@somewhere.com&integrationkey=xxxxxxxxxxxx&providerid=1&id=JOBS&lastupdated=2024-02-12T14:24:09

{
  "DataSources": [
    {
      "Id": "JOBS",
      "Rows": [
        [
          "10011",
          "10011 - Test Generic Project",
          "TEST",
          "12345AA",
          "19 Nov 2013",
          "12 Somewhere Road"
        ],
        [
          "ZDS654",
          "ZDS654 - Maintenance Costs to Excavator",
          "TEST",
          "",
          "27 Nov 2013",
          "24 Somewhere Else Road"
        ],
        [
          "ZDF662",
          "ZDF662 - Maintenance Costs for Hitachi Excavator",
          "TEST2",
          "",
          "28 Nov 2013",
          "182 Another Place, Somewhere"
        ]
      ],
      "NewRows": [],
      "DeletedRows": []
    }
  ],
  "LastUpdated": "2024-02-11T15:45:50.4818098+10:00"
}