Create Ticket My Tickets Post Discussion
Welcome
Login  Sign up

Platform Sync - Data Source Connector


Platform Sync (previously known as Hosted GET) lets you set up a data source connector for any external database on any external system for which we do not have a live connector. While it is a bit more technical to configure, it does allow you to get around certain platform limitations, like our 50,000-row limit for system data sources, as Platform Sync data sources can have any number of rows. In this article, we'll take a deep dive to give you all the details on Platform Sync, such as when to use it, the pros and cons, and how to set it up. This article assumes that you understand data sources, database systems, and API endpoints, as these are all concepts we will refer to extensively.


Article CONTENTS


Overview


Below is a summary of some of the key benefits of using Platform Sync data sources in your projects:


  • The key advantage of the Platform Sync connector is that you can control the data rows exposed to the app, letting you filter rows down to the individual app user level as desired. This is important for ERP, CRM, and other enterprise systems as the security/permissions around "who gets to see what" is critical.  
  • Your data does not need to be pushed to our platform, so this connector resolves any data sovereignty concerns you may have, as no data is copied to our platform servers as with other data source connectors.  
  • Data Sources that run off of Platform Sync connectors are not subject to the standard 50,000-row limit since the rows are delivered directly to the user's device.
  • Unlike other data source connectors that run on a scheduled time interval (e.g., every 15 minutes), with Platform Sync data sources, rows are fetched each time the App syncs with the platform, giving you access to your data as needed. This might come at the expense of large amounts of data usage or slow app response times for large data sources, so experimentation and testing are advised when attempting a Platform Sync implementation on 
There are performance limits on the app side when the rows become large - this varies across Android and iOS, and also depends heavily on the device specifications. 

As a rule of thumb, performance is affected by the overall file size of the data source more than the number of rows.


For example, 5,000 rows containing 100kb of text data per row can be expected to run slower than 50,000 rows with a total of 1kb of data per row.


Additionally, using "heavy" data sources on many screens that can be navigated sequentially by the user will decrease compounding performance, as those big data sources can get progressively accumulated in device memory.


With Platform Sync, you will need to test the app to find the best row count and screen mix for your specific data set and usage requirements.


Regarding implementing a Platform Sync data source, due to the technical complexity and requirements, it is assumed that you have software development resources to create the web service the Platform Sync connector needs.


Setting up a Platform Sync Data Source Connector


For test purposes, we recommend you use a REST API development service like - https://www.mockable.io

  1. Create a Data Source as normal or select an existing one for use.
  2. On the Rows page of your Data Source, manually define and organize your Data Source's columns to match the output of your web service.

    The order of the columns must match up with what your web service returns in order for any column references to work.

    If you change column orders on your web service response at any stage, you will need to update the columns on the Rows page to match.
  3. On the Settings page of your Data Source, make sure that you set the External Id field with some value for identifying this Data Source.

    Remember this External ID value since you will need it when you work through the GET web service details later in this document.
  4. Also, on the Settings page, add a Platform Sync connector via the "Add Connector" button.

Specify the web address of your web service endpoint in the provided Target GET URL field.

Note that an External ID is required when using a Platform Sync Data Source Connector.


    5. You can also set a global Data Source Sync URL for your organization (defined in 

        Organization Setup > Global Data Source URL) and specify that the connector uses

        it. This allows the reuse of the same service endpoint across multiple data sources.



GET Request Essentials


Most of the settings for the integration with your Platform Sync API can be obtained from the Organization Setup screen. This includes the Integration Key (Private Token) and Provider ID (Company ID).

IMPORTANT!
Your web service MUST return a response in the following structure. Failure to comply with this format will result in failure to download rows on your app.
Your response MUST be in JSON format.


We also support gzip and deflate compression on responses. This is recommended for large data sets since text data compresses very well, thus conserving mobile data allocations and massively reducing download time.


Our support for compression is identified through the "Accept-Encoding" header set by the app on its REST requests.

 Your web service/server should encode the entire response stream using gzip or deflate, which will then be recognized by our app code.


This is usually an automatic behavior at the webserver level, though you should check that your server configuration has enabled compression for JSON content.

 This is beyond our support scope regarding how to apply response encoding - you'll need to confirm this with your software developers.


If you use Microsoft IIS as your web server, this article may help check your compression settings.


This service may also help identify if your server response is indeed being compressed.


GET Request Syntax


The app issues GET requests to your service, passing the values below as query string parameters.


This is the format of a typical request that is made by the app to your Platform Sync endpoint, with user@somewhere.com as the logged-in user.

GET https://yourdomain.com/xx/rowsearch?email=user@somewhere.com&integrationkey=xxxxxxxxxxxx&providerid=1&id=JOBS&lastupdated=2024-02-12T14:24:09

The parameters passed on the request are detailed below:

Param Name

Data Type

Required

Description

ids

String

Yes

Comma-separated list of External IDs for each Data Source that the app wants to synchronize.
 You set the External Id on your Data Sources via our platform's App Builder>Data Sources>Settings page. The image below shows where the external ID for the data source is set.

providerId

Integer

Yes

Your unique Provider Id found on your Organization Setup page (Organization&Users>Organization Setup)
 Use this with the IntegrationKey and Email to authenticate the request. ProviderID is the same as Company ID.

integrationkey

String

Yes

Your unique Private Token is on your Organization Setup page (Organization&Users>Organization Setup).
 Use this with the Provider and Email to authenticate the request.

email

String

Yes

Email address of the logged-in app user making the request.
 Use this to identify the user in your system and apply any user-specific filtering you wish against the returned Data Source rows.

lastupdated

DateTime

(YYYY-MM-DDTHH:MI:SS)

Yes

The last time that the app checked for Data Source updates.

Use this to implement incremental row updates, passing back just the new/updated and deleted rows since the time the app checked in.

This is recommended for large Data Sources to save server resources and prevent exhausting mobile data allocations.


You can also use this value to determine whether there is anything at all to update since the last time the app pinged your service.

If nothing has changed, you can simply return a response with no DataSources property defined.


The first time the app checks for rows, the LastUpdated value will be at least 1 year in the past.  

We suggest you monitor for this and return the full data set in the Rows property of your response.


lastupdateds

String

No

Comma-separated list of DateTime values in the format "YYYY-MM-DDTHH:MI:SS" that represents the last time the app requested an update for the corresponding data source. The values are listed in the same order as the external IDs in the Ids parameter.


Each date in the list represents the last time the app checked for Data Source updates for the specific data source. Use this to implement incremental row updates, passing back just the new/updated and deleted rows since the time the app checked in. This is recommended for large Data Sources to save server resources and prevent exhausting mobile data allocations.


You can also use these values to determine whether there is anything at all to update since the last time the app pinged your service. If nothing has changed for the related data sources, you can simply return a response with no DataSources property defined.


The first time the app checks for rows for a specific data source, the related last updated value will be left blank in its position in the comma-separated list. We suggest you return the full data set in the Rows property of your response in this case.

 

Given that this connector is REST-based, you can test your web service directly via your web browser using a REST plugin like the Postman plugin for Google Chrome.

In this example, we assume you have set the "Target GET URL" on your Platform Sync Connector to https://yourdomain.com/xx/rowsearch.

We also assume that the Data Source has an External ID of "JOBS".

Your endpoint MUST return the data in the JSON format specified below:


GET Response Syntax


{
  "DataSources": [
    {
      "Id": "JOBS",
      "Rows": [
        [
          "10011",
          "10011 - Test Generic Project",
          "TEST",
          "12345AA",
          "19 Nov 2013",
          "12 Somewhere Road"
        ],
        [
          "ZDS654",
          "ZDS654 - Maintenance Costs to Excavator",
          "TEST",
          "",
          "27 Nov 2013",
          "24 Somewhere Else Road"
        ],
        [
          "ZDF662",
          "ZDF662 - Maintenance Costs for Hitachi Excavator",
          "TEST2",
          "",
          "28 Nov 2013",
          "182 Another Place, Somewhere"
        ]
      ],
      "NewRows": [],
      "DeletedRows": []
    }
  ],
  "LastUpdated": "2024-02-11T15:45:50.4818098+10:00"
}

GET Response Details


Below is a detailed description of how each node on the JSON response body returned by your API endpoint should be formatted.

Input Name

Data Type

Required

Description

DataSources

Collection of DataSource

No

Collection containing the Data Sources that have been requested.

If nothing has changed since the LastUpdated received from the app request, then you can simply return a response with no DataSources property defined or, if returning multiple data sources, then no Rows, NewRows, and DeletedRows properties set on those you do not wish to update.


LastUpdated 

DateTime

(YYYY-MM-DDTHH:MI:SS)

Yes

The last time that the rows for these Data Sources were updated. Generally, we recommend that this should be the current date and time on your host server when processing the request.


This value is stored by the app and sent exactly as received in the LastUpdated and LastUpdateds fields of the next GET request made by the app.


This gives a central point of time for incremental updates in particular since the app will echo your server times rather than use its own unreliable device time.


The “T” in the required format is a delimiter.

The "Rows" array in the GET response from your endpoint (detailed in the table below) is a collection of values (Val), which are the individual column values for that Row.


Every Row must contain at least two Vals. The first Val should be the unique identifier or key that will be used to uniquely identify this specific row of values returned by your endpoint. The first Val is also critical as the key for performing incremental deletes through the DeletedRows field mentioned above.
The second Val should be the default displayable label text of the row.  You can override this default by configuring the Display Options field in the Data Source "Settings" page.

Input Name

Data Type

Required

Description

Id

String

Yes

The External ID of the DataSource for which the associated Row data applies.
Must match the External ID configured in the Data Source -> Settings page on the platform.

Rows

Collection of Row

No

A collection of Rows that will completely replace all current Data Source rows on the app.
If you specify Rows, any values in NewRows and DeletedRows values will be ignored.

NewRows

Collection of Row

No

A collection of rows should be added/updated to the existing data source rows on the app.
 Use this field to perform incremental inserts/updates for your Data Source.

DeletedRows

Collection of Row

No

A collection of Row Items that should be deleted from the existing Data Source rows on the app.
Use this field to perform incremental deletes of your Data Source.
 Each Row should contain a single Val element that is your unique identifier for the Row to delete.


Each value in the "Rows" array corresponds to a column from the data set your endpoint is returning. Below is a summary of how each returned value should be formatted.

Input Name

Data Type

Required

Description

Val

String

Yes

A column value for this row can contain any string value.

If you want this column to display an icon/image in the mobile app, specify the Val as an HTTP URL to the image in question.
The image linked must be in PNG or JPG/JPEG format.

If the GET fails, your web service should return a 400-level HTTP status.


Example Code


Attached to the bottom of this document is a C# file that outlines the request and response structures as simple classes.

Even if you are not using C#, reviewing the properties and structure in code form can be very helpful in getting started with your own Platform Sync service.


Troubleshooting / Testing


Once you have set up a Platform Sync connector on your Data Source, the Rows page for that Data Source will automatically attempt to pull back rows from the given GET URL.


The Rows page makes the same request as the app, using your current logged-in user as part of the request.


 This is a quick and easy way to see if your Platform Sync web service understands the GET request and returns rows in the correct format.


What do I do when Platform Sync works on my IOS app but not Android?


This is almost certainly due to how Android treats SSL certificates - essentially, Android is more strict about SSL certificate chains than iOS.

SSL certificate chains involve the primary SSL certificate for your web service endpoint domain and any intermediate certificates issued by your Certificate Authority.


If you attempt to connect to a web address that features an incomplete SSL chain, then Android will fail the connection.


If your Android devices cannot sync, you can test if the issue is related to an incomplete SSL chain by browsing your Platform Sync service URL from your default Android web browser.

You will likely get a connection error with text such as "the certificate authority is invalid".


You can further verify your SSL configuration using a testing service like SSL Labs:

https://www.ssllabs.com/ssltest/

 

If your SSL installation has issues, you will see warnings such as "chain incomplete" on the SSL Labs test results.


To resolve the issue, you must ensure that the Platform Sync web service server has the intermediate certificates from your Certificate Authority installed.

For example, if you use GoDaddy, the best place to start would be here:

https://support.godaddy.com/help/article/868/what-is-an-intermediate-certificate?countrysite=au

Once you have the correct certificate chain in place, the Platform Sync data sources will sync to your Android devices since they can now connect to your web service.

You can confirm the certificate chain by simply checking the Platform Sync URL in your Android web browser. When resolved, it should no longer show certificate errors.



Did you find it helpful? Yes No

Send feedback
Sorry we couldn't be helpful. Help us improve this article with your feedback.