SharePoint Online to ElasticSearch Integration Tutorial

Overview

In this tutorial we will walk through how to set up a job in Simflofy that will index the content in your SharePoint Online repository into your Elasticsearch system so that you can view and manage your documents in our TSearch platform. This tutorial will include the use of the PII Task which is used to determine how many and which documents contain PII or Personally Identifiable Information.

Step 1. Setting up your SharePoint Online Repository Authentication Connection.

This will allow simflofy to connect to your SharePoint Online Repository.

To set up your SharePoint Authentication Connection

1. Select Authentication from the Connections section of the navigation menu.
2. At the bottom of the Authentication Connections page select the Create New Auth Connection button.
3. In the New Auth Connection form
   1. Name your connection. In this example we will use SPO
   2. Select the connection type which for this example will be SharePoint REST Auth Connector
4. Next edit your new SharePoint Authentication Connection with the following input
   1. Give your connection a description. Example: SharePoint REST Auth Connector
   2. Enter the username and password for your SharePoint System
   3. Enter the Server URL for your SharePoint system. Example: https://simflofy.sharepoint.com
   4. Leave the rest of the fields as is and click the Save button at the top of the page to save your changes and return to the Authentication Connections Page

Step 2. Set up your Elasticsearch Output Connection.

This will allow Simflofy to connect to your ElasticSearch Repository.

To set up your Elasticsearch Output Connection:

1. At the bottom of the Authentication Connections page select the Create New Auth Connection button.
2. In the New Auth Connection form
   1. Name your connection. In this example we will use rm-Elastic Search Authentication Connector
   2. Select the connection type. In this case we will use Elasticsearch Authentication Connector
3. In the next page we will edit your new Elasticsearch connection
   1. In the server URL field enter the URL for your Elasticsearch Server. For example: http://127.0.0.1:9200/
   2. Leave the other fields as is and click on the Save button to save these changes.

Step 3. Set up your SharePoint Integration Connection.

Using this connection Simflofy will generate a query, or use one provided, to retrieve unique ids for documents.

To set up your SharePoint Integration Connection:

1. Select Integration from the Connections section of the navigation menu.
2. At the bottom of the Integration Connections page click the Create Integration Connection button.
3. In the New Integration Form
   1. Name your integration connection. In this example we will use the name rm-SPO
   2. For the connection type choose SharePoint REST Connector from the drop-down list. You can also begin typing SharePoint in the search field to filter the drop-down list and then select it.
   3. Click save to edit this new integration connection
4. In the Edit Connection: RM-SPO page enter the following
   1. Give your connection a description. Example: SharePoint Online
   2. Choose the Authentication Connection you created in Step 1. SPO
   3. Leave the rest of the fields as is and click Save to lock in your changes and return to the Integration Connections page

Step 4. Set up your ElasticSearch Integration Connection

In Output mode, connectors push content and metadata. Many of them can also build version series' from the source systems.

To set up your Elasticsearch Integration Connection:

1. At the bottom of the Integration Connections page click the Create Integration Connection button.
2. In the New Integration Form
   1. Name your integration connection. Here we will use the name rm-ElasticSearch
   2. Chose the Elasticsearch Authentication Connection we created in Step 2. rm-Elastic Search Authentication Connector
   3. Leave the rest of the fields as is and click Save to finish creating this connection and return to the Integration Connections page.

Step 5. Create your Content Service Connection.

Simflofy Content Service connections offer public REST endpoints that allow for integration with external applications.

To create a content service connection for your Simflofy Repository:

1. Select Content Service from the Connections section of the navigation menu
2. At the bottom of the Content Service Connection page click the Create New Content Service Connection button
3. In the new Content Service Connection page enter the following information
   1. Connector ID: name your content service connection for this example we will use the name SharePointRM
   2. Description: Give your connection a description. Here we will use SharePoint Online RM Demo Content Service
   3. Type: The type for this connection will be SharePoint REST Content Service Connector
   4. Security Mode: Choose Authentication Connection as the security mode. And select the SharePoint Authentication Connection we created in Step 1 from the dropdown SPO
   5. Under the Connection Configuration Tab enter your SharePoint Site Name. Example: sites/SimflofyDemo
   6. Leave all other fields as is and click Save to finish the creation of this connection and return to the Content Service Connection Page

Step 6. Create a Job Mapping for your Content Service Connection

Content mappings will allow you to map custom parameters to properties in the destination system.

To create a Job Mapping for ElasticSearch:

1. Under the Integrations section of the navigation menu select Job Mappings
2. At the bottom of the Job Mappings page click the Create New Job Mapping button
3. If you don't name your job Simflofy will assign a name to it. Ours is Job Mapping 1628095318249
4. Enter the following mappings for Title and Date Created
   1. For the Title field mapping enter
      1. Source: Document.Title
      2. Target: cmis:description
      3. Type: String
      4. Click Add New Mapping
         1. Select Field Mapping from the drop-down
   2. For the Document field mapping enter
      1. Source: Document.Created
      2. Target: cmis:creationDate
      3. Type: Date Time
      4. Click Add New Mapping
         1. Select Field Mapping from the drop-down
5. Click Save to save this mapping for use in your Job Configuration.

Step 7. Set up your integration job.

This will integrate your SharePoint repository with your Elasticsearch Server. Allowing you to view your SharePoint files in TSearch through your Elasticsearch connection

To set up your integration job:

1. Select List Jobs from the Integration section of the navigation menu
2. At the bottom of the Jobs page click the button Create New Job
3. In the Create New Job Form
   1. Give your new job a name. For this example we will use rm-SPO
   2. For Repository Connection select the SharePoint Integration connection we created in Step 3 from the drop-down. rm-SPO
   3. For Output Connection select the Elasticsearch Connection we created in Step 4 from the drop-down. rm-ElasticSearch
   4. Leave the other fields as is and click save to continue setting up this new integration job.
4. In the Details Tab
   1. Under Content Service Connector, add the SharePoint Content Service Connection you created in Step 5. SharePointRM
5. In the Tasks Tab we will be adding the Default Tika Extractor Task.
   1. Select Tika Extractor Task from the dropdown
   2. Click the plus button to edit the task properties
      1. Uncheck the Fail Document on Extract Error box
      2. Uncheck the Remove Binary After Extraction box
      3. Leave all other fields as is and select Done
6. Next in the Task Tab we will be adding the Default PII Detection Task
   1. Select PII Detection Task from the dropdown
   2. Click the plus button to edit the task properties
      1. Check the Break up pii data into individual fields box.
      2. Leave all other fields as is and select Done
7. In the Mappings Tab under the Select Additional Mappings section
   1. Select the Job Mapping we created in Step 6. Job Mapping 1628095318249
8. The RM-SPO tab is where you will add any necessary configurations for the SharePoint Integration connection you are using as the repository.
   1. Repository Tab:
      1. Enter your SharePoint Site Name. Example: sites/SimflofyDemo/
      2. Leave all other fields as is
9. The RM-Elasticsearch Tab is where you will enter any additional settings needed to use Elasticsearch as your Output integration connection under the Server Tab.
   1. Under Index Name add the index where the documents will be stored. In this example we will use the index rm
   2. Leave all other fields as is
10. Click Save to save your job configurations and return to the Jobs page

Step 8. Run and Monitor the job

This will integrate the chosen content from your SharePoint repository to your Elasticsearch repository allowing you to view the content in the TSearch Platform. It will also identify and add the necessary fields for PII.

To run and monitor this job:

1. Select Run and Monitor Jobs from the Integration section of the navigation menu
2. Find the job created in Step 7. RM-SPO
3. Select the triangle next to the job to run this Integration job.

Depending on how many files you have stored this could take more than a few minutes. To monitor the progress of this integration you can click the Refresh button at the top of the page. You can also set Simflofy to automatically refresh every 30 seconds, every minute, or every 5 minutes.

Once the integration is complete the status will change to green and state Complete.

Congratulations! You have successfully integrated your SharePoint Online Content into ElasticSearch!