Document version history
Version number | Modified by | Modifications made | Date modified | Status |
1.0 | Shelf | Initial release of the document. | 5 April 2024 | Active |
Document purpose
This document has been developed for those customers’ end users who are using Shelf Knowledge Management System (Shelf KMS) with the Content Integration Layer feature to manage and handle various content both stored in Shelf KMS and in external platforms, including local file systems, integrated via CIL. The document’s goal is to answer the following questions:
- What is the Shelf Content Integration Layer?
- What purposes does this feature serve?
- What is a Universal connector?
- How is it configured?
- How is it used in practice?
If you require further assistance, feel free to contact us at support@shelf.io or from the in-app chat support within the Shelf platform.
Glossary
Prior to starting to learn about the Universal connector and its configuration and usage, you need to learn the terminology used by Shelf.
Shelf KMS (Shelf Knowledge Management System) is a knowledge management platform that employs AI capabilities to create, edit, process, store, export/import, and otherwise handle various content. The platform sports multiple AI-enabled features such as Search Copilot, Analytics Copilot, Semantic Search, Content Intelligence etc., all of which are intended to optimize and centralize customer organizations’ knowledge and provide fast, accurate, and consistent answers.
The Content Integration Layer (CIL) in Shelf Knowledge Management System (Shelf KMS) is a feature that allows for the seamless integration of various types and formats of content into the system. It enables users to import and organize content from various sources and knowledge repositories.
A connector is an entity that defines which external Content Management System Shelf KMS needs to connect to via Content Integration Layer (e.g. the Universal connector is a set of API parameters to define how to connect Shelf to Universal). Shelf KMS has a set of preconfigured connectors (Universal, Universal, Sharepoint, Confluence connectors, etc.) but also allows for creating the custom ones.
A connection profile is a feature that allows users to establish a connection to various external applications or sources. It enables seamless integration and collaboration between Shelf KMS and other platforms such as external content repositories. Using Shelf’s connection profiles, users can import and synchronize various content and generally streamline workflows between Shelf KMS and external systems. This feature enhances the overall efficiency and effectiveness of knowledge management processes by enabling easy access to relevant information from different sources within the Shelf platform.
A sync flow is a feature that is responsible for the synchronization of content from external sources to Shelf KMS. Shelf’s sync flows can work on schedule or on demand, making it a versatile tool for content synchronization and prompt knowledge delivery.
S3 Token is a temporary access credential used to authenticate and authorize access to Shelf resources (Knowledge Management System and other products) for external applications and systems. Shelf token replaces both username and password.
JSON, which stands for JavaScript Object Notation, is a lightweight data-interchange format commonly used for transmitting data between a server and a web application, as well as for storing configuration data. JSON is human-readable and easy for both humans and machines to parse and generate, making it a popular choice for data exchange in web development and APIs.
Prerequisites
For proper operation of your Universal connector and successful synchronization of your content from the Universal platform into Shelf KMS, the following preconditions must be met:
- Admin Panel must be accessible on your Shelf KMS account
- Content integration utility CLI must be made available for you. Reach out to your organization’s administrator(s) to learn more.
- Content items (articles, documents, etc.) must be available in your local file system or storage.
Content Integration Layer and its purpose
The Content Integration Layer feature makes it possible to integrate various content types and formats into a single platform, Shelf KMS, from where users can further derive useful information to deliver accurate and relevant answers to their clients.
The Content Integration Layer ensures that content is indexed, tagged, and categorized appropriately, making it easily searchable and accessible to users within the knowledge management system. It provides capabilities for content mapping, transformation, and synchronization, enabling the consolidation and synchronization of data from multiple sources.
CIL also ensures that the latest version of content is always available to users and facilitates efficient content updating and maintenance.
Overall, the Content Integration Layer plays a crucial role in providing a centralized repository of relevant and up-to-date information, making it easier for users to find and access the knowledge they need.
The detailed information about the CIL feature can be found in the relevant chapters of the Shelf Content Integration Layer Feature User Guide.
Understanding the Content Integration Layer workflow
In-depth configurations and use cases of the Shelf Content Integration Layer are described in the dedicated Shelf Content Integration Layer Feature User Guide mentioned above, but the short example of the CIL workflow is as follows:
- You access the Content Integration Layer feature in your Shelf KMS (Admin Panel).
- You select one of Shelf-preconfigured connectors or create your custom one (based on the external or local content repository from which you need to sync content)
- You create a connection profile that uses the selected or created connector, and configure it: indicate credentials (username/password/API key/token/etc.) to access your internal repository. In our case, we talk about Shelf Token which you can get in accordance with the procedure laid down in the respective section below.
- You create a sync flow that uses the created connection profile and configure this sync flow to run on schedule or on demand. Scheduled sync flow run is not supported for Shelf’s Universal connector.
- You run the created sync flow manually (if set to run on demand) and get your content from your external repository or, in our case, local file storage right in Shelf KMS.
For the proper operation of the Universal connector to sync your content from a local file system or storage, you need to perform certain actions and configurations in Shelf and on your computer. Read the chapters below to learn what exactly you need to do.
Configurations
This chapter and its subsections describe the steps you need to take to get your Universal connector configured and ready for use.
Accessing Content Integration Layer
From the standard workflow described above you see that to start configuring and using your Universal connector you need to access the Content Integration Layer feature. To do so, perform the following steps:
- Log into your Shelf KMS account using valid credentials.
- Once logged in, go to the Admin Panel by selecting the respective option in the dropdown menu.
Figure 1. Accessing Admin Panel in Shelf KMS
Figure 2. Accessing CIL modules in Admin Panel
Configuring Universal connector
In fact, no configurations are needed for your Universal connector. It is prebuilt and preconfigured by Shelf. You can see it in the connector list that appears once you select the Custom Connectors option in the Content Integration Layer section.
Figure 3. Viewing the Universal connector in CIL
What you need to do next is to create and configure a connection profile dedicated to work with your content source such as your local file system.
Configuring Universal connection profile
The Universal connection profile is a feature that uses the prebuilt Universal connector and allows Shelf KMS to access your file system, and communicate with it to find and sync content stored there. But to enable its proper operation, you need to create and configure it as described in the procedure below.
- In the Admin Panel in Shelf KMS, go to the Content Integration Layer section, find and select the Connection Profiles option.
- In the Connection Profiles window that appears, you see the list of available connection profiles. Neither the prebuilt nor other custom connection profiles can be used for communicating with Universal, so you need to create a new connection profile. Click the Create Connection Profile button to start.
Figure 4. Accessing CIL connection profiles and starting creating new connection profile
Figure 5. Configuring the Universal connection profile in Shelf
❗Note:
Fields with the asterisk symbol (*) are mandatory. If they are left blank, you are not able to save the connection profile you are creating.
Figure 6. Viewing the created Universal connection profile in Shelf
Obtaining S3 token
This step does not require major configurations. You just need to open the created Universal connection profile, either from its More actions menu or simply by clicking on it in the connectors list.
Figure 7. Opening Universal connection profile
Once the Universal connection profile page opens, find and copy the S3 Token as shown in the figure below. Save it to your local file for further use.
Figure 8. Viewing and copying S3 Token
Installing and configuring Content integration utility CLI
Once you have received from your organization’s administrator(s) the download link or the ready package of the Content integration utility CLI, download it and get ready for use.
Figure 9. Viewing Content integration utility CLI in MacOS
In case you have been provided the binary package of the utility, make sure to build the application using the following command:
yarn && yarn build && yarn make-binary |
If your PC’s operating system is Linux or MacOS, you need to add run permissions for your application using the Terminal app:
MacOS
chmod +x ./content-integration-cli-macos |
Linux
chmod +x ./content-integration-cli-linux |
Once done, you need to run the Content integration utility CLI application. Make it using the terminal (run as administrator) and OS-specific commands:
MacOS
./content-integration-cli-macos |
Linux
./content-integration-cli-linux |
Windows
./content-integration-cli-win.exe |
Using Content integration utility
After you run the utility, follow the prompts you see on the screen.
1. Select the proper region for your Shelf KMS account.
Figure 10. Selecting Shelf KMS account region in CLI
2. Next, enter the S3 Token you have obtained after creating the Universal connection profile.
Figure 11. Entering S3 Token in CLI
3. Then, provide a desired name for your configuration. In our example, it is Shelf_Universal.
Figure 12. Entering configuration name in CLI
4. Finally, indicate the path to your content storage in your local file system. Use full or relative paths. Note that if the directory (folder) name has whitespace(s) in it, you need to enclose the entire path into quotation marks (e.g. “path”).
Figure 13. Finding the content folder in your local file system and entering its path in CLI
Once done, hit Enter on your keyboard.
If you have entered everything correctly, the Content integration utility fetches the content items from the indicated location and displays the respective details in the Terminal window.
Figure 14. Viewing the details of content fetching by CLI
You can now close the CLI utility and go to the Shelf Admin Panel to do further steps.
📢 | Note that you can avoid making most of the above mentioned steps when using the Content integration utility by executing the following command in the Terminal app: Also, replace “path_to_local_content_storage” with the real path to your content storage directory. |
---|
Configuring Universal sync flow
Now that you have configured the Universal connector, Universal connection profile, installed and run the Content integration utility to fetch content from your local file storage, you need to create and configure the Universal sync flow. Sync flows are basically sets of instructions telling Shelf KMS what content and when it needs to pull from content sources, in our case - local file storage.
To create and configure the Universal sync flow, perform the following steps.
Figure 15. Starting creating Universal sync flow
Figure 16. Configuring the Universal sync flow
Once done, click the Create button to finish your Universal sync flow creation.
Running Universal sync job
Once you have created the Universal sync flow, you need to start the sync manually at any time (the On-schedule run type is not supported for the Universal connector).
To manually start syncing your content from Universal into Shelf, follow the below steps.
Figure 17. Manually running content syncing from Universal into Shelf
Figure 18. Confirming the sync start
Figure 19. Viewing the running Universal sync job
❗Note:
The sync job can be triggered manually unless there is a sync job running at that time.
If there is a running sync job in this sync flow when you attempt to start a new job, this new job will not be created and, respectively, the content sync will not be started. In this case you will see the error message informing that the job is already running.
If there are no other sync jobs running in this sync flow, the Jobs tab window refreshes, and a new job with the IN PROGRESS status appears in the table grid.
Figure 20. Stopping the sync job
Checking syncing results
Once the sync job triggered manually or on schedule is finished, you will see it to change the status to SUCCESS (if the sync was successful) or FAILED (if some issues occurred during the sync). In the latter case, check your configurations in the Universal connection profile and sync flow and rerun the sync job.
Figure 21. Viewing the successfully completed sync job
If the sync was successful, you can see the recently sync content directly from this window. To do so, open the Recent Sync tab and check the content items synced from Universal into Shelf.
Figure 22. Viewing the recently synced content from Universal
Note that the tab shows not more than 50 content items, so if you have synced more items, you will see them only after going back to the Shelf KMS homepage and selecting the respective source - Universal.
Configuring Universal permissions in User Groups
Even when you successfully configured the Universal connector, connection profile, sync flow, run the sync job and pulled the content from your local repository, you need to add Universal collections to the permitted sources via Shelf’s User Groups feature. Otherwise, you will not be able to search for the synced content and otherwise manage it in Shelf KMS.
To set the mentioned collection permissions, go back to the Admin Panel where find and select the User Groups option in the left sidebar menu. Select the needed user group from the list to open it.
Figure 23. Accessing User Groups in Shelf
In the selected user group window that opens, go and select the Collection Permissions tab. Once it opens, click the Add Collection button.
Figure 24. Starting to set Universal collection permissions in Shelf
In the form that opens, find and select the needed Universal collections you want to allow.
Figure 25. Selecting Universal collections
Confirm your choice by clicking the Add button. Once the collections are added, they appear in the list.
Figure 26. Viewing the added Universal collections
Once done, you are mostly finished with all the configurations needed to sync content from Universal into Shelf KMS using the Universal connector.
Understanding content items in Universal connector reality
Opposite to other Shelf connectors that are syncing content from external sources - Contentful, Confluence, Zendesk, Sharepoint, Google Drive etc. - Shelf’s Universal connector syncs content from local storages and has its own understanding of content and content items.
Collections, folders, and articles (content items)
Collections
When syncing content, the CLI utility assigns all the synced content to the default collection named root-collection. However, you may use your own list of collections for the synced content by creating a respective _collections.json file (2) and placing it in the root of the directory (1) from where you sync content to Shelf.
Figure 27. Creating _collections.json in the primary folder root
The _collections.json must follow the below schema:
[ { id: string, // *required; min length: 1, max length: 500 name: string // *required; min length: 1, max length: 500 } ] |
Folders and articles/content items
Shelf’s Universal connector recognizes nested folders (that is, folders inside the synced file directory) as articles/content items, and all files, including child folders, stored in such nested folders are considered attachments.
Figure 28. Understanding content concepts of Shelf’s Universal connector
The content items resulting from the sync may be enriched with properties contained in particular meta files -_global.meta.json and _meta.json.
Global meta
The _global.meta.json file is placed in the root of the synced directory and serves as a set of mappings between external content keys and Shelf CIL keys.
The schema is as follows:
{ mappings: Record<string, string>, urlTemplate: string, // in format of url with '{id}' included in // order to use for external url build } |
For example, {mappings: {"Article title": "title"}} maps the Article title property of the articles stored in your local directory with the title of the imported CIL items.
In other words, the _global.meta.json file serves as a dictionary that helps Shelf’s CIL understand external content metadata and translate it correctly into the synced content metadata.
Exemplary _global.meta.json:
"mappings": { "title": "Article Title", "id": "_id", "updatedAt": "updated_at", "createdAt": "created_at", "tags": "Tags" }, "urlTemplate": "https://some.com/{id}" } |
Our real _global.meta.json:
{ "mappings": { "collectionIds": "collections" } } |
Article meta
The article meta or _meta.json is placed inside an article (nested) folder and defines the actual values of article properties, such as id, createdAt, title, and more, allowing CIL to construct the article during the import.
Exemplary article meta:
{ "Article Title": "meta-configured-title", "Tags": [ "some", "asd", "sadasd" ], "_id": "external-id", "updated_at": 'ISODATE', "createdAt": "ISODATE", } |
Our real article meta telling that this article belongs to Collection 2:
{ "collections": [ "2" ] } |
Random meta
If _meta.json is not provided for the article (nested folder), the first encountered JSON file is used by Shelf CIL to parse the article metadata.
your-import-folder/ ├─ _collections.json ├─ _global_meta.json ├─ article-1/ │ ├─ article-1-attachemnt.pdf ├─ article-2/ │ ├─ _meta.json ├─ article-3/ │ ├─ some-random-meta.json <--- Will be taken as meta file │ ├─ some-random-meta-2.json |
Default meta
If the conditions mentioned above are not applicable or if the meta JSON file provided with the content is not complete, the default meta is applied.
The schema of the applied default meta properties is as follows:
id - derived from the hash of the directory path
updatedAt - based on the directory's update time
createdAt - based on the directory's creation time
title - derived from the directory's title
collectionIds - added to the root collection by default
Viewing Universal resources in Shelf
After completing all the configurations on your Universal connector and associated features and options, you can now view and work with the content synced into Shelf KMS from your local file repository.
Go back to Shelf KMS homepage and select Universal as the needed source in the navigation panel.
Figure 29. Viewing content synced from Universal into Shelf
Note that you cannot open and view content synced via the Universal connector in Shelf KMS. However, this content is searchable and can be processed by other Shelf applications such as Shelf Search Copilot, Shelf Content Intelligence, and Shelf Content Maintenance.