Copilot

Introduction

Copilot is a suite of AI-powered features integrated into TQ Data Foundation, designed to assist users in creating, discovering, and managing data more efficiently. Copilot features are only available in SaaS environments.

Use Copilot Chat with TQ Data Foundation

Copilot Chat answers questions about TQ Data Foundation based on the official documentation.

Note

Currently, only single questions are supported and not full dialog. Dialog support will be available in the next release.

Enable Copilot Chat with TQ Data Foundation

Please contact support to enable the Chat with TQ Data Foundation feature for your installation, if it’s not already enabled.

Use Copilot Chat with TQ Data Foundation

You can find the Copilot Chat feature on the start page, at the bottom right side. Press the chat icon to open the chat panel.

When the chat panel opens, a list of frequently asked questions appears as examples to help you get started. To ask your own question, enter it in the input field at the bottom and press Enter or the send button to submit your question.

After submitting your question, it may take a short while for the answer to be generated. Once generated, the answer appears in the chat panel. You can scroll through the chat history to review previous questions and answers.

Some answers may contain many details, which can make them too small to read comfortably in the chat panel. In such cases, press the expand button to make the answer full screen, as shown below:

Use Copilot Linking

Copilot Linking leverages the Vector Index to add properties that refer to other asset collections using the search methods provided by the Vector Index. Based on properties given in the configuration, Copilot Linking will search for matching resources in the target asset collection. Any asset collection for which the Vector Index has been enabled can be used as a target for Copilot Linking.

Enable Copilot Linking for a property

Copilot Linking must be configured in the property shape in an ontology. The following example shows how to enable Copilot Linking for the related property of SKOS Concept. The content of the source properties is used to search for matching resources in the target asset collection. For the related property, definition and preferred label should contain the information that the target should match. In other cases, there could be a dedicated literal that matches better. For example, if there is already a literal for brand and Copilot Linking should add a property to a catalog of brands, that literal property should be used as the source. The asset collection that contains the target must be selected in graph. A search options data structure must be added where further settings can be configured. The search options can be used by multiple Copilot Linking property configurations.

Copilot Linking configuration for SKOS Concept related

Search options can be used to tweak the search for better results. The most important settings are:

Parameter	Description
search alpha	The relative weight of keyword and vector search for the hybrid search in the range between 0 and 1. 0 is pure keyword search, and 1 is pure vector search. As the hybrid search includes a normalization step, setting this value to 0 or 1 may not give the same result as changing the method to keyword or vector.
search limit	The upper limit of results that will be shown.
search method	The search method used by the Vector Index. - exactPhraseMatch uses the keyword search to find full matches of a phrase. For example, New York doesn’t match York, only New York. - hybrid combines the results of keyword and vector. It gives a high probability to exact matches and adds semantic similarity to the mix. As it gives the best results for most use cases, it’s used as default. - keyword uses BM25 to rank exact matches. - vector uses the configured embedding model to calculate the cosine similarity as base for the probability.
search threshold	A threshold value for the search score. Only results with a score value greater or equal to the threshold will be shown.

Note

A separate ontology asset collection should be used for the Copilot Linking configuration if the underlying ontology is generic and not designed for a single target. The ontology for the configuration can be added under Settings → Includes, like in the following example where SKOS arXiv AI Linking contains the Copilot Linking configuration.

Applying Copilot Linking suggestions

Copilot Linking based suggestions are shown in the Problems and Suggestions panel. Run AI Linking Suggestions must be enabled in the dropdown menu on the top right. It can be used combined with other actions. If this is not wanted, all other Run actions should be unchecked. The Apply button will create the suggested property. If the property shape allows multiple values, multiple suggestions can be applied.

Note

Problems and Suggestions can be triggered for smaller batches using batch actions. In the tree of the Taxonomy Concepts panel, batch actions can be triggered in the dropdown menu that opens on right click.

Use Copilot Natural Language to SPARQL

Enable the Copilot Natural Language to SPARQL Feature

To enable the Natural Language to SPARQL feature for an asset collection, the ontology and the asset collection must each be indexed with the Vector Index, and a Natural Language to SPARQL configuration must be created. It is recommended to use the configuration wizard for this, see Natural Language to SPARQL Configuration Wizard. For advanced setups, the configuration can also be created manually, see Manual Configuration.

Natural Language to SPARQL Configuration Wizard

The configuration wizard guides you through all steps that are needed to enable the Natural Language to SPARQL feature for an asset collection, including the Vector Index of the ontology and of the asset collection.

It is opened from the asset collection page under Settings > Manage. The AI Features section shows whether Natural Language to SPARQL is already configured, and the Vector Index section provides the Configure NL to SPARQL button that starts the wizard.

Configure NL to SPARQL button in the Manage settings of an asset collection

The wizard first asks how you want to configure the feature:

Automatic: The system analyses the ontology and picks the best classes, properties and thresholds. This is fast and works well for most collections.
Manual: You choose classes, properties and search parameters yourself, which gives full control over the configuration.

Choice between automatic and manual configuration

In both cases the wizard creates the Vector Index configurations and the Natural Language to SPARQL configuration, and then starts the indexing tasks in the background.

Automatic Wizard Configuration

After choosing Automatic, select the Ontology that should be used to generate the SPARQL queries. The list contains the ontologies that are imported by the current asset collection. Classes and properties are then selected automatically based on their relevance, and can be adjusted later with the manual configuration.

Under Advanced settings, the following options can be tweaked:

Class threshold: The minimum number of instances that a class must have to be included.
Property threshold: The minimum number of instances that must use a property for it to be included.
Clear the existing configuration: Removes any existing Natural Language to SPARQL and Vector Index configurations on both the ontology and the asset collection before the new ones are created.
Push existing data: Builds the Vector Index for both the ontology and the asset collection immediately after the configuration has been created. Disable this option to review and tweak the generated configuration before the indexing is triggered.

Both thresholds serve to keep noise out of the indexed data, so that internal system data, such as the definition of the asset collection or its settings, does not end up in the Vector Index alongside the actual assets. The default values work for most cases. For very small datasets, however, it may be necessary to lower the thresholds, as otherwise too few classes and properties are selected.

Automatic configuration with the advanced settings expanded

Press Start Configuration to create the configuration and start the indexing. The wizard shows the progress of the indexing tasks for the asset collection and for the ontology. The indexing runs in the background, so the window can be closed and you will receive a notification once it is complete.

Progress of the indexing tasks started by the wizard

When all tasks are complete, the Manage settings report that Natural Language to SPARQL is configured and that the Vector Index is up to date, including the number of indexed instances.

Manage settings after the configuration has been created

Manual Wizard Configuration

After choosing Manual, the wizard walks through three steps. The banner at the top of each step shows whether the ontology and the asset collection are already indexed.

Step 1: Select ontology

Select the ontology that should be used for the Natural Language to SPARQL queries. The list contains the ontologies that are imported by the current asset collection. If the ontology or the asset collection is not indexed yet, the wizard points out that the missing indexes will be created automatically when the configuration is started.

Step 1 of the manual configuration, selecting the ontology

Step 2: Configure

Select the classes that should be indexed with the + button next to Classes to index. The classes are taken from the selected ontology, and the number of instances in the asset collection is shown next to each selected class.

Adding a class to the index in step 2 of the manual configuration

The Properties section lists the properties that are indexed for all selected classes. Select the properties for labels, names, descriptions, comments, and identifiers, and deselect the ones that are not relevant for querying.

The Search parameters control how the assistant searches the index:

Keyword: Exact and fuzzy text matching, which is fast and reliable for precise terms.
Vector: Semantic similarity search, which is better for natural language queries.
Hybrid: Combines keyword and vector search, where the Alpha value weights the balance between the two.

The Thresholds (lower bound) value is the minimum score that a result must reach to be returned, and can be raised to reduce noise in the results. The defaults work for most cases.

Step 2 of the manual configuration, with classes, properties and search parameters

Step 3: Review and start

The last step summarizes the selected ontology and class, the resulting index configuration, the search settings, and the indexing tasks that will run.

Step 3 of the manual configuration, reviewing the configuration

Starting the configuration creates the configurations and runs the indexing tasks in the background, in the same way as the automatic configuration. A notification is shown once the indexing is complete.

Manual Configuration

For advanced setups, the configuration can also be created manually as an alternative to the configuration wizard, see Natural Language to SPARQL Configuration Wizard.

Configure the Vector Index for the Ontology

To enable the Copilot Natural Language to SPARQL feature for an asset collection, it is necessary to index the ontology that is used. It requires configuring a predefined list of classes and properties in the ontology. This is required for finding relevant classes and properties for a given prompt. For more details on how to configure the Vector Index, please refer to Enabling the Vector Index for an Asset Collection.

The following classes must be configured:

Class: owl:Class
Property (rdf:Property)
Property Shape (sh:PropertyShape)

And the following properties must be configured:

labels: rdfs:label, order 1, keyword true
names: sh:name, order 2, keyword true
comments: rdfs:comment, order 3, keyword false
descriptions: sh:description, order 4, keyword false

The configuration should look like this:

Vector Index configuration for the ontology

Configure the Vector Index for the Asset Collection

It’s also required to configure the Vector Index for the asset collection that should be queried. There is no predefined configuration for that use case. The classes should be selected to cover all relevant resources that could be queried, while avoiding classes from imported asset collections that generate noise or represent irrelevant metadata. Properties for labels, names, descriptions, comments, and identifiers should be selected. For more details, see the Vector Index documentation.

Configure Natural Language to SPARQL for the Asset Collection

It’s mandatory to configure the SPARQL ontology graph. Select the ontology that has been configured for the Vector Index in the previous step. Optionally, tweak the SPARQL data search options and SPARQL ontology search options, like set a threshold to reduce the noise in the results or change the alpha value to give different weight to keyword and vector search results. It’s possible to select an alternative SPARQL LLM (Large Language Model), but the default model is recommended for most use cases.

Natural Language to SPARQL configuration

Use the Copilot Natural Language to SPARQL Feature

When everything is configured, you can use the Natural Language to SPARQL feature. An additional input field appears at the bottom of the SPARQL Query panel, where you can enter your prompt. After entering your prompt, press the arrow up button to generate the SPARQL query. Please note that it may take a short while for the query to be generated.

Prompt input for Natural Language to SPARQL

Once generated, the query appears in a separate box for your review. Since generating the query is not deterministic, there is a button to regenerate the query if the initial result is not satisfying. If you are satisfied with the generated query, you can transfer it to the main SPARQL query editor by pressing the green accept button.

Query generated by Natural Language to SPARQL

Use Copilot Tagging Content

Any asset collection for which the Vector Index has been enabled can be used by the AutoClassifier in the Content Tag Set. Unlike using Maui Server, this method doesn’t require a training step.