Tag / Azure AI Search

by Simon Doy

Teams AI Library Blog Series: Episode #3 – Setup your Teams AI Library Application


Introduction

This blog post is part of a series of posts around delivering a Microsoft Teams app that uses Microsoft Teams Toolkit, Teams AI Library, and Azure AI Services to have an Open AI chatbot reason over Microsoft SharePoint content.

If you have not read the previous posts, I would suggest starting at the first post to get your bearings.

In the previous post, we setup the Azure AI and Azure Open AI infrastructure, so do that first.

Now we are getting to the exciting part which is where we create and set up the Teams AI Library application. This involves the following steps:

  • Create your Teams AI Library Project
  • Setup of Teams AI library.
  • Clone Teams AI Library Extension GitHub Repository.
  • Configure the environment variables.

As mentioned previously the solution is using an existing sample and modifying it to use the Teams AI Extension library that I have put together.

Create your Teams AI Library Project

The Microsoft Teams AI library GitHub repository can be found on GitHub and in this example will build on top of the Twenty Questions Bot and repurpose it for our needs.

The reason we are using this sample is that it gives us the scaffolding, I used the chef bot sample as well as inspiration for this sample which uses a local data source to augment its output.

I will be explaining how to setup the application using Microsoft Visual Studio Code.

So, to start, clone the Repository from within Microsoft Visual Studio Code.

Wait for the Git repository to be cloned and open the folder in Visual Studio Code.

Clone Teams AI Library Extension GitHub Repository

Next, get the code from the GitHub Repository.

Currently, I have not published the code to an NPM registry. I started to configure the code base to publish to a GitHub NPM package host but have not finished the setup yet.

So, clone the GitHub repository to your local machine. In the following example, we are cloning it into your c:\dev\teams-ai-library-extensions folder.

This can be done via Visual Studio Code by doing the following:

For details on how to do this using Git CLI, check out Cloning a repository – GitHub Docs.

Setup your Teams Application

Set up the Microsoft Teams application by doing the following.

With the Microsoft Teams AI Library sample Visual Studio code open do the following:

  1. In the root JavaScript folder, install and build all dependencies
    • cd teams-ai/js
    • yarn install
    • yarn build

  • In a terminal, navigate to the sample root.
cd teams-ai/tree/main/js/samples/04.e.twentyQuestions
  • Duplicate the sample.env in this folder. Rename the file to .env.
  • Add your bot’s credentials and any other related credentials to that file. We will need to use the Azure Open AI configuration. So, keep the AZURE_OPENAI_KEY, AZURE_OPENAI_ENDPOINT variables and fill them in appropriately.
  • Add the following new Environment Variables in to your .env file
    • AZURE_SEARCH_ENDPOINT=
    • AZURE_SEARCH_KEY=
    • AZURE_OPENAI_DEPLOYMENTMODEL=
    • AZURE_SEARCH_INDEXNAME=
    • BOT_ID=
    • BOT_PASSWORD=
  • Update config.json and index.ts with your model deployment name.

Install Teams AI Extension NPM Library

Now that you have your application downloaded with dependencies set up then we need to install the Teams AI Extension module using npm.

Based on that the step to clone the Teams AI Extension library has been cloned to c:\dev\teams-ai-extension then install the Teams AI Azure AI Search Extension into your ChefBot application by doing the following:

  • From VS Code.
  • Open Terminal
  • npm install c:\dev\teams-ai-library-extension\teams-ai-azure-ai-search-datasource –save

This will install the library and once installed we can reconfigure the application to use the Azure AI Search Data Source.

Update the index.ts file with the following steps.

Add the import statement.

import {AzureAISearchDataSource, AzureAISearchDataSourceOptions} from 'teams-ai-azure-ai-search-datasource';

At line 120 add the following code:

const dataSourceOptions: AzureAISearchDataSourceOptions = {

    azureOpenAiEndpoint: process.env.AZURE_OPENAI_ENDPOINT!,

    azureOpenAiKey: process.env.AZURE_OPENAI_KEY!,

    azureOpenAiDeploymentId: process.env.AZURE_OPENAI_DEPLOYMENTMODEL!,

    azureSearchEndpoint: process.env.AZURE_SEARCH_ENDPOINT!,

    azureSearchKey: process.env.AZURE_SEARCH_KEY!,

    azureSearchIndexName: process.env.AZURE_SEARCH_INDEXNAME!,  

};

At line 130 replace the code with this:

let dataSource:AzureAISearchDataSource = new AzureAISearchDataSource('teams-ai', dataSourceOptions);

planner.prompts.addDataSource(

    dataSource

);

This will remove the existing data source which is referencing a locally created vector database.

The last step is to update the prompts in Teams AI Library. The Teams AI library system prompt is augmented using prompts which are held in folders in the ./src/prompts folder.

Each folder within this prompts folder can hold a different configuration. The prompt folder that is being used by the application to configure the AI is managed by the action planner in this example.

You will see the following code:

const planner = new ActionPlanner({

    model,

    prompts,

    defaultPrompt: 'chat'

});

The defaultPrompt setting in this example uses the chat folder.

Now, let’s change the prompt to get the result that we want.

However, before we make any changes, let’s explain a bit about the files in this folder. There are two files to check out:

  • config.json
  • skprompt.txt

The config.json is used to configure the way that the prompt works and allows you to change how the interaction with the Azure Open AI Service. There is not a huge amount of information in the Teams AI Library. However, I have found that changing the type to completion improves the results.

Additionally, you may want to update the temperature. The temperature is a number between 0.0 and 1.0 and controls how precise or creative the results are. 0 is precise and 1 is very creative. The temperature will control whether you get a response as it might be that if you have a temperature of 0 and the bot does not know the answer then it will say so. If you have a temperature of 1.0 then you will always get a result back which will include results which are not factually correct.

The other file is skprompt.txt

This is used to configure your AI and augment the prompt to describe how you want it to behave. There are a lot of interesting ways to configure the prompt and I would recommend having a play but also using GitHub and see what other people are doing.

I have found that having a last sentence that states and uses the following information to formulate your response. For example:

The following is a conversation with an AI assistant called Mr Legal
Mr Legal is an expert in Legal and Law matters and is an assistant that is used by legal fee earners within law firms to help them understand how they can tackle their legal matters.
The voice of Mr Legal should be authorative and professional.

Base your response using the following text and include the citations:

Response Formatter Library

The [responseFormatter.ts](https://github.com/SimonDoy/teams-ai-library-samples/blob/main/teams-ai-app-azure-ai-search/src/responseFormatter.ts) libray is from the ChefBot sample and can be found in my the [Teams AI library sample repository](https://github.com/SimonDoy/teams-ai-library-samples/blob/main/teams-ai-app-azure-ai-search/src/responseFormatter.ts).

Copy and paste this into the index.ts file and add the line to line 25

import { addResponseFormatter } from './responseFormatter';

Add the line to line 145

addResponseFormatter(app);

Save the file and that should be the last coding change.

Configuration of Environment Variables

The environment variables are set in the env file which is found in the root of the project alongside the package.json. The environment variables should be updated using the following information:

  • AZURE_OPENAI_KEY – this is the access key found in your Azure Open AI Service settings.
  • AZURE_OPENAI_ENDPOINT – this is the URL that connects to your Azure Open AI Service.
  • AZURE_SEARCH_ENDPOINT – this is the URL that connects to your Azure AI Search Service.
  • AZURE_SEARCH_KEY – this is the access key found in your Azure AI Search Service settings.
  • AZURE_OPENAI_DEPLOYMENTMODEL – this is the name of the GPT deployment that has been created using the Azure Open AI portal
  • AZURE_SEARCH_INDEXNAME – This is the name of the search index that holds the content that your AI bot is going to use to reason over.

Let’s put it all together

Now, all being well your application will work!

So, start running the application using F5 to start Visual Studio Code deploying and debugging your code.

Wait for the application to spin up your infrastructure and bot service.

Eventually your browser will start and open up Microsoft Teams and prompt you to add the app.

Add the app to Microsoft Teams.

You should be presented with a chat. Use the chatbot and start asking for information that is related to the content that you have stored in the index.

Final Sample Code

To help you get your code running I have provided the Teams AI Library Sample Github Repository to allow you to review your code to the original.

Making tweaks

Now, you will want to try different prompts and tweak them to change the response that comes back from the AI platform.

Conclusion

I hope that you find this useful and can get this setup and working with your own data. The extension is not battle-hardened and I am sure there are plenty of issues that can come up with it but please raise issues via GitHub and if I get time, I will look.

One thing to be aware of is that the SharePoint Indexer for Azure AI Search Service is still in beta and there are issues when documents are deleted and documents are renamed as the index process does not pick up these changes, so be careful.

As an alternative the Azure Blob Storage Indexer has been released and has been made generally available.

I will look at putting in a bonus episode to explain how to configure the Azure Blob Storage Indexer but if you cannot wait, take a look at the Microsoft article, [Azure AI Search How to Index Azure Blob Storage](https://learn.microsoft.com/en-us/azure/search/search-howto-indexing-azure-blob-storage).

Happy code and let me know how you got on.

by Simon Doy

Teams AI Library Blog Series: Episode #2 – Teams AI Library App using Azure AI Search – Ready your Stack!


Introduction

This blog post is part of a series of posts describing the setup of a Microsoft Teams application that uses Team Toolkit, and Microsoft Teams AI library and extends it to allow you to deliver an application that can reason over SharePoint content using Azure Open AI Services.

If you have come to this and have not read the introduction post, then please starter there. It will give you more context on what we are trying to achieve.

As mentioned previously the solution will require the following components to be set up:

  • Azure Open AI Service
  • Azure AI Search
  • Setup of an Index for SharePoint content.

There is an assumption being made that you already have your SharePoint content setup that you wish to be reasoned over with your GPT model hosted in Azure Open AI.

Steps to delivery

The steps required to deliver the solution are as follows:

  • Setup of Azure Open AI Services.
  • Setup of Azure AI Search.
  • Setup of SharePoint Index.
  • Or optionally Setup of Azure Blob Storage Index

Setup of Azure AI Search

The Azure AI Search needs to be set first up, in this article we will configure it through the Microsoft Azure Portal.

Please check out the Create a search service in the portal – Azure AI Search | Microsoft Learn article for instructions on setting up the service.

Please note that depending on the region that we create the Azure AI Search within, then different regions have different functionality, you can use the Azure Products by Region | Microsoft Azure page to find out whether Azure AI Search has the correct features. I have had good success with France Central.

We want our Azure AI Search to be able to support the Semantic Ranking feature, since I set up the Azure AI Search service back in December, the rollout has continued and delivered this feature into most of the major Azure Data Centres now.

Setup Azure Open AI Service

The setup of the Azure Open AI Service requires that your subscription is enabled for Azure Open AI Services. This is to help manage the demand for AI services in the various Azure Data Centre. To request access you will need to fill out the Azure Open AI Service request form to get access.

Once you have filled out the form, it may take a few hours or so to receive a reply that your form has been processed.

Follow the instructions, How-to: Create and deploy an Azure OpenAI Service resource – Azure OpenAI | Microsoft Learn, to set up the Azure Open AI Services.

Setup of the Azure AI Search Index

The following guides will describe the process of setting up the search index for either SharePoint or Azure Blob Storage.

SharePoint Index Configuration

The SharePoint Index configuration uses the following process, SharePoint indexer (preview) – Azure AI Search | Microsoft Learn.

I have created a GitHub Repository, azure-ai-resources, https://github.com/SimonDoy/azure-ai-resources which has a Postman collection which will make it easier for you to set up your SharePoint indexer.

Please follow the instructions in the README, https://github.com/SimonDoy/azure-ai-resources/blob/main/azure-ai-search-postman/README.md

I want to highlight two things to make sure you have success when you set up the indexer.

  • The instructions discuss the permissions that your Microsoft Entra ID Application needs to have. Make sure you use Microsoft Graph and not SharePoint permissions.
  • Understand the limitations of SharePoint Indexer which is still in preview. It does not work well with changes to documents and document library names.
    • I prefer the Azure Blob Storage Indexer for this reason.

Once you have the search index setup, you can check its progress by browsing your Azure AI Search and checking the status of the index.

To do this:

  • Browse https://portal.azure.com
  • Find your Azure AI Search instance
    • I like to pin the Azure AI service to the left navigation using the All services menu item, search for Azure AI and click on the star to favourite.
  • Click on Indexers
  • Check status of your index using the name that you provided when setting up the index.

Next Steps

This article explained how to set up the Azure AI infrastructure for this solution. You have now got the base of your solution with an Azure Open AI Service, Azure AI Search Service, and set up the process for indexing the content in SharePoint.

The next steps are to start configuring the Teams AI Library application and bring in our Teams AI extension which will allow us to access the Azure AI Search Service and use that information to reason over the data in Azure AI Search.