Tag / Microsoft 365 Copilot

A user asking questions of a chat bot
by Simon Doy

My Experiences with Copilot Studio – Gen AI Agents Not Behaving as Expected? Check Your Question’s Settings


Introduction

At iThink 365, we are seeing an increase in the number of project where we are delivering Copilot Agents that are built on Microsoft 365 and Copilot Studio for our customers.

Each time we do one, we learn more and pick up tips and tricks on the way.

In this post, I wanted to share some learnings I have had as we try and do agents which are more agentic and use the orchestration features. I wanted to go through a scenario which I am sure you will have had, and some tips and tricks to help you get some control around.

Copilot Studio is going through a period of rapid change and development. New features are being rolled out each week, and it is constantly changing. Sometimes this leads us to think that something is buggy when in fact it is misunderstanding with how the platform can be configured.

In this blog post, I wanted to share an example of this.

The example is when you use the Question activity to get a response from the user.

One of the behaviours we saw was that the user would choose an option or type a response. On processing of the response, the agent would rather than continue in the same topic, it would switch and execute another topic (https://learn.microsoft.com/en-us/microsoft-copilot-studio/guidance/topics-overview). What the heck, why was it doing that?

Well, it turns out that the Copilot Studio’s Question activity has a plethora of settings. These settings allow you to control the processing logic, so let’s delve into those settings.

The first set of options is the “Question behaviour”, which controls how the question is processed.

For example:

  • Can a user skip this question?
  • Does the agent enforce the question, and if so, how many times does it try?

The third option is how the question treats interruptions.

The interruption was the secret to the problem we were seeing, more on that shortly. This set of options allows you to control whether a user can switch to a different topic and, if it can switch, which topic(s) the Agent can switch to.

The reasoning is sound, if you are a user, it may be pretty frustrating to have to go down a particular route and set of questions before you can get the answer you are looking for.

However, as a developer, there will be times when you want to ensure that the user cannot deviate from the process.

So, coming back to our problem, we were seeing that the user’s response was being seen by the Agent as a match for one of the other topics. The user experience was terrible because now they were taken off down a path that was not the expected path, and certainly did not end up with the result and experience that we wanted them to have.

The solution was to uncheck “Allow switching to another topic”, and the issue stopped happening.

Now, what if you did want them to be able to switch to another topic, but you wanted to restrict which topics they can switch to. Well, in order to do that, you can check “Allow switching to another topic” and then select the topics they can switch to.

You may think, how do we ensure that the user experience is what we want it to be. This is only validated via testing with real users. Often we have seen users use language or terms which we were not aware of and then topics are not triggered as expected.

So make sure you test this piece with real users via user acceptance testing before you launch. By doing this, you can ensure the configuration of the question and topics is right. Then the user gets the experience of picking up / switching to the topics with the right content.

Conclusion

I hope you uncover this post in your time of need, and it helps you uncover the plethora of options that the Question activity has.

Please let us know if you have any question or if it helped you using the comments below and thanks for reading.

An image of person thinking about whether to use which AI Search tool
by Simon Doy

Knowledge Copilot Agents – To use or not to use Azure AI Search?


Introduction

Updated: 14th January 2025 – The guidance has been updated by Microsoft and the Enhanced Search which uses Copilot Semantic Index does not require every user to have a Microsoft 365 Copilot license.

https://learn.microsoft.com/en-us/microsoft-copilot-studio/knowledge-copilot-studio

This article discusses some of the things to consider when deciding whether to use Azure AI Search as the data source for Copilot Agents.
The AI world is moving quickly, and Microsoft is also moving quickly with constant updates and changes to its propositions.

However, one of the issues that we have found at iThink 365 when building Knowledge Copilots for our customers is that the quality of the Copilot and the user experience are directly related to the quality of the search results underpinning it.

Generally, we have used Azure AI Search as our search engine which we have seen has provided dramatically better results for our customers. The reason is the type of search technology underpinning the Copilot. Azure AI Search is built for AI workloads and can be configured to use several different search technologies including Semantic, Keyword, Vector and a hybrid of these search technologies.

However, the use of Azure AI Search does increase the cost of running the Copilot because you are paying for Copilot Studio, Azure Open AI and Azure AI Search resources. Whilst some people would say that this is not worth it due to the increase in cost, it really is the only solution for getting the results that customers expect.

The reason for the Azure AI Search approach is that the base Copilot Studio uses the basic Microsoft 365 search engine when Copilot is accessing knowledge held in SharePoint. This experience uses Keyword based searches to give results and we have found that the experience for customers has not been as good.

However, it is important to keep up to date with Microsoft and their changes. Microsoft have been listening to the feedback from their customers, MVPs and partners and so at Ignite 2024 they announced the introduction of Copilots built on Copilot Studio to use the Microsoft 365 Copilot Semantic Index. This has the potential for really improving the experience for users and brings the experience more inline with what we have seen with Azure AI Search.

There is something to be aware of, to be able to take advantage of the Microsoft 365 Copilot Semantic Index your organisation and users need to be licensed for Microsoft 365 Copilot.
Without the licenses, your Knowledge Copilot will not be able to use the premium search capabilities and therefore Copilot Semantic Index.


 
Questions to help make your decision

So, our decision on whether to use Azure AI Search or not is based on the following questions:

  • Does one user have a Microsoft 365 Copilot license?
  • Are the documents being accessed with knowledge in them, either PDF, DOCX or PPTX and are they less than 512MB in size?
  • Is all the information that needs to be accessed held in supported documents in Microsoft SharePoint?

If the answers to these questions are yes, then my suggestion is to do a proof of concept using Copilot Studio and the premium search experience and test to see if the user experience is good enough so you do not need to use Azure AI Search.

However, if the answer is no to either or following testing the user experience then I would recommend using Azure AI Search to give our customers the experience that they deserve.

Conclusion

I hope that this helps you understand when to use which approach with your Microsoft Copilot solutions, in particular those with Knowledge Copilot and Copilot Agents.