Author / Simon Doy

Unknown's avatar

Posts by Simon Doy

I am an avid SharePoint enthusiast who works as an Independent SharePoint Consultant based in Leeds, United Kingdom. I am one of the organisers of the Yorkshire SharePoint User Group in the United Kingdom. I have been designing and building SharePoint solutions since 2006.
A user asking questions of a chat bot
by Simon Doy

My Experiences with Copilot Studio – Gen AI Agents Not Behaving as Expected? Check Your Question’s Settings


Introduction

At iThink 365, we are seeing an increase in the number of project where we are delivering Copilot Agents that are built on Microsoft 365 and Copilot Studio for our customers.

Each time we do one, we learn more and pick up tips and tricks on the way.

In this post, I wanted to share some learnings I have had as we try and do agents which are more agentic and use the orchestration features. I wanted to go through a scenario which I am sure you will have had, and some tips and tricks to help you get some control around.

Copilot Studio is going through a period of rapid change and development. New features are being rolled out each week, and it is constantly changing. Sometimes this leads us to think that something is buggy when in fact it is misunderstanding with how the platform can be configured.

In this blog post, I wanted to share an example of this.

The example is when you use the Question activity to get a response from the user.

One of the behaviours we saw was that the user would choose an option or type a response. On processing of the response, the agent would rather than continue in the same topic, it would switch and execute another topic (https://learn.microsoft.com/en-us/microsoft-copilot-studio/guidance/topics-overview). What the heck, why was it doing that?

Well, it turns out that the Copilot Studio’s Question activity has a plethora of settings. These settings allow you to control the processing logic, so let’s delve into those settings.

The first set of options is the “Question behaviour”, which controls how the question is processed.

For example:

  • Can a user skip this question?
  • Does the agent enforce the question, and if so, how many times does it try?

The third option is how the question treats interruptions.

The interruption was the secret to the problem we were seeing, more on that shortly. This set of options allows you to control whether a user can switch to a different topic and, if it can switch, which topic(s) the Agent can switch to.

The reasoning is sound, if you are a user, it may be pretty frustrating to have to go down a particular route and set of questions before you can get the answer you are looking for.

However, as a developer, there will be times when you want to ensure that the user cannot deviate from the process.

So, coming back to our problem, we were seeing that the user’s response was being seen by the Agent as a match for one of the other topics. The user experience was terrible because now they were taken off down a path that was not the expected path, and certainly did not end up with the result and experience that we wanted them to have.

The solution was to uncheck “Allow switching to another topic”, and the issue stopped happening.

Now, what if you did want them to be able to switch to another topic, but you wanted to restrict which topics they can switch to. Well, in order to do that, you can check “Allow switching to another topic” and then select the topics they can switch to.

You may think, how do we ensure that the user experience is what we want it to be. This is only validated via testing with real users. Often we have seen users use language or terms which we were not aware of and then topics are not triggered as expected.

So make sure you test this piece with real users via user acceptance testing before you launch. By doing this, you can ensure the configuration of the question and topics is right. Then the user gets the experience of picking up / switching to the topics with the right content.

Conclusion

I hope you uncover this post in your time of need, and it helps you uncover the plethora of options that the Question activity has.

Please let us know if you have any question or if it helped you using the comments below and thanks for reading.

A Shadowy figure in a foggy background.
by Simon Doy

Copilot Studio and Transforming Untyped Objects.


Introduction

The Copilot Studio experiences with UntypedObjects is a bit painful at the moment.

This article helps to show an approach that I took to be able to work with them in Copilot Studio when building Copilot Agents.

Firstly, let’s explain what an UntypedObject is. It is effectively a dynamic object which Copilot Studio does not know what the structure of it is. The object gains a dynamicProperty property which should allow you to read properties but within Copilot Studio currently that seems to go wrong.

The Problem

If you are working with data that Copilot Studio cannot resolve the type of, then you will be working with an UnTypedObject.

An example that I had was with Dataverse where I using a Environment variable to state which environment to connect to. This meant that the data that was being retrieved could not be resolved at design time and so it returned back a record of UntypedObject as shown in the screenshot below.

A screenshot of a computer AI-generated content may be incorrect.

I tried all sorts of different approaches to see if I could make use of the object but ultimately it was the dynamicProperties property that was causing problems as Copilot Studio seemed to want to use that to convert the data and of course my real object did not have that as a property.

In the end, I stumbled on a solution using the Parse JSON activity.

A Solution

I am hoping that there are nicer solutions coming from the Copilot Studio team but this is how I turned the UntypedObject into a Typed Object.

I tried a few different approaches, including using PowerFX Types to see if I could manage the conversion of Type. However, these ended up not working or hitting syntax error issues.

Examples of PowerFX functions

https://learn.microsoft.com/en-us/power-platform/power-fx/untyped-object

As I said previously, I then stumbled on the ParseJson activity.

Using the ParseJson Copilot Studio activity, along with a copy of the JSON string object, helped me get over the issues.

Here is what I did. First of all I outputted the whole table to a message using a Send Message activity, as shown below.

A screenshot of a computer AI-generated content may be incorrect.

Then, I copied an example of the Json and used that with the ParseJson activity.

A screenshot of a computer AI-generated content may be incorrect.

Here is the schema for the Parse JSON activity.

A screenshot of a computer AI-generated content may be incorrect.

The output of the ParseJson gave me a variable with a known shape.

Now that I have a variable with a known type, we are golden and everything is working as you would expect.

The data can be used as normal with other activities, such as Power Automate.

A screenshot of a computer AI-generated content may be incorrect.

Problem solved!

Conclusion

As we spend more time working on Copilot Studio solutions, I am sure these experiences will improve. Anyway, I hope you stumble on this and it helps you out.

I spent a few hours trying different things and ultimately found this solution.

Thanks for reading.