Tag / microsoft

An illustration of an airplane being flown from the position of the cockpit.
by Simon Doy

Gotchas discovered building a Custom Engine Copilot with GPT-4o and Copilot Studio


Introduction

This article highlights some gotchas that I have hit when building a Copilot Studio with a Custom Engine Copilto using GPT-4o. The aim is to help you solve these problems if you have similar issues.

So, firstly what are we talking about when we talk about Custom Engine Copilots?

Well, Copilot Studio can be configured to use an external hosted AI model for example using Azure AI Services and GPT-4o. This allows us to use a more powerful or more suitable language model such as GPT-4o instead of the out-of-the-box LLM that Microsoft currently provide.

The benefits are better reasoning with better results. Our experience with our customers has shown some great results when using GPT-4o.

The way of using a custom engine Copilot is using the Generative Answers capability within Copilot Studio.

However, there are some gotchas when using these more complex models and I wanted to document them here to save you working out what the issue is.

Gotcha 1: Generative Answers returns no knowledge found

So, we have seen that if something goes wrong when you using Open AI Services then you get a no knowledge found.

You can try this out using the Test Your Copilot feature for your Copilot with Copilot Studio.

I will be honest this took a while to find out what the issue was but by using Azure Open AI Services https://oai.azure.com/ you can test the model to make sure it is working with your data.

We kept getting issues with Generative Answers saying there was no knowledge found. In the end, it turned out to be due to a trailing slash missing for the Azure AI Search endpoint.

So check your Open AI connection settings, make sure that you have a trailing slash on the Azure AI Search / Cognitive Search endpoint URL.

i.e https://azureaisearch.search.windows.net/

and not https://azureaisearch.search.windows.net

We have also seen the issue with your model being throttled and the result is that you get the same no information was found that could help answer this.

When you try the same prompt from Azure Open AI Services you get this error message, Server responded with status 429, the rate limit is exceeded.

Make sure you have increased the rate limit to cover the number of tokens that need to be processed.

You can do this using the Azure Open AI Studio by going to the Deployments, choosing your model and then editing the model settings and increasing the Tokens Per Minute Rate Limit. For testing we are setting this to 100K but for Production, you are likely to need to increase further.

Gotcha 2: Generative Answers returns answers but they are not that great.

This issue is subtle and is unfortunately hidden by the Generative Answers. The experience that we were getting was that using Azure Open AI Services we got really good detailed responses back. However, when we tried the same prompt in Copilot Studio we got very simple responses back which were nowhere as good as those from Azure Open AI Services.

The issue turned out to be related to Gotcha 1 where we were getting no results back from the Open AI model and we had this option switched on in the Generative Answers action. So then the Generative Answers would use the knowledge that it has in its model.

So we would get a response like this one

Which is not bad but not as good as the GPT-4o version which is shown below.

So the fix is to switch off the “Allow the AI to use its own general knowledge” option.

Gotcha 3: Generative Answers sometimes return great answers and sometimes errors out.

So this issue seems to occur with GPT-4o models but not GPT-4 based models and I suspect that this is down to the amount of detail in the answers coming from the model.

When using Generative Answers and Copilot Studio you can return the information back to the user in two ways:

  • Ask Generative Answers to send a message to the user.
  • Take the response and assign it to a variable.

These options can be found in the Advanced section of the action.

If you ask for generative answers to send a message then you sometimes get errors being reported.

Instead do the following:

  • Assign the response from the model into a variable, use Text Only.
  • Check to see if a response is returned and then if it is write out the message using a Send a message activity.

See the following screenshot:

Once you have assigned the LLM response to the variable then add the condition and do the following:

You will find the responses much more reliable.

Conclusion

In this blog post, I explain some of the issues/gotchas that I have seen when building Custom Engine Copilots using GPT-4o. We covered some of the issues that I have seen and provided ways to solve them.

I hope that helps!

if you need a hand then get in touch with us at iThink 365, https://www.ithink365.co.uk.

by Simon Doy

Manage and deliver your 90-Day Action Coach Success Plan with Microsoft 365 – Part Two


Introduction

If you are working with an Action Coach Business Coach like Iain Strachan as I am you will be used to the 90-day strategy planning session.

In these 90-day planning sessions, we define 4 Goals which are 3 business goals and one personal goal and build a plan on how to achieve those goals over the next 90 days.

This is the second post in a series on how you manage, track and deliver your 90 day Action Coach Success Plan, if you have not read the first post, then start there.

Viva Goals – The Key To Tracking Your Goals and Metrics

Viva Goals is part of the Microsoft Viva suite, allowing businesses to define objectives and key results. These OKRS are goals that are measured using key results to track your progress and define when the objective has been met.

You can access Viva Goals through the web address, https://goals.cloud.microsoft/.

Business leaders need to be able to bring their teams on the journey with them and have a way to communicate and track the progress of business strategy initiatives. Viva Goals is a great way to achieve this.

With Viva Goals, you keep the progress of your OKRs up to date by you and your teams checking in.

The frequency of check-in is configurable but to keep in line with Action Coach success pack delivery you would check in once a week to update your goals progress.

However, one of the challenges with OKRs is the process of keeping the data up to date and correct.

First of all you can start manually entering the stats you are measuring but you really want to make sure that it is easy to update, track and monitor.

With the initial version of Viva Goals, there were some integrations that allowed the check-ins to take place automatically but they were a little limited.

Fast forward to today and now the options are much more interesting and there are two integrations that we will focus on today.

The Viva Goals team have enabled Power BI, Planner and Project for the web integration. The Power BI integration is really a game changer because through Power BI we can integrate with a huge number of different data sources automatically by linking Power BI visualisations to the Viva Goals key result.

The important aspect of Power BI is that you create the Power BI with the visualisations which display the metrics that we wish to measure. These measurements are key to when you are building your 90 strategy plan, in your workshop you will be asked how you would measure the goal and track whether it has been achieved.

There are a couple of options for how you would track these metrics, you can manually track them but if you really want to automate and track these metrics then you should look at ways to pull this data from your business systems.

An example might be the net profit margin figure which you could manually enter which is fine, however, if you want to make life easier then you can automate it by pulling from Xero into Power BI through some integration.

When businesses are looking to achieve their goals there is often activities that need to be planned and tracked. Traditionally with the Action Coach this is done via a Microsoft Excel spreadsheet which is fine but often I would find that keeping in mind what the activities are and what was to do this week would require me to keep checking on the Excel spreadsheet.

However, with Viva Goals and Planner integration, it is possible to make the activities that go along with the goal part of your key result that supports the objectives.

You can create an initiative and then link that initiative to Planner. In the previous section, we talked about how we create a bucket for each objective/goal and using an initiative we can link the objective to that bucket in Planner and see the progress of the tasks that are taking place.

Let’s delve into how we might set this all up, via this video.

How do you get your data into Power BI?

Well this is the key to it all.

When you start, I suggest keeping it simple and tracking your key results metrics manually. As you build your capability you can start to look to automate it as shown in the video.

Of course, we at iThink 365, would love to help you get your data, Power BI, Viva Goals and Planner setup and give you something that you can then maintain and enhance.

Conclusion

By combining these Microsoft 365 services together, you can build a really powerful and excellent tool that will help you manage, track and deliver your 90-day plan.

I wish you good luck with your 90-day planning!

Let us know if this helped you and reach out if we can help you!