Category · 64 posts
ai
Agent 365: who's keeping an eye on your agents?
Agents are popping up everywhere. Makers build them in Copilot Studio, developers ship them on Foundry, SharePoint sites come with a ready-made one, and partners sell them in the store. That's great, until someone asks a simple question: how many agents do we have, who owns them, and what are they allowed to touch?
aiFrom WordPress.com to a static site on Azure
baeke.info ran on WordPress.com since 2011. That worked fine for a long time, but lately I write every post in Markdown, in a git repo, with Claude Code next to me. Getting that Markdown into WordPress meant converting it to block markup, uploading images through the admin page and checking that every block was valid.…
azureAutopilots: agents with their own mailbox, Teams account and boss
Most agents live behind a chat box. You open a web app or Copilot, type a question, get an answer and close the tab. That's fine for questions. It's less fine for work that arrives by email at 7am, needs a look at a PDF, and should be answered in the same thread before anyone…
aiAlways-on assistants that are asleep most of the day
I wanted an assistant that is always there. I write to it at 7am and the answer is ready at 7:01. It remembers what I told it last week. I don't have to start it, and I don't have to explain myself again.
aiBuilding Event-Driven Agents with Azure Functions
Microsoft now has several ways to build AI agents. You can create prompt agents in Foundry, build agentic workflows with Logic Apps, or deploy your own agent code as a Foundry hosted agent. There is another option that fits a different type of problem: the Azure Functions serverless agents runtime. This runtime lets you define…
aiHosting an AG-UI Agent on Microsoft Foundry
A while back I wrote about building an AI agent server with AG-UI and the Microsoft Agent Framework. That post walked through what AG-UI is, how you stand up a small agent server, and how a web client talks to it. Quick recap before we move on: AG-UI is a protocol between a frontend and…
aiBuilding an AI Agent Server with AG-UI and Microsoft Agent Framework
In this post, I want to talk about the Python backend I built for an AG-UI demo project. It is part of a larger project that also includes a frontend that uses CopilotKit: This post discusses the Python AG-UI server that is built with Microsoft Agent Framework. All code is on GitHub: https://github.com/gbaeke/agui. Most of…
aiCreating presentations with Microsoft Agent Framework and GAMMA API
Introduction Imagine automating the creation of professional presentations through a coordinated series of intelligent agents. In this blog post, we’ll explore how to build a workflow that combines research, outline generation, content creation, and design, all orchestrated through the Microsoft Agent Framework. We’ll build a practical example that generates presentations using: Azure OpenAI Responses API for…
aiDurable Execution for AI workflows
If you follow me on LinkedIn, you know I have talked about agentic workflows versus agents. Although everybody talks about agents, workflows are often better suited to many tasks. A workflow is more deterministic, is easier to reason about and to troubleshoot. Anthropic also talked about this a while ago in Building Effective Agents. In…
aiEnd-to-end authorization with Entra ID and MCP
Building an MCP server is really easy. Almost every language and framework has MCP support these days, both from a client and server perspective. For example: FastMCP (Python, Typescript), csharp-sdk and many more! However, in an enterprise scenario where users use a web-based conversational agent that uses tools on an MCP server, those tools might…
aiDeploying a multi-agent solution with MCP and A2A to Azure Container Apps
In previous posts, we discussed multi-agent scenarios, how A2A servers work (here and here) and how to deploy the infrastructure to host a multi-agent application on Azure with Azure Container Apps and AI Foundry. In this post, we will take a look at deploying the different components of the solution as containers in Azure Container…
aiDeploying AI Foundry Agents and Azure Container Apps to support an Agent2Agent solution
In previous posts, I discussed multi-agent solutions and the potential use of Google’s Agent2Agent protocol (A2A). In this post, we will deploy the infrastructure for an end-to-end solution like follows: Here’s a short description of the components. Component Description Foundry Project Basic Foundry project with a private endpoint. The private endpoint ensures private communication between…
aiUsing tasks with streaming in Google Agent2Agent (A2A)
In a previous post we created a simple A2A agent that uses synchronous message exchange. An A2A client sends a message and the A2A server, via the Agent Executor, responds with a message. But what if you have a longer running task to perform and you want to inform the client that the task in…
aiGoogle’s A2A: taking a closer look
In the previous post, I talked about options to build multi-agent solutions. The last option used Google’s A2A. A2A provides a wrapper around your agent, basically a JSON-RPC API, that standardizes how you talk to your agent. In this post we take a closer look at the basics of A2A with simple synchronous message exchange.…
aiBuilding multi-agent solutions: what are your options?
When we meet with customers, the topic of a “multi-agent solution” often comes up. This isn’t surprising. There’s a lot of excitement around their potential to transform business processes, strengthen customer relationships, and more. The first question you have to ask yourself though is this: “Do I really need a multi-agent solution?”. Often, we find…
aiBuilding Configurable AI Agents with the OpenAI Agents SDK
In this post, I will demonstrate how to build an AI agent system where agents can collaborate in different ways. The goal is to create agents that can either work independently with their own tools or collaborate with other agents through two distinct patterns: using agents as tools or handing off control to other agents.…
aiCreating an agent with Hugging Face smolagents and Azure OpenAI
Artificial Intelligence (AI) agents have garnered significant attention, with numerous posts discussing them on platforms such as LinkedIn and X/Twitter. In that sense, this post is not different. Instead of theory though, let’s look at building an agent that has a reasoning loop in a very simple way. Although you can build an agent from…
aiUsing Bing Search to ground LLM responses
We often get the question to build an assistant based on the content of a website. These assistants often get implemented in one of two ways: Turn-based chat assistant: user can ask a question and follow-up questions Enhanced search: user asks a questions without the option to ask follow-up questions; this is often used to…
aiCreating an agent with the Azure AI Agent SDK
Azure AI Agents Service simplifies building intelligent agents by combining advanced AI models, tools, and technology from Microsoft, OpenAI, and partners like Meta and Cohere. It enables integration with knowledge sources such as Bing, SharePoint, and Azure AI Search, and lets agents perform actions across Microsoft and third-party applications using Logic Apps, Azure Functions, and…
aiUsing WebRTC with the OpenAI Realtime API
In October 2024, OpenAI introduced the Realtime API. It enables developers to integrate low-latency, multimodal conversational experiences into their applications. It supports both text and audio inputs and outputs, facilitating natural speech-to-speech interactions without the need for multiple models. It addresses the following problems: Simplified Integration: Combines speech recognition, language processing, and speech synthesis into…
aiUsing the Azure AI Inference Service
If you are a generative AI developer that works with different LLMs, it can be cumbersome to make sure your code works with your LLM of choice. You might start with Azure OpenAI models and the OpenAI APIs but later decide you want to use a Phi-3 model. What do you do in that case?…
aiCreate a Copilot declarative agent that calls an API with authentication
In a previous post, we looked at creating a Copilot declarative agent. The agent had one custom action that called the JSONPlaceholder API. Check that post for an introduction to what these agents can do. Using a dummy, unauthenticated API is not much fun so let’s take a look at doing the same for a…
aiCreating a Copilot declarative agent with VS Code and the Teams Toolkit
If you are a Microsoft 365 Copilot user, you have probably seen that the words “agent” and “Copilot agent” are popping up here and there. For example, if you chat with Copilot there is an Agents section in the top right corner: Above, there is a Visual Creator agent that’s built-in. It’s an agent dedicated…
aiUsing your own message broker with Diagrid Catalyst
In a previous post, I wrote about Diagrid Catalyst. Catalyst provides services like pub/sub and state stores to support the developer in writing distributed applications. In the post, we discussed a sample application that processes documents and extracts fields with an LLM (gpt-4o structured extraction). Two services, upload and process, communicate via the pub/sub pattern.…
aiWriting an multi-service document extractor with the help of Diagrid’s Catalyst
Many enterprises have systems in place that take documents, possibly handwritten, that contain data that needs to be extracted. In this post, we will create an application that can extract data from documents that you upload. We will make use of an LLM, in this case gpt-4o. We will use model version 2024-08-06 and its…
aiToken consumption in Microsoft’s Graph RAG
In the previous post, we discussed Microsoft’s Graph RAG implementation. In this post, we will take a look at token consumption to query the knowledge graph, both for local and global queries. Note: this test was performed with gpt-4o. A few days after this blog post, OpenAI released gpt-4o-mini. Initital tests with gpt-4o-mini show that…
aiTrying out Microsoft’s Graph RAG
Whenever we build applications on top of LLMs such as OpenAI’s gpt-4o, we often use the RAG pattern. RAG stands for retrieval augmented generation. You use it to let the LLM answer questions about data it has never seen. To answer the question, you retrieve relevant information and hand it over to the LLM to…
aiEmbracing the Age of AI Transformation: A New Era for Innovation and Expertise
⚠️ This is an AI-generated article based on a video to transcript generator I created to summarise Microsoft Build sessions. This article is used as an example for a LinkedIn post. This article is based on the Microsoft Build keynote delivered on Tuesday, May 21st, 2024. It was created with gpt-4o. The post is unedited…
aiLoad balancing OpenAI API calls with LiteLLM
If you have ever created an application that makes calls to Azure OpenAI models, you know there are limits to the amount of calls you can make per minute. Take a look at the settings of a GPT model below: Above, the tokens per minute (TPM) rate limit is set to 60 000 tokens. This…
aiUse Azure OpenAI on your data with Semantic Kernel
I have written before about Azure OpenAI on your data. For a refresher, see Microsoft Learn. In short, Azure OpenAI on your data tries to make it easy to create an Azure AI Search index that supports advanced search mechanisms like vector search, potentially enhanced with semantic reranking. On of the things you can do…
aiSo you want a chat bot to talk to your SharePoint data?
It’s a common request we hear from clients: “We want a chatbot that can interact with our data in SharePoint!” The idea is compelling – instead of relying on traditional search methods or sifting through hundreds of pages and documents, users could simply ask the bot a question and receive an instant, accurate answer. It promises to be a much more efficient and user-friendly experience. The appeal is clear: Improved user experience Time savings Increased productivity But…
aiEmbedding flows created with Microsoft Prompt Flow in your own applications
A while ago, I wrote about creating your first Prompt Flow in Visual Studio Code. In this post, we will embed such a flow in a Python application built with Streamlit. The application allows you to search for images based on a description. Check the screenshot below: There are a few things we need to…
aiRetrieval with the Azure OpenAI Assistants API
In two previous blog posts, I wrote an introduction to the Azure OpenAI Assistants API and how to work with custom functions. In this post, we will take a look at an assistant that can answer questions about documents. We will create an HR Assistant that has access to an HR policy document. In addition,…
aiUsing tools with the Azure OpenAI Assistants API
Introduction In a previous blog post, I wrote an introduction about the Azure OpenAI Assistants API. As an example, I created an assistant that had access to the Code Interpreter tool. You can find the code here. In this post, we will provide the assistant with custom tools. These custom tools use the function calling…
aiA look at the Azure OpenAI Assistants API
Introduction A while ago, I looked at the OpenAI Assistants API. In February of 2024, Microsoft have released their Assistants API in public preview. It works in the same way as the OpenAI Assistants API while being able to use it with Azure OpenAI models, deployed to a region of your choice. The goal of…
aiDeploy a flow created in Prompt Flow with Docker
Update: this post used an older version of Prompt Flow at the time. It had some issues with building and running the Docker image without issues. In version 1.5.0, it should work fine because the Dockerfile now also installs gcc. In the previous post, we created a flow with Prompt Flow in Visual Studio Code.…
aiWriting your first flow with Prompt Flow in Visual Studio Code
In this blog post, we will create a flow with Prompt Flow in Visual Studio Code. Prompt Flow is a suite of development tools to build LLM-based AI applications. It tries to cover the end-to-end development cycle, including prototyping, testing and deployment to production. In Prompt Flow, you create flows. Flows link LLMs (large language…
aiBuilding an Azure AI Search index with a custom skill
In this post, we will take a look at building an Azure AI Search index with a custom skill. We will use the Azure AI Search Python SDK to do the following: create a search index: a search index contains content to be searched create a data source: a datasource tells an Azure AI Search…
aiFinding images with text and image queries with the help of GPT-4 Vision
With the gpt-4-vision-preview model available at OpenAI, it was time to build something with it. I decided to use it as part of a quick solution that can search for images with text, or by providing a similar image. We will do the following: Describe a collection of images. To generate the description, GPT-4 Vision…
aiUsing Integrated Vectorization in Azure AI Search
The vector search capability of Azure AI Search became generally available mid November 2023. With that release, the developer is responsible for creating embeddings and storing them in a vector field in the index. However, Microsoft also released integrated vectorization in preview. Integrated vectorization is useful in two ways: You can define a vectorizer in…
aiCreating a custom GPT to query any knowledge base with actions
A while ago, OpenAI introduced GPTs. A GPT is a custom version of ChatGPT that combine instructions, extra knowledge, and any combination of skills. In this tutorial, we are going to create a custom GPT that can answer questions about articles on this blog. In order to achieve that, we will do the following: create…
aiTrying the OpenAI Assistants API
If you have ever tried to build an AI assistant, you know that is not a simple task. In almost all cases, your assistant needs access to external knowledge such as documents or APIs. You might even want to provide your assistant a code sandbox to solve user queries with code. When your assistant is…
aiUsing Azure Database for PostgreSQL as a vector store
When we build LLM applications, there is always a recurring question: “What vector store will we use?”. In Azure, there are several native solutions. Some of them were discussed in previous posts. Azure Cognitive Search: supports vector search but also hybrid search with semantic reranking as discussed here Azure Redis Cache Enterprise: check my blog…
aiUse Azure OpenAI Add your data vector search from code
In the previous post, we looked at using Azure OpenAI Add your data from the Azure OpenAI Chat Playground. It is an easy-to-follow wizard to add documents to a storage account and start asking questions about them. From the playground, you can deploy a chat app to an Azure web app and you are good…
aiImprovements in Azure OpenAI Add your data
In a previous post, I talked about the Add your data feature in the Azure OpenAI Chat playground. Recently, there have been some updates to this feature, including vector search. Let’s take a look at the updated experience and focus on vector search. Starting point I have some PDF documents in a storage account container.…
aiUse the Azure Add your data feature from code
In a previous post, we looked at Azure Add your data (preview) that can be configured from the Azure OpenAI Chat playground. In a couple of steps, you can point to or upload files of a supported file type (PDF, Word, …) and create an Azure Cognitive Search index. That index can subsequently be queried…
aiBuilding a chatbot in Azure that works with your data
When I talk to customers about Azure OpenAI, I am often asked how to build a chatbot that uses your own data in the simplest way possible while still allowing for some customization. In Azure, there are a few solutions. We will look at one of the solutions in this post. Note: don’t feel like…
aiSemantic Kernel Planner 101
Introduction If you are a developer who wants to build AI-first apps with natural language processing and large language models, you might be interested in Semantic Kernel (SK), a lightweight and open-source SDK that aims to simplify the integration of AI with conventional programming languages. SK is part of the CoPilot Stack and Microsoft is…
aiEnhancing Semantic Search with a Streamlit UI
In a previous blog post, we discussed two Python programs, upload_vectors.py and search_vectors.py. These programs were used to create and search vectors, respectively. The upload_vectors.py script created vectors from chunks of a larger text and stored them in Pinecone, while the search_vectors.py script enabled semantic search on the text. In this blog post, we will…
aiEnhancing Blog Post Search with Chunk-based Embeddings and Pinecone
In this blog post, we’ll show you a different approach to searching through a large database of blog posts. The previous approach involved creating a single embedding for the entire article and storing it in a vector database. The new approach is much more effective, and in this post, we’ll explain why and how to…
aiStoring and querying for embeddings with Redis
In a previous post, we wrote about using vectorized search and cosine similarity to quickly query a database of blog posts and retrieve the most relevant content to a natural language query. This is achieved using OpenAI’s embeddings API, Pinecone (a vector database), and OpenAI ChatCompletions. For reference, here’s the rough architecture: The steps above…
aiPinecone and OpenAI magic: A guide to finding your long lost blog posts with vectorized search and ChatGPT
Searching through a large database of blog posts can be a daunting task, especially if there are thousands of articles. However, using vectorized search and cosine similarity, you can quickly query your blog posts and retrieve the most relevant content. In this blog post, we’ll show you how to query a list of blog posts…
aiStep-by-Step Guide: How to Build Your Own Chatbot with the ChatGPT API
In this blog post, we will be discussing how to build your own chat bot using the ChatGPT API. It’s worth mentioning that we will be using the OpenAI APIs directly and not the Azure OpenAI APIs, and the code will be written in Python. A crucial aspect of creating a chat bot is maintaining…
aiCreating and deploying a model with Azure Machine Learning Service
In this post, we will take a look at creating a simple machine learning model for text classification and deploying it as a container with Azure Machine Learning service. This post is not intended to discuss the finer details of creating a text classification model. In fact, we will use the Keras library and its…
aiCreating and containerizing a TensorFlow Go application
In an earlier post, I discussed using a TensorFlow model from a Go application. With the TensorFlow bindings for Go, you can load a model that was exported with TensorFlow’s SavedModelBuilder module. That module saves a “snapshot” of a trained model which can be used for inference. In this post, we will actually use the…
aiVirtual Node support in Azure Kubernetes Service (AKS)
Although I am using Kubernetes a lot, I didn’t quite get to trying the virtual nodes support. Virtual nodes is basically the implementation on AKS of the virtual kubelet project. The virtual kubelet project allows Kubernetes nodes to be backed by other services that support containers such as AWS Fargate, IoT Edge, Hyper.sh or Microsoft’s…
aiMicrosoft Face API with a local container
A few days ago, I obtained access to the Face container. It provides access to the Face API via a container you can run where you want: on your pc, at the network edge or in your datacenter. You should allocate 6 GB or RAM and 2 cores for the container to run well. Note…
aiRunning a GoCV application in a container
In earlier posts (like here and here) I mentioned GoCV. GoCV allows you to use the popular OpenCV library from your Go programs. To avoid installing OpenCV and having to compile it from source, a container that runs your GoCV app can be beneficial. This post provides information about doing just that. The following GitHub…
aiUsing the Microsoft Face API to detect emotions in photos and video
⚠️ IMPORTANT: the Face API container was retired early 2021. The container image is not available anymore. In a previous post, I blogged about detecting emotions with the ONNX FER+ model. As an alternative, you can use cloud models hosted by major cloud providers such as Microsoft, Amazon and Google. Besides those, there are many…
aiDetecting emotions with FER+
In an earlier post, I discussed classifying images with the ResNet50v2 model. Azure Machine Learning Service was used to create a container image that used the ONNX ResNet50v2 model and the ONNX Runtime for scoring. Continuing on that theme, I created a container image that uses the ONNX FER+ model that can detect emotions in…
aiResNet50v2 classification in Go with a local container
To quickly go to the code, go here. Otherwise, keep reading… In a previous blog post, I wrote about classifying images with the ResNet50v2 model from the ONNX Model Zoo. In that post, the container ran on a Kubernetes cluster with GPU nodes. The nodes had an NVIDIA v100 GPU. The actual classification was done…
aiRecognizing images with Azure Machine Learning and the ONNX ResNet50v2 model
Featured image from: https://medium.com/comet-app/review-of-deep-learning-algorithms-for-object-detection-c1f3d437b852 In a previous post, I discussed the creation of a container image that uses the ResNet50v2 model for image classification. If you want to perform tasks such as localization or segmentation, there are other models that serve that purpose. The image was built with GPU support. Adding GPU support was pretty…
aiCreating a GPU container image for scoring with Azure Machine Learning
In a previous post, I discussed how you can add an existing Kubernetes cluster to an Azure Machine Learning workspace. Adding an existing cluster is necessary when the workspace does not support auto creation of a cluster. That is the case when you want to use the Standard_NC6s_v3 virtual machine image. I also used a…
aiAttaching Kubernetes clusters with NVIDIA V100 GPUs to Azure Machine Learning Service
Azure Machine Learning Service allows you to easily deploy compute for training and inference via a machine learning workspace. Although one of the compute types is Kubernetes, the workspace is a bit picky about the node VM sizes. I wanted to use two Standard_NC6s_v3 instances with NVIDIA Tesla V100 GPUs but that was not allowed.…
ai