WEBVTT 00:00.360 --> 00:05.840 And welcome to week five, day two, when we get deeper into Autogen agent chat. 00:06.160 --> 00:09.680 So as a reminder, I'd like to show these things a couple of times. 00:09.720 --> 00:13.640 Autogen which is multiple different things. 00:13.680 --> 00:17.880 There's Autogen core, which is the kind of infrastructure for agents interacting. 00:18.040 --> 00:23.440 There is Autogen agent chat, which is a thing that's quite like crew and OpenAI agents SDK. 00:23.640 --> 00:28.280 And then built on top of that are things we will not be looking at the no code, low code platform studio 00:28.480 --> 00:31.840 and their particular tool Magentic one. 00:32.000 --> 00:36.800 But we are going to be talking about Autogen and today it's more on Autogen agent chat. 00:37.160 --> 00:41.680 And you remember we talked about the core concepts of models and messages and agents and we got to see 00:41.680 --> 00:41.920 them all. 00:41.960 --> 00:43.640 And we didn't actually talk about teams. 00:44.080 --> 00:46.480 Today we're going to take it a little bit deeper. 00:46.560 --> 00:49.320 We always like to take it one more step deeper. 00:49.320 --> 00:52.920 But most of this is going to be concepts pretty familiar to you. 00:52.960 --> 00:55.240 We're going to talk about going multi-modal. 00:55.520 --> 00:56.800 Uh, that's that's I guess new. 00:56.800 --> 00:57.920 We haven't really done that before. 00:57.920 --> 00:59.880 So that's going to be a new one to try out. 00:59.920 --> 01:04.710 We're going to talk about structured outputs, which is an old chestnut that you know well. 01:05.030 --> 01:07.110 We're going to look at something interesting. 01:07.350 --> 01:14.110 We can actually wrap tools in long chain so that we can call long chain tools from Autogen, which is 01:14.110 --> 01:18.110 going to be super convenient, especially as we're quite experienced with long chain tools now. 01:18.150 --> 01:22.790 And because there's an enormous ecosystem of so many tools to choose from. 01:22.790 --> 01:25.870 So this is quite a luxury to have access to this. 01:26.270 --> 01:29.150 And then since we put teams up there, we better look at teams. 01:29.150 --> 01:31.870 So we're going to give that a quick whirl as well. 01:32.750 --> 01:35.390 And then there's a special guest entry as well. 01:35.390 --> 01:39.790 Something else just to put out there that I will show you that might entertain you. 01:39.830 --> 01:40.870 All right that's enough. 01:40.870 --> 01:41.750 Let's go do it. 01:42.150 --> 01:42.670 Okay. 01:42.710 --> 01:43.710 So we are in. 01:43.990 --> 01:45.430 So we're in week five. 01:45.470 --> 01:48.390 We're going to the second lab for the second day. 01:48.390 --> 01:49.630 Week five, day two. 01:50.070 --> 01:51.190 Autogen agent chat. 01:51.230 --> 01:52.430 Going deeper. 01:52.550 --> 01:58.870 So I've got a bunch of imports, including I've shoved load dot env into the imports as well so we can 01:58.870 --> 02:00.150 get on with stuff. 02:00.470 --> 02:00.830 All right. 02:00.830 --> 02:07.340 So I'm going to start by showing you how you can have like a multi modal conversation Conversation which 02:07.340 --> 02:12.700 isn't just about text, but you can send pictures along with the text and have that be part of it. 02:13.260 --> 02:21.180 So we've got here that we've got a URL, and that URL is linking to an image from my website. 02:21.460 --> 02:27.980 And then we're going to open that image using a very standard Python images library. 02:27.980 --> 02:30.700 The Python image library is what Pil stands for. 02:30.980 --> 02:37.740 And we're going to to create an autogen image from that and then take a look at it. 02:37.740 --> 02:38.540 So here it is. 02:38.540 --> 02:44.660 It's a cool picture that's meant to be evocative of sort of going into the world of AI. 02:45.660 --> 02:48.660 So we're now going to create a multi-modal message. 02:48.700 --> 02:50.660 So you remember in the past we've had text message. 02:50.780 --> 02:52.300 Well, this is analogous. 02:52.300 --> 02:57.500 It's a multi-modal message and it has content which has a list. 02:57.540 --> 03:01.660 It has like a bit of text to describe the content of this image in detail. 03:01.820 --> 03:05.020 And then it's got the image and the source is user. 03:05.020 --> 03:05.780 That would be me. 03:06.100 --> 03:11.530 So just running that is all it takes to create a multi-modal message. 03:11.530 --> 03:17.010 And then sending that to GPT four or mini is just as simple as doing exactly what we did before. 03:17.010 --> 03:21.770 We create the model client and then we create an assistant agent. 03:21.770 --> 03:26.370 I'm calling it the describer or the real name description agent. 03:26.850 --> 03:31.930 The model client you pass in, of course, which is the LLM, the system message. 03:31.930 --> 03:33.930 You are good at describing images. 03:33.930 --> 03:34.570 Fair enough. 03:34.930 --> 03:38.610 And then uh, then the user prompt, of course is right there. 03:38.850 --> 03:46.290 And then the we pass in that multimodal message and we also just say the cancellation token. 03:46.530 --> 03:51.330 And then we will print that markdown version of the reply. 03:51.490 --> 03:56.930 And this will take, take a little while because it's got to take in that image and it's got to get 03:56.930 --> 03:59.930 back all of the response from it. 03:59.930 --> 04:04.810 So I have to let this sentence run on quite a bit to see what we get back. 04:04.810 --> 04:05.530 And here it is. 04:05.930 --> 04:06.970 Uh, not too bad. 04:07.170 --> 04:09.330 Uh, so you'll see that it has worked. 04:09.570 --> 04:14.080 Uh, the image depicts a brightly colored, stylized space that combines elements of a workspace with 04:14.080 --> 04:16.600 whimsical, otherworldly doorway. 04:17.080 --> 04:20.920 And then we've got, of course, because models are great at markdown, that's what they love. 04:20.960 --> 04:23.760 That's that's one of the languages they speak natively. 04:24.480 --> 04:27.760 You get the room setting, furniture, doorway, decorative elements. 04:28.240 --> 04:32.800 The overall, the image cleverly intertwines a workspace focused on technology and coding with imaginative 04:32.800 --> 04:37.880 and vibrant elements that evoke the limitless possibilities of AI. 04:38.200 --> 04:39.520 So it definitely got the joke. 04:39.560 --> 04:44.600 It understood what the image was about, and it did a fine job of describing it. 04:44.600 --> 04:47.920 And that is an example of a multi-modal message with Autogen. 04:48.520 --> 04:53.480 And then for my next trick, I'm really going to throw things at you rapid fire today because you know, 04:53.520 --> 04:58.000 much of this stuff we're going to do structured outputs, something you know well. 04:58.400 --> 05:00.280 And it made it very easy in Autogen. 05:00.280 --> 05:01.040 It really is. 05:01.200 --> 05:08.000 So this is a subclass of the Pydantic base model called image description. 05:08.000 --> 05:09.000 So that's what I'm calling it. 05:09.000 --> 05:12.840 And it's a class that I'm going to want to populate with the answer from the LLM. 05:13.120 --> 05:20.100 And so I give it four For fields seen, message, style and orientation, which is how I want the model 05:20.140 --> 05:22.740 to describe the image in a particular structure. 05:22.740 --> 05:25.580 Perhaps I'm going to put that on my on my on my user interface. 05:25.620 --> 05:29.300 Perhaps I'm going to catalog it in some way, write it into a SQL database. 05:29.820 --> 05:35.180 So I then use pedantic field to be able to describe each one. 05:35.180 --> 05:41.220 The scene is briefly overall seen in the image message, the point that the image is trying to convey 05:41.860 --> 05:45.740 style, the artistic style and orientation. 05:45.740 --> 05:51.460 So we're going to going to ask it a more kind of analytical question about like, what is this, a portrait 05:51.460 --> 05:57.860 image, a landscape image, or a square image, which requires it to have a slightly more, uh, more 05:57.900 --> 06:00.340 of a meta understanding of what it's looking at? 06:00.580 --> 06:03.820 Uh, so, uh, we'll see how it can fare with this. 06:03.820 --> 06:07.100 So this is the pedantic object that we want it to populate. 06:07.380 --> 06:09.220 And here is the code. 06:09.540 --> 06:14.140 Uh, as before, we create the model client, the the GPT four mini. 06:14.380 --> 06:16.260 And then this is our agent. 06:16.420 --> 06:22.130 It's it's again the description agent, same model client, same prompt and this is the only change 06:22.170 --> 06:25.690 output content type equals and we pass in the pedantic object. 06:25.890 --> 06:26.850 Super simple. 06:26.850 --> 06:29.170 It's made it just that we state what we want. 06:29.490 --> 06:35.850 And so what we now, what we expect is that when we call on messages, the same, same thing that you 06:35.890 --> 06:42.930 use to evoke, uh, to invoke, to use the term, uh, the same thing you used to to call this LM, 06:43.210 --> 06:45.010 um, what comes back? 06:45.290 --> 06:48.850 The reply we expect to simply be a type of this object. 06:48.850 --> 06:51.290 It's going to have replied with this object. 06:51.290 --> 06:56.730 And remember, it feels as if the model is able to reply with a Python object. 06:56.970 --> 06:58.730 And what's going on behind the scenes. 06:58.730 --> 06:59.810 I know you know this. 06:59.850 --> 07:01.170 It's all just JSON. 07:01.370 --> 07:07.610 This is converted into some sort of a JSON spec, and the model returns JSON and the wrapper code then 07:07.610 --> 07:10.050 populates this object from the JSON. 07:10.650 --> 07:12.170 Okay, let's run that. 07:14.290 --> 07:19.650 We're doing the same the same image again, which means I should have run it while I was prattling away 07:19.690 --> 07:22.760 so I wouldn't have to fill the dead air with my nonsense. 07:22.760 --> 07:23.480 But it's done. 07:23.720 --> 07:25.000 And what we've got back. 07:25.000 --> 07:25.880 I just printed it. 07:25.920 --> 07:28.920 It is indeed an image description object. 07:29.120 --> 07:34.080 It is an instance of this and you can see scene is populated. 07:34.080 --> 07:36.160 Let's let's print this out nicely. 07:36.160 --> 07:42.120 So I'm using this thing called text wrap that prints something that's formatted, uh, to, to wrap 07:42.120 --> 07:44.240 around at the end of a certain number of characters. 07:44.240 --> 07:48.960 We'll print the scene, the message, the style and the orientation. 07:49.320 --> 07:51.280 Uh, here it all is. 07:51.600 --> 07:57.080 Uh, it's nice and crisp, seeing a colorful and imaginative interior of a room showcasing a workspace 07:57.080 --> 08:00.960 and a door leading to a vibrant portal message. 08:01.320 --> 08:07.760 The image conveys the theme of creativity and the potential of AI, suggesting a gateway to new possibilities. 08:08.320 --> 08:14.680 Style A vibrant, illustrative style with bold colors and exaggerated features, giving it a surreal 08:14.680 --> 08:18.320 and playful appearance and orientation landscape. 08:18.320 --> 08:22.160 It correctly identified the basics that this is in fact a landscape picture. 08:22.480 --> 08:28.790 Uh, if we come back to the picture as well, I will point out something quite clever, which is I only 08:28.790 --> 08:33.790 realised, actually now talking about this with you, that, uh, nowhere on here. 08:33.790 --> 08:38.310 It's not clear why it sees this as something to do with AI. 08:38.670 --> 08:44.310 And the answer is, you can see because the writing AI is up here, that's the only clue, I think, 08:44.350 --> 08:49.510 unless I'm missing something obvious, that's the only clue that that is what this diagram this is taken 08:49.510 --> 08:53.830 from, from a, from a course I did, which is about, you know, the doorway, the what it means to 08:53.870 --> 09:00.630 go into the world of AI and, uh, the, uh, um, yeah, it's funny that I'd given this course and 09:00.630 --> 09:04.030 I'd had this as my, my image for it, and so I'd had AI in there. 09:04.030 --> 09:09.150 And it's funny that it spots that and includes that in the, in the description. 09:09.190 --> 09:11.550 Very, very impressive I've got to say. 09:11.550 --> 09:12.710 But that wasn't the point of this. 09:12.710 --> 09:19.190 The point of this was to show you that we can use structured outputs easily, and we can get back our 09:19.190 --> 09:19.910 data. 09:19.910 --> 09:22.230 That is according to this schema. 09:22.230 --> 09:26.510 And this would allow us to go in and do more processing with this if we needed to.