WEBVTT 00:00.280 --> 00:07.920 Okay, so it's time to evaluate who did best and what better way to evaluate who did what best than 00:07.920 --> 00:09.480 to use an LM for this task. 00:09.520 --> 00:12.000 It would be tedious to have to read through all of these and decide. 00:12.200 --> 00:13.760 First, let's just understand where we are. 00:13.760 --> 00:15.440 So we had two lists. 00:15.440 --> 00:19.040 One was called competitors and one was called answers. 00:19.320 --> 00:20.760 I love the way cursor does that. 00:20.760 --> 00:25.040 And so we can just print them out and we'll see that we've got these two lists. 00:25.160 --> 00:25.600 There they are. 00:25.640 --> 00:26.840 These are our competitors. 00:26.880 --> 00:28.160 I hope you recognize them. 00:28.160 --> 00:30.000 And these are their answers. 00:30.480 --> 00:35.440 So one one thing that's nice to do right now, it would be sort of nicer to be able to pair these up 00:35.440 --> 00:37.320 and say which one is which. 00:37.480 --> 00:42.920 There's a really useful Python function called called zip that is worth knowing about. 00:42.920 --> 00:49.640 It's like a pro thing to use, and you can iterate through these two collections together like this 00:49.640 --> 00:55.120 I can say for look haha, what am I doing here? 00:55.160 --> 00:56.160 I just press tab. 00:56.440 --> 00:59.400 You can say for competitor and answer. 00:59.400 --> 01:03.410 Those are the two we want to be iterating through competitors and answers in. 01:03.450 --> 01:09.290 And if you say zip competitors answers, then it will just iterate through through these two lists together. 01:09.290 --> 01:12.850 And competitor and answer will have each one. 01:12.850 --> 01:13.930 And it's written all the code. 01:13.930 --> 01:15.650 For me, that's exactly what I wanted to do. 01:15.650 --> 01:20.970 So if I do this now, you'll see that it's printing each competitor and its answer one after another. 01:21.010 --> 01:21.850 Very nice. 01:22.250 --> 01:26.810 Okay, but let's bring it together into one string called together. 01:27.410 --> 01:30.290 And I also wanted to show another construct here called enumerate. 01:30.290 --> 01:36.570 If you have a list like answers, a list of things and you want to iterate through them, but you also 01:36.610 --> 01:42.290 want to track the index number as you do, you probably like people that don't know about enumerate 01:42.330 --> 01:44.290 often have a kind of count equal zero. 01:44.330 --> 01:46.330 Count plus equals one in your loop. 01:46.370 --> 01:47.010 Not needed. 01:47.050 --> 01:53.730 Enumerate is a nice little trick you can say for index, comma, answer in enumerate answers, and then 01:53.730 --> 01:55.090 you will just step through. 01:55.290 --> 01:57.650 And so we will say response from competitor. 01:57.650 --> 02:02.620 And we're going to add one to the index so that the first one is is competitor number one rather than 02:02.660 --> 02:05.020 competitor number zero, which just doesn't sound as nice. 02:05.220 --> 02:11.180 So we run that and then uh, why don't I just just show you what that looks like? 02:11.180 --> 02:12.860 But I'm sure you believe me. 02:13.940 --> 02:16.940 It looks like this. 02:17.180 --> 02:20.740 And if I print it, we'll see it looking a bit more pleasant than that. 02:21.340 --> 02:22.260 Here you go. 02:22.540 --> 02:24.060 So you can see response. 02:24.100 --> 02:24.700 Oh, look at that. 02:24.700 --> 02:25.820 There's a mistake. 02:26.660 --> 02:27.140 Just as well. 02:27.140 --> 02:28.020 I did print it, isn't it? 02:28.020 --> 02:30.100 That's the benefit of printing these things. 02:30.100 --> 02:33.460 You do that, then you get to see, uh, see what's going on. 02:33.500 --> 02:33.900 Wow. 02:34.980 --> 02:35.620 Okay. 02:35.820 --> 02:36.900 I hope you you caught that. 02:36.900 --> 02:37.740 That's embarrassing. 02:37.740 --> 02:38.900 But I think I'll keep it. 02:38.900 --> 02:44.420 So you get to see that, uh, that not only can one make mistakes, but this gives you a great way to 02:44.460 --> 02:46.700 inspect what's going on and see what's happening. 02:46.900 --> 02:50.140 So when I print that now, we see response from competitor one. 02:50.460 --> 02:53.700 And presumably down below it will have response from competitor two. 02:54.380 --> 02:55.220 All right. 02:55.540 --> 02:58.020 So now we're going to have a new bit of text. 02:58.060 --> 03:01.820 You are judging a competition between that many competitors. 03:02.020 --> 03:04.380 Each model has been given this question. 03:04.900 --> 03:07.740 Your job is to evaluate each response. 03:07.740 --> 03:09.060 And there is the. 03:09.380 --> 03:17.940 We say we want you to respond in JSON and only in JSON and follow this format a JSON with results and 03:17.940 --> 03:22.460 then a competitor number, a second best competitor number, third best competitor number, and so on. 03:23.060 --> 03:24.700 And here are results from the competitors. 03:24.700 --> 03:29.820 And now respond with the JSON with the ranked order of the competitors. 03:29.820 --> 03:30.860 Nothing else. 03:30.900 --> 03:33.780 Do not include number formatting or code blocks. 03:34.500 --> 03:36.460 So just a few things to point out about this. 03:36.620 --> 03:40.740 First of all, you may not be familiar with the triple quotes. 03:40.780 --> 03:46.500 If you used triple quotes in a Python string, you get to have an entire block of text without needing 03:46.500 --> 03:48.420 lots of quote marks and pluses and things. 03:48.420 --> 03:51.220 So this is a nice trick for a block of text. 03:51.260 --> 03:55.420 It's also, of course used for for for docstrings, so I'm sure you have come across it before. 03:55.980 --> 04:01.430 Another thing to know is that I've sort of assumed you're familiar with f strings before, but one nice 04:01.430 --> 04:07.990 trick to know is that if you actually want to have a curly brace within your string, then you can do 04:07.990 --> 04:11.590 that by having two curly braces in an F string. 04:11.590 --> 04:17.870 So that's why there are two curly braces here, because we actually want one curly brace to appear in 04:17.870 --> 04:18.990 the string itself. 04:18.990 --> 04:19.350 We're not. 04:19.350 --> 04:21.670 We don't want we don't want this to be interpreted as code. 04:22.350 --> 04:23.070 All right. 04:23.310 --> 04:30.270 So with that, if I print this to to show you what on earth I'm prattling away about, what you should 04:30.270 --> 04:33.510 see is you're judging a competition between six competitors. 04:33.510 --> 04:35.470 Each model has been given this question. 04:35.630 --> 04:37.430 Your job is to evaluate. 04:37.710 --> 04:40.670 Here is the how to give us the results. 04:40.670 --> 04:42.750 And here are the responses from each. 04:42.950 --> 04:49.070 And there you see it in there I'm also going to mention you see at the end here I say do not include 04:49.070 --> 04:51.670 markdown formatting or code blocks. 04:51.710 --> 04:52.910 It's always worth doing that. 04:52.910 --> 04:58.720 Otherwise these models love to add in a little extra JSON tag around things. 04:58.720 --> 05:03.440 So if you use this text then you make sure you get pure JSON back. 05:03.480 --> 05:08.760 Okay, so now that we've got this all set up, we're going to put this into messages. 05:08.760 --> 05:11.880 And then we're going to call an LM and get our results. 05:12.040 --> 05:12.880 Let's do that. 05:13.440 --> 05:13.880 Okay. 05:13.920 --> 05:15.000 It's judgment time. 05:15.000 --> 05:21.960 So we're going to put this judge text into a messages list as usual with the normal structure. 05:22.240 --> 05:24.680 And now let's choose a judge. 05:24.680 --> 05:29.600 So I'm picking O3 mini a little bit pricier I think than some of the others. 05:29.640 --> 05:31.120 So they should check the vellum leaderboard. 05:31.120 --> 05:33.680 And you may want to just go with a with a cheap one. 05:33.880 --> 05:37.200 Like like like using GPT four or mini. 05:37.200 --> 05:41.120 I'm going to go with O3 mini because we're going to try and have something that really pays attention 05:41.120 --> 05:41.640 to this. 05:41.640 --> 05:48.280 Now, I realized the first competitor in the list is GPT four mini, and so it's a bit -- that 05:48.280 --> 05:50.920 some of the same OpenAI models are judging themselves. 05:50.960 --> 05:55.440 And it'll be interesting to see whether it thinks that the model from its own family is the winner, 05:55.480 --> 06:00.090 and we're not telling it the model names, we're only giving it a rank number, so it's not going to 06:00.130 --> 06:00.450 know. 06:00.450 --> 06:02.330 So we'll see how number one fares. 06:02.370 --> 06:04.130 Actually, the number one's answer did look really good. 06:04.170 --> 06:05.450 So it might do very well. 06:05.450 --> 06:06.170 We will see. 06:06.450 --> 06:09.930 But we're letting O3 mini make the call. 06:09.970 --> 06:11.570 Let's see how it does. 06:12.770 --> 06:13.530 Off it goes. 06:13.530 --> 06:14.730 It's thinking about it. 06:15.450 --> 06:16.730 It's taking its time. 06:16.770 --> 06:17.850 It's doing some reasoning. 06:17.850 --> 06:19.370 And here's the answer. 06:19.370 --> 06:24.730 So it doesn't put the uh GPT four mini at the top. 06:24.770 --> 06:25.850 It puts model number three. 06:25.850 --> 06:31.210 And you probably remember AI model number two was uh, was anthropic model number three, I forget. 06:31.210 --> 06:32.130 Was it deep league? 06:32.170 --> 06:33.450 We'll soon find out. 06:33.490 --> 06:40.290 So we are now going to, uh, load this, which has come back in perfect JSON just as we wanted. 06:40.290 --> 06:42.130 We're going to load it into a dictionary. 06:42.130 --> 06:44.010 We're going to pluck out results. 06:44.010 --> 06:45.690 So now we're going to have this. 06:45.730 --> 06:50.010 We're going to iterate through it using again this enumerate approach. 06:50.170 --> 06:59.060 For each one we're going to look at this string And subtract one and look it up in the competitors list 06:59.060 --> 07:00.220 and print the results. 07:00.220 --> 07:06.700 So if you're following me, this is just going to print out from best to worst the names of the models 07:06.700 --> 07:07.860 in our results. 07:07.900 --> 07:09.220 Drum roll please. 07:09.540 --> 07:10.420 Here we go. 07:10.860 --> 07:13.300 So it was Gemini two zero flash. 07:13.340 --> 07:14.300 That is our winner. 07:14.540 --> 07:16.540 And GPT four mini came second. 07:16.740 --> 07:21.060 Llama 3.3 came third and then Deep Seek and then Claude 3.7. 07:21.060 --> 07:24.780 But bottom of the list was the flailing llama 3.2. 07:24.820 --> 07:26.100 That kind of gave up halfway. 07:26.300 --> 07:31.100 So there are the unscientific results of our judgment. 07:31.140 --> 07:35.620 Of course, it will be great fun for you to try this in a more scientific way. 07:35.660 --> 07:40.660 For example, you could have each of the different competitors come up with the rankings and then use 07:40.660 --> 07:42.020 that to take averages. 07:42.020 --> 07:46.700 That kind of thing would be rather more interesting way of assessing it than just simply going with 07:46.700 --> 07:48.700 whatever oh three mini tells us. 07:48.780 --> 07:56.110 So anyway, this was a really interesting experiment designed to show you how you can collaborate between 07:56.110 --> 07:56.790 LMS. 07:57.630 --> 08:02.430 So hopefully you were paying attention during this and the last lecture, and you've been able to identify 08:02.590 --> 08:07.950 which agentic workflow patterns were used in this example. 08:07.990 --> 08:09.150 Pattern or patterns? 08:09.350 --> 08:10.630 There might be a couple. 08:10.670 --> 08:14.510 If you're not sure, then go back through it and have a think and look back at the diagrams. 08:14.510 --> 08:19.750 And it might be some sort of hybrid or mish mash between a couple of them and the exercise for you. 08:19.750 --> 08:21.030 First of all, do that. 08:21.030 --> 08:26.750 And then secondly, please pick one of the patterns that interests you and add that to this mix. 08:27.070 --> 08:29.750 The goal of this lab was was twofold. 08:29.750 --> 08:35.790 First of all, to experiment with different APIs and see for yourself how the OpenAI API is used so 08:35.790 --> 08:41.310 frequently across many models, except anthropic, and also to experiment with with the sort of the 08:41.310 --> 08:43.030 basic API structure and the prompts. 08:43.270 --> 08:48.670 And the second goal was to experiment with this orchestration between models, asking a question, having 08:48.670 --> 08:55.430 multiple models answer, getting another model to assess the output all this stuff is about interactions 08:55.430 --> 09:00.630 between models, dividing up a bigger problem into smaller problems, and a little bit of autonomy in 09:00.630 --> 09:04.150 terms of coming up with whatever question the first model wished to. 09:04.590 --> 09:08.950 So the, uh, yeah, add another agentic design pattern. 09:08.950 --> 09:10.270 I'd love to see it. 09:10.270 --> 09:16.230 And then once you've done that, consider doing a PR and putting it in community contributions. 09:16.230 --> 09:18.910 There'll be instructions in the resources on how to do that. 09:18.950 --> 09:23.550 It's a good tip is to note to delete outputs of your notebook before you do it. 09:23.550 --> 09:25.590 So there's not lots of junk in there. 09:25.590 --> 09:29.150 And be sure that you're only pushing things in that community contributions folder. 09:29.190 --> 09:30.110 There'll be instructions. 09:30.110 --> 09:35.550 It'll be fabulous to see what you're doing, and to see some interesting design patterns added in to 09:35.590 --> 09:37.150 this kind of exercise. 09:37.590 --> 09:41.470 And then final thought for this lab is about commercial implications. 09:41.470 --> 09:47.470 I do want to always bring it back to how you can, how you can think about this in a commercial setting. 09:47.630 --> 09:54.800 And to be honest, I it's hard to be specific here because this is so universally applicable. 09:55.000 --> 10:00.440 Really, any time that you have something that you want to to generate something, you want an LM to 10:00.480 --> 10:03.040 take care of these kinds of patterns. 10:03.080 --> 10:09.480 Being able to send the same request to multiple LMS, being able to then evaluate responses and either 10:09.520 --> 10:14.680 select the best or perhaps select a couple of best ones, or use that to give feedback. 10:14.920 --> 10:21.800 All these kinds of patterns are used to increase the robustness and the accuracy of models, and to 10:21.800 --> 10:24.520 be able to solve harder and harder problems. 10:24.640 --> 10:28.520 So really this is, as I say, universally applicable. 10:28.520 --> 10:33.000 So you should be able to pick whatever commercial problem you can think of that you're applying AI to, 10:33.280 --> 10:38.920 whether it's a summarization kind of problem or a generative problem, writing an email, building a 10:38.920 --> 10:41.360 document, writing a business requirements document. 10:41.360 --> 10:44.760 And you can think of how you could apply these kinds of techniques. 10:44.800 --> 10:46.560 Sending the request to multiple models. 10:46.600 --> 10:48.480 Voting on the best outcome. 10:48.480 --> 10:52.040 This is something that you should be able to apply to your projects right away.