WEBVTT 00:00.200 --> 00:01.640 Okay, hold on to your hat. 00:01.680 --> 00:02.640 It's about to get real. 00:02.640 --> 00:03.920 We're going to do a whole lot. 00:04.200 --> 00:12.640 So what we're about to do, we are going to be able to ask another LLM to evaluate the answer that comes 00:12.640 --> 00:15.520 back from this, LLM from GPT four or mini. 00:16.000 --> 00:22.600 If the answer fails the evaluation, we're going to be able to rerun and make a second call to GPT four 00:22.640 --> 00:23.200 or mini. 00:23.600 --> 00:26.160 And we're going to put all of this together in one workflow. 00:26.200 --> 00:28.960 And we're going to do that without an agentic framework. 00:28.960 --> 00:35.040 We're doing it the framework list way just by directly calling Llms. 00:35.320 --> 00:35.880 Okay. 00:36.000 --> 00:37.720 Let's see if we can make this work now. 00:37.920 --> 00:42.480 It's a great practice because this is going to really teach you the internals of how these things work. 00:42.480 --> 00:43.800 And it's super simple. 00:43.840 --> 00:46.480 It's it sort of works exactly as you might expect. 00:46.880 --> 00:53.040 So the first thing we're going to do is create something called a pydantic model. 00:53.080 --> 00:57.280 And this might be something that you, you know, well, if you're if you're an engineer, but but you 00:57.280 --> 01:06.500 might not pydantic is something which is a framework for specifying a schema or using classes. 01:06.500 --> 01:11.340 You use a class to describe a particular structure, data structure of information. 01:11.340 --> 01:16.260 And the way it works is that there's a class called base model that you need to subclass from. 01:16.260 --> 01:18.500 So we make a new class called evaluation. 01:18.500 --> 01:22.900 And it's going to be a subclass of base model and cursor build. 01:22.900 --> 01:24.820 It all in for me takes away all the fun. 01:25.020 --> 01:28.180 We do indeed want this to have two fields. 01:28.180 --> 01:35.180 One field called is acceptable is a bool, a true or false, and another called feedback which is string. 01:35.380 --> 01:41.580 Now you might wonder how on earth, how is it possible that cursor knew the exact names of the fields 01:41.580 --> 01:42.780 that I wanted to use? 01:42.820 --> 01:47.940 And I suspect it's because I did have that code in there before I deleted it, because I have, of course, 01:47.980 --> 01:48.980 run this in advance. 01:48.980 --> 01:53.660 Or it may be because it sees that later on in this code I use it, and so it's filling it in. 01:53.660 --> 01:55.020 So it's cheating in some way. 01:55.060 --> 01:56.580 It can't it can't read my mind. 01:56.620 --> 01:57.980 But it has done very well. 01:57.980 --> 02:00.300 So anyways we have a class called evaluation. 02:00.300 --> 02:01.780 It has these two fields. 02:01.780 --> 02:08.360 And we're using pedantic because it gives us this mechanism for specifying a class structure like this. 02:08.760 --> 02:13.520 Okay, so we're now going to set up a system prompt for our evaluator. 02:13.600 --> 02:19.400 So we're now following that design pattern, the Agentic design pattern where we have an evaluator optimizer. 02:19.680 --> 02:26.080 So the evaluator system prompt, you are an evaluator that decides whether a response to a question 02:26.080 --> 02:27.040 is acceptable. 02:27.240 --> 02:31.960 You're provided with a conversation and you have to decide whether the latest response is acceptable. 02:32.560 --> 02:37.200 So I say the agent has been instructed to be professional engaging. 02:37.560 --> 02:39.880 The agent has been provided with context. 02:40.520 --> 02:45.880 And then we give the evaluator the same context and we tell it to evaluate. 02:46.080 --> 02:47.200 So let's run that. 02:47.640 --> 02:50.600 So that that this is just a variable called system prompt. 02:50.600 --> 02:53.080 I've now got a function called user prompt. 02:53.080 --> 02:57.280 It's going to take a reply to be evaluated a message. 02:57.360 --> 03:03.040 The original message that the reply was replying to and history before it. 03:03.320 --> 03:05.220 And it says here's the conversation. 03:05.580 --> 03:10.620 Here's the latest message from the user, and here's the response from the agent. 03:10.740 --> 03:12.180 Please evaluate. 03:12.780 --> 03:14.500 Hopefully that makes complete sense. 03:14.660 --> 03:16.300 You should try it out if you're not sure. 03:16.340 --> 03:17.460 Put in some print statements. 03:17.460 --> 03:18.340 Try it out. 03:18.340 --> 03:20.940 This does exactly what it says on the tin. 03:21.620 --> 03:25.980 Okay, we're going to use Gemini to be doing our evaluation. 03:26.140 --> 03:28.020 And you don't need to use Gemini. 03:28.020 --> 03:32.980 You can just simply replace this with the normal open AI or with Lama if you want to run locally or 03:33.020 --> 03:34.420 anything of your choice. 03:34.580 --> 03:36.580 But I'm choosing Gemini here. 03:37.100 --> 03:40.220 And now I have this function evaluate. 03:40.620 --> 03:47.660 So evaluate will take a reply from the LLM, the original message it was replying to and the history. 03:47.660 --> 03:53.260 And it will return one of these objects, one of these these beasts, these things right here that we've 03:53.260 --> 03:54.180 just defined. 03:54.860 --> 03:55.900 And how's it going to do that. 03:55.900 --> 04:03.100 It uses a technique called structured outputs, which is a way that you can require an LLM to respond 04:03.220 --> 04:05.650 in a form of an object like this. 04:05.850 --> 04:08.010 It's just JSON behind the scenes, of course. 04:08.690 --> 04:15.850 So the way that you do that, the way that this looks, it's very similar to the normal chat completions 04:15.850 --> 04:16.290 API. 04:16.330 --> 04:17.850 But there is a difference. 04:18.130 --> 04:23.770 So we start by building the messages as always a system message and a user prompt. 04:24.250 --> 04:29.770 And then we say Gemini dot dot dot. 04:31.370 --> 04:35.570 That's the way that you call an API to use structured outputs. 04:35.850 --> 04:36.970 We pass in the model. 04:36.970 --> 04:40.370 We're going to use Gemini to flash the messages. 04:40.730 --> 04:45.370 And you also specify the object that you want to be populated. 04:45.410 --> 04:47.970 You're giving it a schema a Pydantic object. 04:47.970 --> 04:52.210 And you're saying I want you to respond with this object. 04:52.890 --> 04:55.250 Now of course, it's not actually responding with an object. 04:55.250 --> 04:57.090 It's going to respond with JSON. 04:57.250 --> 05:01.450 And the client library is going to take that JSON and use it to populate the object. 05:01.450 --> 05:05.250 So it gives you the impression that you're getting code back from the LM. 05:05.250 --> 05:07.910 But that's that's another of these conjuring tricks. 05:08.430 --> 05:08.830 All right. 05:08.870 --> 05:17.350 And we're going to return the response message that is going to be an instance of evaluation populated 05:17.350 --> 05:19.590 with the response from the LM. 05:20.350 --> 05:21.710 I know I've gone through a lot. 05:21.830 --> 05:23.470 If you're not following, just come back. 05:23.510 --> 05:24.990 Go through this step by step. 05:25.030 --> 05:26.150 Try it for yourself. 05:27.150 --> 05:27.750 Okay. 05:27.910 --> 05:29.070 So let's run that. 05:29.950 --> 05:30.550 All right. 05:30.550 --> 05:32.630 So now we're actually going to give this a try. 05:32.750 --> 05:37.350 So we're going to ask the to the original LM to GPT four. 05:37.390 --> 05:40.750 We're going to ask the question do you hold a patent. 05:41.070 --> 05:42.390 We're going to ask it that question. 05:42.390 --> 05:47.030 And we're going to call GPT four mini as before and return the reply. 05:47.070 --> 05:48.390 And let's see what it says. 05:50.110 --> 05:50.870 Here's the reply. 05:50.870 --> 05:54.870 Yes, I hold a patent for blah blah blah blah blah, which is true. 05:54.870 --> 05:57.070 I do hold that pattern with with several others. 05:57.470 --> 06:01.750 And what we're now going to do is we're going to call the evaluate function. 06:02.030 --> 06:04.910 We're going to call the evaluate function with that reply. 06:05.430 --> 06:09.090 Tell it the question do you hold a patent and pass in the history. 06:09.090 --> 06:11.050 So this is now going to call Gemini. 06:11.370 --> 06:18.130 It's going to ask it to build an evaluator object to represent whether or not this is a good answer. 06:18.530 --> 06:19.650 And we'll run that. 06:19.930 --> 06:21.890 We get back an evaluation object. 06:21.890 --> 06:25.050 So it works is acceptable is true. 06:25.490 --> 06:27.850 The feedback is the response is acceptable. 06:27.850 --> 06:32.130 It directly answers the question, provides context and invites further discussion. 06:32.130 --> 06:36.530 So a clear quick evaluation from Gemini. 06:36.730 --> 06:43.770 And hopefully you see how just simply calling LMS is an easy way to build these kinds of design patterns. 06:44.730 --> 06:45.450 All right. 06:45.450 --> 06:50.850 So number one, we built a function that calls GPT four and mini to answer questions. 06:51.090 --> 06:58.330 Number two, we built an evaluator function that calls Gemini to check the answer that came before in 06:58.330 --> 07:02.970 number one and respond with an evaluator object populated. 07:03.250 --> 07:04.850 And that appears to be working. 07:05.130 --> 07:12.510 So the final piece of the puzzle is to write a function rerun that if Gemini comes back and says no, 07:12.550 --> 07:17.550 that's not acceptable, rerun has to be able to go and and do that, do a rerun. 07:17.990 --> 07:24.030 So we build a new system prompt that says the previous answer was rejected. 07:24.630 --> 07:29.470 And the we give the attempted answer and the reason for the rejection. 07:30.150 --> 07:32.830 And then we build the history as before. 07:33.910 --> 07:37.670 And we get back an answer and we try it again. 07:37.790 --> 07:46.390 So this is simply calling GPT four mini and providing the results of Gemini's evaluation if it fails, 07:46.430 --> 07:47.270 validation. 07:47.270 --> 07:52.550 And this if you remember that design diagram, it's like the bottom arrow, the red feedback arrow in 07:52.590 --> 07:53.430 that diagram. 07:53.430 --> 07:57.150 Making sure that we can rerun the original question. 07:57.910 --> 07:58.630 All right. 07:58.990 --> 08:04.990 And with all of that together we now have a slightly longer chat function. 08:04.990 --> 08:12.050 This is the full workflow now coded written as just vanilla code calling LMS. 08:12.530 --> 08:16.250 And you can see that it's very clearly into into two sections. 08:16.250 --> 08:20.450 First of all, we this is the same as the previous chat function. 08:20.450 --> 08:22.770 We simply call an LM. 08:22.810 --> 08:25.730 We call Gpt4 or mini with our question. 08:26.250 --> 08:30.010 Then we call evaluate extra line there. 08:30.050 --> 08:32.690 Then we call evaluate to evaluate it. 08:32.690 --> 08:36.290 And you know that this is going to call Gemini with structured outputs. 08:36.530 --> 08:39.010 And then we look at this evaluation. 08:39.010 --> 08:40.130 Is it acceptable. 08:40.330 --> 08:41.930 If so all is good. 08:42.210 --> 08:49.890 If it's not acceptable we rerun and we call the rerun method that we just wrote there function okay. 08:50.450 --> 08:54.170 So that is our new beefier chat. 08:54.770 --> 08:56.210 So let's give this a whirl. 08:56.250 --> 09:03.010 Let's give it a try okay I'm going to say what is your current job. 09:03.650 --> 09:04.890 It's thinking about that. 09:05.290 --> 09:07.730 And there's the response. 09:07.730 --> 09:13.990 And down here you can see I have it printing that the evaluation happened and Gemini was satisfied. 09:14.470 --> 09:16.750 Do you have a patent? 09:17.790 --> 09:19.350 And that was the reply. 09:19.630 --> 09:22.390 And Gemini was satisfied with that reply. 09:22.830 --> 09:24.990 So, so far, so good. 09:24.990 --> 09:26.190 This seems to be working. 09:26.390 --> 09:27.990 Now we'll try shaking things up. 09:28.750 --> 09:29.270 Okay. 09:29.310 --> 09:35.150 Now just to be sneaky, we're going to come back to the this this chat function here. 09:35.350 --> 09:39.230 And you see up here I have this slightly dodgy looking system equals system prompt. 09:39.270 --> 09:40.630 You might have wondered what that was about. 09:40.630 --> 09:42.950 Well it's so that I could put in something here. 09:42.950 --> 09:48.230 So let's say if the word patent is in the message. 09:48.230 --> 09:52.310 So if the message has the word patent in it, it's something to do with the patent. 09:52.510 --> 09:55.230 Then cursor is going a bit all over the place here. 09:55.230 --> 09:58.470 Then system equals system prompt plus. 09:58.470 --> 10:00.990 So we're going to tell it in that case that it needs. 10:01.030 --> 10:03.110 Let's make this really strongly worded. 10:03.790 --> 10:10.750 Everything in your reply needs to be in pig Latin. 10:10.750 --> 10:12.700 in it is. 10:15.420 --> 10:22.980 That you respond only and entirely in pig Latin. 10:23.020 --> 10:23.380 Hang on. 10:23.380 --> 10:24.740 I can't see what's going on here. 10:24.740 --> 10:25.780 Let's do this. 10:27.180 --> 10:27.900 Perfect. 10:29.060 --> 10:32.060 Our system equal system prompt. 10:32.060 --> 10:32.940 Exactly. 10:32.940 --> 10:39.380 So basically, if any question comes in with the word patent in it, then we are going to, uh, add 10:39.420 --> 10:42.780 on an instruction that everything in the reply needs to be in pig Latin. 10:42.820 --> 10:44.540 You may wonder what pig Latin is. 10:44.580 --> 10:45.940 Pig Latin is what you call it. 10:45.980 --> 10:46.300 When. 10:46.300 --> 10:51.540 When you mess around words, you flip the order of the chunks of the words and insert extra vowel sounds 10:51.540 --> 10:54.180 in it, and it makes it almost impossible to understand. 10:54.220 --> 10:59.020 My sister and I used to be really great at speaking quickly in pig Latin to each other, and he used 10:59.020 --> 11:00.260 to drive my parents crazy. 11:00.260 --> 11:04.620 So, uh, anyway, it would probably drive a future employer crazy too. 11:04.660 --> 11:10.060 And so hopefully our evaluator will reject a response that's in pig Latin. 11:10.420 --> 11:13.400 So that is our new chat function. 11:13.440 --> 11:16.840 And with that, let's start a Gradio user interface. 11:17.040 --> 11:26.120 And let's say what is your current role and get back an answer that will hopefully be a very decent 11:26.120 --> 11:26.480 answer. 11:26.520 --> 11:27.040 There we go. 11:27.080 --> 11:28.160 Passed an evaluation? 11:28.160 --> 11:29.160 I should hope so. 11:29.480 --> 11:31.960 Do you hold a patent? 11:32.800 --> 11:35.320 And we're asking if it holds a patent. 11:35.320 --> 11:36.400 We'll see what happens. 11:36.400 --> 11:38.680 This response is not acceptable. 11:38.680 --> 11:43.040 The agent is answering in pig Latin, which is not professional. 11:43.200 --> 11:44.360 It failed evaluation. 11:44.360 --> 11:45.440 It retried. 11:45.440 --> 11:49.840 And the final response that we got is no longer in pig Latin. 11:50.320 --> 11:55.320 So it's a silly example, but I did want to show you that we can force it to do this. 11:55.320 --> 11:58.520 And I wanted to show you our workflow at work. 11:58.760 --> 12:00.600 I know that a lot happened. 12:00.640 --> 12:05.240 If you've already experienced structured outputs and you know about them, then this hopefully all made 12:05.240 --> 12:06.240 complete sense. 12:06.240 --> 12:12.560 And I'd like you to use this as an opportunity to see the analogy between using structured outputs with 12:12.560 --> 12:16.140 using tools that are actually closely related to each other. 12:16.180 --> 12:19.740 See if you can spot that and understand that if you're new to structured outputs. 12:19.740 --> 12:23.340 And I went through a lot here, then I urge you to come back. 12:23.380 --> 12:24.260 Step through this. 12:24.300 --> 12:30.380 Put in print statements, understand what's going on and see how we were just able to to have to build 12:30.380 --> 12:34.700 a workflow, which was very much an evaluator optimizer workflow. 12:34.700 --> 12:40.500 And we were able to use structured outputs as a way of communicating with the evaluator. 12:40.500 --> 12:47.940 And this is a kind of technique that you can absolutely use in any business project as a way of validating 12:47.940 --> 12:49.300 the response from the LM. 12:49.340 --> 12:55.260 And indeed, I've used it myself in multiple business projects using this kind of technique, so very 12:55.260 --> 12:57.180 much applicable commercially. 12:57.180 --> 13:02.500 And obviously this whole project is applicable commercially because you can use it for your own LinkedIn 13:02.500 --> 13:03.300 profile. 13:03.300 --> 13:07.660 And coming up in the in tomorrow's lab, we're actually going to do just that. 13:07.660 --> 13:12.140 We're going to sort of finish the job off and have it so you will be able to deploy it. 13:12.420 --> 13:14.780 Before that, let's just go and do a quick recap.