WEBVTT 00:00.120 --> 00:07.440 So the worker router then is a Python function that we will use in our edge, in our conditional edge 00:07.440 --> 00:10.280 to decide which way to route control. 00:10.520 --> 00:11.920 And it's very simple. 00:11.920 --> 00:14.000 It's going to take the most recent message. 00:14.000 --> 00:15.880 It's going to see if it's a tool call. 00:15.920 --> 00:19.760 If so it returns tools if not evaluator. 00:19.760 --> 00:25.040 And that is to say that when our worker, our assistant has come up with an answer, if it's not involving 00:25.040 --> 00:27.600 a tool call, then it needs to be evaluated. 00:27.600 --> 00:29.480 And that's what we get to right now. 00:29.880 --> 00:35.520 So for the evaluation, we have first of all, this little utility function format conversation that's 00:35.520 --> 00:36.640 just used to take in. 00:36.840 --> 00:43.520 I just wrote that to to transform a list of these message objects into something which says like user 00:43.520 --> 00:47.520 assistant, user assistant in just a nice little text summary. 00:47.520 --> 00:54.400 And you'll see why right now, as we come on to look at the evaluator code, we run these two cells 00:54.560 --> 00:56.680 and we will talk about the evaluator. 00:56.680 --> 00:58.240 So this is the evaluator. 00:58.240 --> 00:59.640 This function right here. 00:59.640 --> 01:00.760 It is a node. 01:00.800 --> 01:03.480 It takes a state and it returns a state. 01:03.480 --> 01:09.980 And it's meant to represent the LM, which is going to be assessing our our assistant, our worker, 01:09.980 --> 01:14.420 and deciding if it's ready to return to the user or it needs to go back for more. 01:14.740 --> 01:18.380 And so it all comes down to some prompting and let me take it through you, through it. 01:18.380 --> 01:23.580 Remember that we're using structured outputs that will require that the model returns a particular type 01:23.580 --> 01:24.460 of object. 01:24.740 --> 01:30.340 So first we take the most recent response which is of course the assistance attempt. 01:30.340 --> 01:34.500 We take that out of the state object the messages collection. 01:34.660 --> 01:37.140 So then we come up with this system prompt. 01:37.420 --> 01:42.380 You are an evaluator determines if a task has been completed successfully by an assistant. 01:42.420 --> 01:45.220 Assess the last response based on the criteria. 01:45.220 --> 01:50.940 Respond with your feedback and a decision on whether the success criteria is met and whether more input 01:50.940 --> 01:54.700 is needed from the user and then the user message your. 01:54.740 --> 01:56.180 This is going to be a bit more detailed. 01:56.180 --> 01:59.220 You're evaluating a conversation between the user and the assistant. 01:59.380 --> 02:05.220 You decide what action to take based on the last response from the assistant, the entire conversation 02:05.220 --> 02:09.990 with the assistant along with the user's original request and all replies is here. 02:09.990 --> 02:15.230 And this is going to use this little utility thing, which is just going to say like human, sorry, 02:15.270 --> 02:15.710 not human. 02:15.710 --> 02:20.630 It's going to say user assistant, user assistant with with the whole conversation so far. 02:20.670 --> 02:23.030 So it's going to look very simple in language. 02:23.510 --> 02:30.470 The success criteria for this assignment is and then I'm plucking out of the state this success criteria. 02:30.630 --> 02:34.070 And as I hope you've guessed, this is something that's going to be set right at the very beginning 02:34.070 --> 02:35.590 when we invoke the graph. 02:35.590 --> 02:37.350 So that's going to be passed in by the user. 02:37.350 --> 02:39.470 And it'll be maintained throughout our graph. 02:39.470 --> 02:45.550 So we can pluck it out and just insert it in the user prompt for this evaluator right here. 02:46.110 --> 02:51.150 And then I say the final response from the assistant that you were evaluating is this last response. 02:51.390 --> 02:54.110 And of course that will already be included in here. 02:54.110 --> 02:59.310 But I just want to be crystal clear so that the the evaluator understands that it's not assessing the 02:59.310 --> 03:03.790 whole conversation, it's just assessing this response right here, which is what's going back to the 03:03.790 --> 03:04.310 user. 03:04.830 --> 03:09.090 Respond with your feedback and decide if the success criteria is met. 03:09.530 --> 03:16.650 Also if more user input is required, either because the assistant has a question, needs clarification, 03:16.650 --> 03:20.290 or seems to be stuck and unable to answer without help. 03:20.850 --> 03:26.130 So you may remember that we already put some of this in the definition of the structured outputs of 03:26.130 --> 03:27.770 the response, right up at the top. 03:28.010 --> 03:32.290 Let me show you that right here in the evaluator output. 03:32.330 --> 03:35.930 We already gave a little description of user input needed right here. 03:36.170 --> 03:38.410 And so you may wonder why I'm repeating myself here. 03:38.410 --> 03:42.810 And the answer is because there's never a harm in being repetitious with prompting. 03:43.010 --> 03:43.610 Be clear. 03:43.650 --> 03:44.490 Be instructive. 03:44.490 --> 03:45.370 Repeat yourself. 03:45.370 --> 03:50.450 These are good things to do in slightly different ways, because it biases the model to doing what we 03:50.450 --> 03:51.570 want it to do. 03:52.530 --> 03:52.770 Okay. 03:52.810 --> 04:00.690 And then finally in here I put if we've already got some feedback in the state object, that means that 04:00.690 --> 04:06.250 the evaluator was already called in this in this very loop and has already provided feedback in the 04:06.250 --> 04:06.850 past. 04:06.970 --> 04:08.370 And so I add in. 04:08.570 --> 04:13.420 Also note that in a prior attempt from the assistant, assistance you provided this feedback. 04:13.820 --> 04:19.460 If you're seeing the assistant repeating the same mistakes, then consider responding that user input 04:19.500 --> 04:20.820 is required. 04:21.500 --> 04:23.020 Now you might think this is very clever. 04:23.620 --> 04:25.260 How did you come up with that and why? 04:25.260 --> 04:25.700 Why that? 04:25.700 --> 04:27.420 And when do I use this kind of thing? 04:27.660 --> 04:31.180 And look, the answer is there's no magic here that is there. 04:31.180 --> 04:36.460 Because I was testing this and it kept messing up by the evaluators, sending back the same problem 04:36.460 --> 04:37.380 again and again. 04:37.380 --> 04:40.060 And so this is the kind of thing that is trial and error. 04:40.100 --> 04:41.540 You experiment, you try. 04:41.540 --> 04:44.660 When something goes wrong, you change the prompt and you try some more. 04:44.700 --> 04:46.940 There's no magic and no no clever rules to this. 04:46.980 --> 04:50.180 This won't always apply, but it applies here. 04:50.180 --> 04:54.380 And if you use a different model or slightly different tasks, you may find that you need to tweak this 04:54.380 --> 04:55.660 or use something different. 04:55.660 --> 04:58.180 And that is what AI engineering is all about. 04:58.180 --> 05:00.060 And that is what what prompting is about. 05:00.340 --> 05:03.980 And so yeah, the answer is it's research and development. 05:04.620 --> 05:05.020 Okay. 05:05.020 --> 05:06.860 We'll finish this off in just a second. 05:07.020 --> 05:12.780 So we then put the system message and the user message, which is called a human message object, together 05:12.780 --> 05:15.440 into one list called evaluator messages. 05:15.480 --> 05:23.720 And it's confusing because we're using this concept of system message and user message in order to talk 05:23.760 --> 05:25.520 to an evaluator. 05:25.520 --> 05:30.440 And this isn't actually a human message, it's actually something where it's a user prompt, but it's 05:30.480 --> 05:32.480 a message that we have manufactured. 05:32.480 --> 05:38.360 But that's just really how you go about building system and user prompts using long chains constructs. 05:38.360 --> 05:40.880 This is a a long chain construct within Landgraf. 05:41.120 --> 05:44.920 Uh, and so, you know, this is a still achieving the same thing. 05:45.400 --> 05:46.040 All right. 05:46.240 --> 05:53.720 Now we then take our LM, which is the evaluator LM with structured outputs, we call invoke with these 05:53.720 --> 05:54.640 messages. 05:54.800 --> 05:59.920 And what comes back will be an instance of that class evaluator output. 05:59.920 --> 06:06.040 It is that pedantic object filled up, uh, and behind the scenes what's going on is that it's been 06:06.080 --> 06:12.080 asked to provide JSON and that JSON has come back and that JSON has been parsed into this object. 06:12.080 --> 06:13.280 That's how it did it. 06:13.600 --> 06:19.460 Uh, and so we're then going to create a new state because we're meant to return a new state, and in 06:19.500 --> 06:21.500 that state we're going to respond. 06:21.500 --> 06:25.460 We're going to add to the messages, because remember, messages has the reducer. 06:25.460 --> 06:30.300 So whatever we reply here gets concatenated accumulated with the existing messages. 06:30.300 --> 06:36.340 We're going to shove in there that the assistant is replying evaluator feedback on this answer and something 06:36.340 --> 06:36.860 in there. 06:37.340 --> 06:39.780 Then we're going to give some, some feedback. 06:40.140 --> 06:46.780 Uh, we're going to uh, and so what we're doing here is we're taking the feedback from the Pydantic 06:46.780 --> 06:49.180 object, and we're putting that in the state. 06:49.180 --> 06:55.060 We're taking the success criteria from the Pydantic object, putting it in the state, taking user input 06:55.100 --> 06:57.900 needed, taking it from the Pydantic. 06:57.940 --> 07:03.100 The structured outputs that came back, put it in the state and return the new state. 07:03.140 --> 07:04.580 Hopefully you followed all that. 07:04.620 --> 07:06.940 If not, you will when it comes together. 07:07.300 --> 07:16.580 And then we've got another of these router uh functions route based on evaluation, if the success criteria 07:16.580 --> 07:20.150 is met or if user input is needed. 07:20.150 --> 07:21.670 Then end the super step. 07:21.670 --> 07:22.670 The super step is done. 07:22.670 --> 07:23.470 It's got a controller. 07:23.470 --> 07:24.950 It's got to pass back to the user. 07:24.950 --> 07:28.350 In either of these two extremes, either we've done great or we've done horribly. 07:28.350 --> 07:29.630 Either those extremes. 07:29.630 --> 07:31.430 We need the user to get involved again. 07:31.830 --> 07:38.190 But but if we didn't meet the success criteria and we don't need help from the user, then it needs 07:38.190 --> 07:40.150 to be passed back to the worker. 07:40.150 --> 07:41.510 We need to cycle back. 07:41.550 --> 07:44.110 The worker has got to try again and improve on this. 07:44.110 --> 07:47.550 Given this feedback, that is the whole idea. 07:47.550 --> 07:48.910 That is the workflow. 07:48.910 --> 07:51.670 And now we come to our graph. 07:51.910 --> 07:54.510 It's very simple and all of this is pretty simple. 07:54.510 --> 07:58.390 I've been making a bit of a meal out of it, explaining this step by step and talking about prompting, 07:58.390 --> 07:59.390 but it's not that hard. 07:59.430 --> 08:02.030 If you go through it yourself, you'll see what I mean. 08:02.510 --> 08:04.390 So this is our graph. 08:04.390 --> 08:07.390 We're going to add our worker node. 08:07.390 --> 08:11.910 We're going to add our tools node and our evaluator node that we just built. 08:12.230 --> 08:16.230 Now some edges we're going to have a conditional edge for the worker. 08:16.230 --> 08:19.230 We're going to use the router that we wrote the worker router. 08:19.390 --> 08:23.210 If it returns tools we're going to go with the node tools. 08:23.210 --> 08:26.410 If it returns a value later, we'll pick the node evaluator. 08:27.530 --> 08:31.410 We're going to add another an edge that goes from tools back to worker. 08:31.410 --> 08:31.690 Again. 08:31.690 --> 08:32.370 Remember this one. 08:32.370 --> 08:35.170 When the tools finishes, it's got a route back to the worker. 08:35.210 --> 08:37.450 That's kind of hokey that you have to do that. 08:37.450 --> 08:39.730 You think it would sort of be be done automatically for you. 08:39.730 --> 08:40.970 But but you have to be clear. 08:41.610 --> 08:48.090 And then another conditional edge from the evaluator based on its evaluation, if it wants to go to 08:48.130 --> 08:49.810 the worker, we put it back to the worker. 08:49.810 --> 08:51.850 If we're done, we're done. 08:52.130 --> 08:57.090 And then we also add a start edge to bring us, first of all to the worker. 08:57.130 --> 09:00.650 Let's run that and look at a picture of this. 09:00.650 --> 09:01.490 There it is. 09:01.770 --> 09:02.770 There it is. 09:02.810 --> 09:06.410 We have ourselves a true agentic workflow. 09:06.890 --> 09:13.610 Um, there's the start goes to a worker that optionally can run tools, which has to come back. 09:13.650 --> 09:17.930 There's a thick line, optionally a dotted line, an optional edge. 09:17.970 --> 09:19.690 A conditional edge is the word sorry. 09:20.050 --> 09:22.810 It will run a tool and then it will definitely come back an edge. 09:23.050 --> 09:28.980 And then when it's done, this is shown as a conditional because it's only if it hasn't decided to run 09:28.980 --> 09:35.100 a tool, then it will come this way and the evaluator chooses either to end in two situations. 09:35.100 --> 09:42.220 Either success criteria is met or user feedback is required, and if not, it comes back to the worker 09:42.620 --> 09:44.380 and or back to the worker. 09:44.380 --> 09:48.500 So this this diagram hopefully has everything coming together for you. 09:48.540 --> 09:52.500 And now scroll back up in the lab and just take a look through those nodes again. 09:52.540 --> 09:55.780 And at this point it should be like oh got it, got it, got it, got it. 09:55.820 --> 10:00.180 And hopefully you'll see that although I made a bit of a meal out of it, it is actually quite quick 10:00.180 --> 10:01.580 to build something like this. 10:01.620 --> 10:07.820 And you see one of Anthropics agentic patterns loud and clear here, although it's a little bit more 10:07.860 --> 10:12.580 than one of their workflows because this has like an infinite loop in it, it can keep going. 10:12.700 --> 10:14.140 And there's a lot of optionality here. 10:14.140 --> 10:19.300 So this is a true agent pattern because in theory this could just keep running and running. 10:19.340 --> 10:23.460 And it has some sort of agency autonomy over what it does. 10:23.700 --> 10:28.420 And surely you're now thinking, well, I want to see this thing and that is what we will do next.