WEBVTT 00:00.280 --> 00:04.360 And so here we are back in cursor to go to lab four. 00:04.880 --> 00:07.520 Because this is a day when we do two labs in one day. 00:07.680 --> 00:12.160 Week four, day four preparing for tomorrow's big project, which is called the Sidekick. 00:12.640 --> 00:16.240 And it's time to introduce structured outputs and a multi-agent flow. 00:16.400 --> 00:23.440 So we've got a bunch of imports and we are going to set our dot env as usual. 00:23.720 --> 00:26.240 And now we're going to be using structured outputs. 00:26.240 --> 00:30.200 And you'll remember the first step with structured outputs is to define the schema. 00:30.400 --> 00:35.400 What are we using to describe the results that must come back from an NLM. 00:35.680 --> 00:40.400 And in particular, the thing that we're going to be working on is an evaluator, something which is 00:40.400 --> 00:45.000 going to decide whether or not an answer from an LLM is good. 00:45.240 --> 00:50.480 And so our evaluator is going to respond using this object in evaluator output. 00:50.520 --> 00:52.160 Or really it's going to respond with JSON. 00:52.160 --> 00:54.280 And the JSON is going to have to conform to this. 00:54.760 --> 00:56.760 So we just describe what this means. 00:56.760 --> 01:01.400 There's going to be a field feedback which is going to be feedback on the workers response. 01:01.800 --> 01:03.720 Uh Success criteria. 01:03.760 --> 01:08.440 Actually, let's change this to the assistance response because that's going to be more words that it 01:08.440 --> 01:09.040 will understand. 01:09.040 --> 01:14.600 Worker is what we will call it in this assist response. 01:14.600 --> 01:15.360 There we go. 01:15.880 --> 01:17.920 Now you get to see me type up. 01:18.400 --> 01:23.880 So feedback on the assistance response success criteria is whether the success criteria has been met 01:24.240 --> 01:26.920 and user input needed. 01:26.960 --> 01:27.440 True. 01:27.480 --> 01:33.200 If more input is needed from the user or clarifications, or if the assistant is stuck. 01:33.640 --> 01:39.040 So this is going to allow we're going to have an evaluator that's going to evaluate the results of our 01:39.040 --> 01:42.120 worker, the assistant, to use their terminology. 01:42.320 --> 01:47.400 And it's going to decide whether it's okay to forward that back to the user, or whether it needs to 01:47.440 --> 01:49.360 go back to the assistant for more work. 01:49.560 --> 01:54.560 And one situation is if the success criteria are met and it's done its job. 01:54.720 --> 01:59.880 But another situation is if the worker seems to be stuck or needs clarifications, in which case it 01:59.880 --> 02:00.880 should return. 02:01.080 --> 02:02.320 So there we go. 02:02.800 --> 02:04.040 Um, okay. 02:04.530 --> 02:07.090 And now to manage state. 02:07.130 --> 02:09.490 You remember with state it can really be any Python object. 02:09.490 --> 02:11.130 It can be a Pydantic object. 02:11.130 --> 02:12.610 But we often use type dicts. 02:12.610 --> 02:13.890 And that's what we're doing here. 02:14.050 --> 02:18.650 And now for the first time, we have some real meaty information to store in the state. 02:18.650 --> 02:23.810 We've always had messages before, but now we have a real state and we've got a bunch of things. 02:23.810 --> 02:29.490 We do still have messages, which is representing the discussion between the user and the assistant, 02:29.490 --> 02:35.450 but we've got other stuff which is going to represent the information being passed from the the evaluator. 02:35.610 --> 02:37.090 Back to the thing. 02:37.090 --> 02:38.890 I'm going to call the worker, the assistant. 02:39.170 --> 02:41.010 Um, and so we're going to have some more stuff. 02:41.010 --> 02:46.170 We're going to have success criteria, which is going to be set up front to define what does it mean 02:46.170 --> 02:52.450 to be successful, feedback on the work that's going to come from the the worker. 02:52.450 --> 02:54.490 And by the way, you use optional like this. 02:54.530 --> 02:57.690 If this can be null, it can be none or it can be a string. 02:58.170 --> 03:07.690 Success criteria met is a bool is true or false and is going to be true if the if there has been a successful 03:07.690 --> 03:12.890 outcome that the criteria met and the the, it doesn't need to go back to the worker for more. 03:13.330 --> 03:19.090 And user input needed is if we need to go back to the user to get some more information. 03:19.090 --> 03:20.890 So that is our state. 03:20.890 --> 03:22.290 So it's a much meatier state. 03:22.290 --> 03:24.650 And this shows you that you can have whatever you want in the state. 03:24.650 --> 03:27.210 The state is really up to you and the flow of your logic. 03:27.210 --> 03:31.810 And that state is like something that has moved through the graph and everyone gets their opportunity. 03:31.810 --> 03:36.530 Every node gets the state and gets its opportunity to to return a new state. 03:36.530 --> 03:38.490 That is some change to that state. 03:38.690 --> 03:45.930 And we're only specifying one reducer ad messages, which means that if one of these nodes returns with 03:45.930 --> 03:48.010 some messages, they will get accumulated. 03:48.010 --> 03:50.970 They will get concatenated with the existing messages. 03:50.970 --> 03:57.730 But if one of these nodes returns user input needed, that will overwrite whatever was in the old state. 03:57.730 --> 04:04.010 So when you return the new state, if you change one of these values here, that becomes the new setting 04:04.050 --> 04:07.580 for anything that is downstream of you in the graph. 04:07.820 --> 04:08.260 Okay. 04:08.300 --> 04:11.420 So next up we set up our playwright tools. 04:11.420 --> 04:16.980 This is the same code as before using the async browser, the playwright browser and the Playwright 04:17.020 --> 04:18.020 Browser toolkit. 04:18.020 --> 04:19.020 So we just run that. 04:19.020 --> 04:20.020 That's pretty simple. 04:20.260 --> 04:23.260 And now we're going to have two llms that we're going to initialize. 04:23.260 --> 04:24.860 One of them is called the worker. 04:24.900 --> 04:27.340 LLM plays the role of the assistant. 04:27.580 --> 04:29.660 It's going to be GPT four mini. 04:29.660 --> 04:35.900 And it's going to be bound to those tools so that it will automatically have the JSON gumpf in it. 04:36.260 --> 04:39.260 And then separately we're going to have an evaluator LLM. 04:39.380 --> 04:41.100 This is a separate LLM. 04:41.340 --> 04:44.300 And it is we're setting it up. 04:44.340 --> 04:46.980 Whereas before we said bind tools. 04:47.020 --> 04:52.540 And now we say with structured output and pass in the Pydantic object. 04:52.540 --> 04:57.260 And that means that the response will conform to this output. 04:57.300 --> 05:00.860 Now not all models are set up to support structured output. 05:00.860 --> 05:04.700 So you may find that some models can't do it if you're if you're going to be playing around with different 05:04.700 --> 05:05.180 models. 05:05.460 --> 05:09.540 If that is the case, of course, the alternative to this is to do it the old fashioned way, which 05:09.540 --> 05:15.340 means instead of using structured outputs like this and passing a pedantic object, you ask the model 05:15.340 --> 05:17.820 in the prompt to respond in JSON. 05:17.860 --> 05:19.060 You give it the schema. 05:19.060 --> 05:21.860 You give, you list out what kind of JSON it should respond in. 05:21.900 --> 05:26.380 You maybe give a couple of examples to make sure that it's really biased to do well, and then you have 05:26.380 --> 05:28.620 to parse that JSON in the response. 05:28.620 --> 05:33.380 And that's all that is happening behind the scenes when we do structured outputs like this. 05:33.660 --> 05:35.900 So that means you congratulations. 05:35.900 --> 05:38.060 You now learn how to do structured outputs with graph. 05:38.060 --> 05:38.940 It's that simple. 05:39.420 --> 05:39.940 Okay. 05:40.180 --> 05:43.820 So this is a bit of a long looking uh, function. 05:43.820 --> 05:48.740 This node, the deaf worker which is representing our worker node, our assistant. 05:48.740 --> 05:51.220 But it's only long because we've got a lot of prompting in here. 05:51.660 --> 05:53.300 So it's a node. 05:53.340 --> 05:57.780 And so as usual it takes a state and it's going to return a state. 05:57.780 --> 06:00.540 And you can see it's returning something with messages. 06:00.540 --> 06:03.700 And we know that messages accumulates because we have the reducer. 06:03.700 --> 06:05.700 So that is going to add on more messages. 06:05.900 --> 06:06.260 Okay. 06:06.300 --> 06:08.380 So we've got like a long system message. 06:08.420 --> 06:09.620 Let me talk you through it. 06:09.660 --> 06:13.950 We say look you're a helpful assistant that can use tools to complete tasks. 06:14.110 --> 06:19.750 You keep working on a task until either you have a question or clarification for the user, or the success 06:19.750 --> 06:21.150 criteria has been met. 06:21.150 --> 06:25.030 And this is the success criteria, and we take it from the state. 06:25.070 --> 06:27.030 It's something that should be held in the state. 06:27.470 --> 06:34.190 You should reply either with a question for the user about the assignment or with your final response. 06:34.470 --> 06:38.830 If you have a question for the user, you need to reply by clearly stating your question. 06:38.870 --> 06:40.150 An example might be this. 06:40.390 --> 06:45.070 It's not super necessary for me to have done that, but I wanted to make it really clear that it will 06:45.070 --> 06:48.670 say when it has a question and sort of force, force the point. 06:49.030 --> 06:54.990 If you finished reply with the final answer and don't ask a question, simply reply with the answer. 06:54.990 --> 06:59.830 And I say that because these models love to reply with things like can I help you with anything else? 06:59.950 --> 07:03.830 And that then might confuse our evaluator that's looking to see if it needs help. 07:03.830 --> 07:06.270 So I want it to be super clear on this front. 07:06.790 --> 07:07.350 Okay. 07:07.550 --> 07:10.070 And of course all these things are subject for experimentation. 07:10.070 --> 07:11.350 There's no hard and fast rules. 07:11.350 --> 07:13.830 It's not like that is a as a rule that you have to put this in the prompt. 07:13.830 --> 07:15.150 I hope you know this by now. 07:15.270 --> 07:20.830 This is this is really the the the part of AI engineering that is about experimentation, R&D. 07:20.950 --> 07:25.390 You can imagine I've been crafting this prompt for the last couple of hours. 07:25.390 --> 07:29.270 So it's something that obviously you hone in on something that works well. 07:29.310 --> 07:32.790 And you may find that particularly with different models or if you have different assignments, it's 07:32.790 --> 07:36.950 something that you have to tweak to get the kind of performance you want anyways. 07:37.190 --> 07:46.430 Uh, if, if we've got in our state something in this, in this field feedback on work, then that means 07:46.430 --> 07:48.910 that an evaluation has happened and it's not gone well. 07:48.910 --> 07:51.590 There's been feedback and it's come back for more. 07:51.750 --> 07:53.790 So then we add to the system message. 07:53.830 --> 07:59.150 Previously you thought you completed the assignment, but your reply was rejected because the success 07:59.150 --> 08:00.950 criteria was not met. 08:01.310 --> 08:04.590 Here is the feedback on why this was rejected. 08:04.590 --> 08:05.870 And then we give the feedback. 08:06.190 --> 08:11.790 And then with this feedback please continue the assignment ensuring you meet the success criteria. 08:12.070 --> 08:16.520 Uh so there it is spelled out in detail. 08:16.880 --> 08:18.240 Um, okay. 08:18.640 --> 08:25.040 And then I've got here some, some slightly hokey code that that looks to see whether there's already 08:25.040 --> 08:27.200 a system message inside there. 08:27.200 --> 08:34.040 And if there already is, then it just replaces whatever system message is there with this system message. 08:34.120 --> 08:40.800 If it doesn't find a system message, then, uh, it, uh, it creates a new one and puts it at the 08:40.800 --> 08:41.360 front. 08:41.560 --> 08:45.600 So this is, this is just to make sure that we handle those various scenarios. 08:45.640 --> 08:50.920 Now, this is a little bit hokey because, uh, Landgraf will have already perhaps have built a system 08:50.920 --> 08:51.320 message. 08:51.320 --> 08:55.240 So, so we have to be a bit careful about this, and it might be worth doing some testing to make sure 08:55.240 --> 08:56.560 that it works for different models. 08:56.600 --> 08:57.120 Okay. 08:57.360 --> 09:03.640 So at the end of all that we then call worker LM with tools and we invoke the messages. 09:03.640 --> 09:07.680 We get back a response and that is what we then return. 09:07.960 --> 09:09.320 So there we have it. 09:09.920 --> 09:10.880 Uh, okay. 09:12.040 --> 09:16.200 Now we now have something pretty interesting called the worker router.