WEBVTT 00:00.200 --> 00:06.320 Okay, so first of all, we have a callback, a radio callback that radio will call into when we press 00:06.320 --> 00:09.400 a button, which will kick off a super step. 00:09.600 --> 00:15.640 So I've got this little thing here, which is a sort of cheeky thing that that comes up with a thread 00:15.640 --> 00:23.560 ID, a random big unique number to make sure that each time that we bring up the interface, it's not 00:23.560 --> 00:27.400 going to continue the conversation from last time or things can get quite hectic. 00:27.600 --> 00:33.160 So this this means that every radio session is going to have a separate thread. 00:33.160 --> 00:36.960 This also means that we could actually run this and have multiple different people that come on and 00:36.960 --> 00:40.000 use the interface and each have different conversations, which is really cool. 00:40.280 --> 00:42.920 So it's not just going to be for us this this sidekick. 00:43.240 --> 00:43.560 All right. 00:43.560 --> 00:49.600 So then the process message, uh, async, this coroutine, as I should call it, this, this function. 00:49.840 --> 00:55.720 Uh, so first of all, it creates a config based on the thread, uh, code that we're going to give 00:55.720 --> 00:56.160 it. 00:56.160 --> 00:58.240 Then it sets the initial state. 00:58.360 --> 00:59.940 So first of all the initial state. 00:59.940 --> 01:04.340 It's going to take the user's message, the message that comes in that's going to be the initial the 01:04.340 --> 01:05.260 initial message. 01:05.580 --> 01:14.900 It's going to take the success criteria as the, uh, the success criteria from, um, the the user 01:14.900 --> 01:19.460 that the user is going to tell us what what they see as being needed for success. 01:19.460 --> 01:25.620 And this state will go through all of the graph and it will be used by the evaluator, the feedback 01:25.620 --> 01:26.620 on work. 01:26.900 --> 01:29.420 Uh, this is what the evaluator can set. 01:29.620 --> 01:31.500 Success criteria met. 01:31.620 --> 01:32.820 This is false. 01:32.820 --> 01:34.420 To start with user input needed. 01:34.420 --> 01:35.980 This is false to start with. 01:36.020 --> 01:37.820 And then here it is. 01:37.820 --> 01:38.500 Here we do it. 01:38.540 --> 01:40.660 We take a graph we call a invoke. 01:40.780 --> 01:44.900 We pass in the state the initial state we pass in our config. 01:45.020 --> 01:48.140 And we wait for that to come back with what comes back. 01:48.140 --> 01:51.620 We then need to build the stuff that's going to appear in the Gradio user interface. 01:51.620 --> 01:53.820 We package up the user's message. 01:53.940 --> 01:59.000 We package up the the the last two things that will have come back. 01:59.040 --> 02:03.640 The second last one will be the assistance reply, and the very last one is going to actually be the 02:03.640 --> 02:05.200 evaluators evaluation. 02:05.200 --> 02:09.520 And we'll show them both coming so that they both appear in the UI, because it'll be nice to see the 02:09.520 --> 02:10.640 evaluation as well. 02:10.680 --> 02:13.680 But you could remove this if you don't want to see the evaluation in the UI. 02:13.960 --> 02:17.280 And then we package that together in the history with with all of this. 02:17.280 --> 02:18.560 And that's what we reply. 02:18.960 --> 02:24.160 There is a little bit of a confusion here if you're following all this, that we're sort of combining 02:24.160 --> 02:32.160 radio's history with the history that's being stored in Landgraf's memory, it would probably be be 02:32.160 --> 02:37.920 elegant, more more elegant here to actually unpack the full history from from the state. 02:37.920 --> 02:39.680 But that would start to get sort of messy. 02:39.680 --> 02:41.040 So I thought, keep it simple. 02:41.080 --> 02:42.720 It this this is fine. 02:43.320 --> 02:47.520 Um, and then I've also got a little callback called reset that just resets the interface. 02:47.520 --> 02:50.400 So we'll just run this that ran quickly. 02:50.960 --> 02:51.160 Check. 02:51.160 --> 02:52.600 I ran the other cells I did. 02:52.880 --> 02:55.040 And now we're going to launch our user interface. 02:55.040 --> 02:58.260 This is going to be a UI for our sidekick. 02:58.460 --> 03:00.260 Let's see what it looks like. 03:00.700 --> 03:01.380 Here we go. 03:01.420 --> 03:02.220 The sidekick. 03:02.220 --> 03:03.620 Personal coworker. 03:03.900 --> 03:09.100 It's got a nice, uh, nice shade of green I got for this UI. 03:09.580 --> 03:14.660 And, uh, we can ask a question and we can put the success criteria. 03:14.660 --> 03:16.900 So we will ask our sidekick the question. 03:16.900 --> 03:24.620 Uh, what is the current USD, GBP exchange rate? 03:25.460 --> 03:29.100 And the success criteria is going to be an accurate. 03:31.180 --> 03:32.980 Accurate answer. 03:33.260 --> 03:34.860 Simple as that. 03:35.060 --> 03:36.260 Let's see how it does. 03:36.380 --> 03:37.020 Here we go. 03:37.220 --> 03:38.380 Press the go button. 03:39.260 --> 03:40.660 Up comes a browser window. 03:41.700 --> 03:43.660 It's put in one as the amount. 03:43.660 --> 03:45.060 So it's driving this browser. 03:45.060 --> 03:49.260 It's looked for the, uh, for the dollar pound exchange rate. 03:49.260 --> 03:50.420 It appears to be on dollar euro. 03:50.460 --> 03:51.620 That's not a good sign. 03:52.020 --> 03:53.420 See what's happening back here? 03:54.060 --> 03:55.300 It's still processing. 03:55.640 --> 03:57.160 I wonder if it's going to work this out. 03:57.480 --> 04:00.960 Well, it seemed to do okay, but I don't see on the screen how it did that. 04:00.960 --> 04:02.760 Maybe that's down here. 04:02.760 --> 04:03.960 Or did it make a mistake? 04:03.960 --> 04:04.720 Let's see. 04:06.520 --> 04:07.000 Aha. 04:07.240 --> 04:08.160 No, it got it right. 04:08.160 --> 04:13.320 I see it scrolled down it for some reason it was defaulted to Euro, but it got the US dollar to GBP 04:13.560 --> 04:15.440 at 0.77463. 04:15.440 --> 04:18.240 And if we come back here let's see what it said. 04:18.240 --> 04:20.240 It did say 0.77463. 04:20.240 --> 04:20.960 So it did. 04:21.000 --> 04:22.080 It drove the user interface. 04:22.120 --> 04:25.880 It was a bit quirky there that it didn't pick the right currency, but it found the information. 04:26.200 --> 04:31.280 Uh, so the answer was the current USD to GBP exchange rate is approximately that number. 04:31.320 --> 04:36.640 The evaluator feedback from our other agent, the assistant, provided approximate exchange rate. 04:36.680 --> 04:42.280 However, the assistant should ideally note that it may change, so it gives some some stern feedback 04:42.280 --> 04:45.920 there, but it still chose to return control to you. 04:46.240 --> 04:47.840 Uh, so it's unclear. 04:47.880 --> 04:49.880 It thinks if it met our needs completely. 04:49.920 --> 04:51.840 Nonetheless, we saw it working. 04:52.040 --> 04:56.350 Uh, but now we need to see in Langsam whether the right messages were exchanged. 04:56.430 --> 04:59.550 Well, here we are in Lang Smith looking at what just happened. 04:59.590 --> 05:08.070 It's worth noting that it it cost us 19,000 tokens or 0.2 of a cent, a 2/10, a fifth of a cent. 05:08.430 --> 05:10.590 So if we look here, we'll see that a lot went on. 05:10.590 --> 05:12.550 Actually look at all of this stuff. 05:12.830 --> 05:19.350 So the worker uh, called a navigate browser tool, it called extract text. 05:19.350 --> 05:20.950 It called Get elements. 05:21.230 --> 05:25.470 Uh, so I guess yeah, this is, this is how it was doing its navigations and running. 05:25.710 --> 05:31.870 Uh, and then you can see down here that the this is where the these are all worker tasks. 05:31.910 --> 05:33.030 Worker tools. 05:33.030 --> 05:33.750 Worker. 05:33.750 --> 05:37.990 And then this is the evaluator getting called at the end, just as we would hope. 05:38.270 --> 05:44.110 And we can actually go all the way into chat OpenAI to see what message was sent to the evaluator. 05:44.190 --> 05:44.790 So here we go. 05:44.790 --> 05:48.590 So this is the results of that function that I belabored a bit earlier. 05:48.830 --> 05:54.170 System you're in evaluator determines if a task has been completed successfully assess the assistance 05:54.170 --> 05:56.050 last response based on the given criteria. 05:56.050 --> 05:57.850 Respond with your feedback, blah blah blah. 05:58.290 --> 05:59.610 Human you're evaluated. 05:59.610 --> 06:01.010 Conversation between user and assistant. 06:01.010 --> 06:03.890 You decide what action to take and then look at this. 06:03.890 --> 06:09.090 This is this is that that utility function that says user assistant, assistant assistant. 06:09.090 --> 06:10.170 And you can see that I made it. 06:10.170 --> 06:15.730 Just print tools use like this so that it's clear to the evaluator what's going on. 06:16.170 --> 06:19.650 The success criteria for the assignment is an accurate answer. 06:19.650 --> 06:21.530 So that's straight from the Gradio UI. 06:21.970 --> 06:26.930 And the final response that you're evaluating is that that is the same of course as this. 06:27.210 --> 06:31.210 And then this is the respond with your feedback blah blah blah blah blah. 06:31.610 --> 06:36.410 And we can see that what it responded with was JSON. 06:36.690 --> 06:37.530 Uh, of course. 06:37.530 --> 06:42.690 And it's JSON that conforms to the structured outputs schema that we provided. 06:42.730 --> 06:47.890 And that's why, uh, Langshan is able to read that in and populate that object. 06:47.890 --> 06:50.850 And that's what our graph then gets returned to it. 06:50.910 --> 06:53.990 and that is how the whole thing fits together. 06:53.990 --> 06:57.270 And there's the route based on evaluation taking us to the end. 06:57.470 --> 07:03.710 So it's kind of cool to see it here in locksmith and see the trace through the agent flow. 07:03.830 --> 07:07.630 And I can tell you obviously I've done some experiments with this with some of my prompts. 07:07.630 --> 07:09.390 Didn't didn't work so great early on. 07:09.390 --> 07:15.150 And there was times when there was like 25 different messages between the agent and the evaluator, 07:15.150 --> 07:16.750 particularly on some harder tasks. 07:16.750 --> 07:21.350 So it was quite interesting to see it bounce around and and a lot of fun. 07:21.350 --> 07:28.350 And obviously improving the prompts is one way to get there, but overall I see this as a really exciting 07:28.350 --> 07:28.910 project. 07:28.910 --> 07:34.030 You can really feel the beginnings of something very substantive here, and when you run this, you 07:34.030 --> 07:40.310 can experiment with giving it bigger tasks, asking it to summarize the various websites, to navigate 07:40.310 --> 07:42.510 around and to summarize back something to you. 07:42.870 --> 07:46.390 And there's lots that you can do here, which is terrific fun. 07:46.390 --> 07:52.690 And so that's going to put put this on pause for now, because when we come back tomorrow, we're going 07:52.730 --> 07:56.730 to try and make this into like a packaged project with a few more features. 07:57.210 --> 07:58.890 But let's go now to the wrap up. 07:58.890 --> 08:00.770 So I realized it was a long day today. 08:00.770 --> 08:07.330 We got through a lot, and I hope that you did have that same journey as me as seeing the visual there 08:07.330 --> 08:15.050 of that, uh, land graph graph and seeing the ability to build an evaluator and an agent that's able 08:15.050 --> 08:19.970 to talk like that and to go backwards and forwards, being able to give it playwright and be able to 08:20.010 --> 08:21.730 operate playwright like that. 08:21.770 --> 08:28.610 It's like we've built our own operator, the, the, uh, the agent from OpenAI, or that we built like 08:28.650 --> 08:30.010 a kind of agent. 08:30.130 --> 08:32.410 And it's been so easy to do it. 08:32.410 --> 08:37.130 And it's so cool to see the graph like this, to be able to interact with it in Langsamt. 08:37.450 --> 08:40.410 So as I say, I'm becoming I'm becoming a true believer. 08:40.610 --> 08:43.650 Uh, I'm becoming, uh, Landgraf advocate. 08:43.650 --> 08:45.450 It's exciting to see it come together. 08:45.450 --> 08:48.250 And tomorrow we'll try and take it one more step forwards. 08:48.290 --> 08:49.130 See you then.