Language models on the command-line w/ Simon Willison

Local

models

are a great way to fully explore them. I have a question. You also mentioned the Aval tool that you are slowly working on. Does it incorporate a data set as well? Because I could not. When I want to do it. At least for my first round of assessments, I'll do it in a notebook or open a Bic where I can tag things as right or wrong and then filter by them so they can make sense. So the idea with the evaluation tools and you don't do this yet, you should log the results to sqlite so you can have a custom dataset interface to help you evaluate them.

I want to make one of those, um, lmis arena style interfaces where you can see two different messages, two different responses from the messages that you've run evils against and click on the one that's better and that gets recorded in the database, plus there's a lot you could do with it because fundamentally sqlite is a great kind of substrate for building tools, it's incredibly fast, it's free, everyone has it, you can use it as a file format to pass things on, like imagine running a bunch of evaluations and then store a lightweight 5 megabyte SQL file on your coworker to take a look at what you've done, that's all possible too, it's cool, but yeah, that's the ambition.

I don't know when I'll be able to invest the time in it, once again, as if people were here. If you are interested in helping or chatting about these things, please get involved. I do. I'm also interested in talking about the SQL Light database and then the data set. One thing that's also nice about LM Studio is that it will tell you "like." it's got some serious product stuff when you run something that's going to give you in your goey the latency and the number of tokens and things like that, we record those things in SQL light and we have it and then we do it seriously. you know, um, benchmarking different

models

, yeah, I've been thinking about my dad's ticket for a while.

It's amazing, that needs to happen. um, yeah, I guess it's tokens per second and the total and total duration exactly. It's going to be interesting to figure out how to do it better. for models where I don't have a good token count but can modify it, just the duration alone would be useful, things start to register in an absolute way, so there's kind of a dividing

line

in some of these questions . First of all a lot of people like wow this is amazing thank you Misha said Simon is a hero thank you an anonymous attendee said this is brilliant I can't believe you did this so everything is great everything is related and I want to develop this question.

I'm saying this a little off topic, but how can you create so many amazing things? I just want to do it. I have a blog post about the idea that you'll raise that as an issue in a GitHub repository. build writing too, so yeah, what structures do you put in your own life to achieve so much? Basically this talk here is about my approach towards personal projects and effectively and really the argument I make here is that you need to write unit tests and documentation because then   you can do more projects because if you haven't you will run into a project like my LM project ok that I haven't touched in two months but because I have a decent set of problems and sort of notes so I'm going to be able to understand that very easily and then the other trick is that I only work on projects that I already know which I can do quickly like I don't have time to take on a six month mega project, but when I look at things like llm I already had the experience of working with the site from the data set.

I knew how to write Python

command

line

applications. I knew how to create plugins. in infrastructures because I had done that for the data set, so I was probably the person on Earth best equipped to create a

command

line tool in Python that has plugins and does

language

model stuff and logging in sqlite and really that It's my kind of The main trick is that I have a lot of things that I know how to do and I'm very good at spotting opportunities to combine them in a way that allows me to build something really interesting but pretty quickly because I have a lot of other things.

Things happen amazing, that, that, that's the trick, your projects are selected, um, and there are also little things that you do, like, um, well, it's not small anymore, right, but today I learned about your blog and what such a wonderful way to find out? It doesn't necessarily have to be something new, but as I'm learning today, just quickly type something you learn and it will tell you the trick for that is every single thing I do, I do it in GitHub edits, so if I'm working on something. Not at all, I'll start a GitHub issue thread in a private repository or a public repo and write notes as I figure it out, and as one of my favorite examples, this is when I wanted to serve an AWS Lambda function with a function.

URL for a custom subdomain that took me 77 comments, all for me to discover because oh my god, AWS is Nightmare and in those comments I'll include links to things I found and screenshots of the horrible web interfaces I have to use and everything. of that kind of thing and then when I went to write a t, I just copy and pasted the mark into the number, so most of my ts take me like 10 minutes because they're basically the kind of semi-structured notes that I had. Already copied, pasted and cleaned up a bit, but this is this, this is in that UM productivity presentation.

I gave this well, just having that it's almost like a kind of um like a scientist's um notebook approach, where whatever you're doing, you write very comprehensive notes about what I have to do next, what I did. I just tried, what worked, what didn't work and you get them all in that sequence and it means I can. I don't remember anything about AWS Lambda now, but next time I want to solve this problem I can go back and read this and it will sort of reset my brain to the point where I can pull, pull, pull the project from where I got to awesome .

I know over time there are a couple of really interesting questions, so if you have a couple more minutes, yes, there definitely is one. Have you thought about using text or Rich um to get a pretty nice result? I think it works. like what LM logs says, look at this LM logs pipe python DM Rich, what is it? It's rich. Markdown you can do, I think so, but maybe I see that there we get rich, it's just that it's pretty printed on my Mark Town, so yeah, I haven't added rich as a dependency, because I'm very, very, um, I'm I'm very protective of my dependencies.

I try to keep them as minimal as possible. You should create a plugin. It would be great if there was a new plugin but I need a new plugin but if there was a plugin you might like. install llm rich and now llm generates things like   that would be super fun so yeah I should do that that would be cool um and just for complete transparency occasionally when I want to have fun playing with the command line I'm with tools like calay and so taking llm to calay has been fun and also taking cows to BS.

The last question is, um. Misha has some GPU rigs and wondering if there is any idea how to run llm with multiple models on different machines but on the same um. So plugins, um, I would solve that with more duct tape, take advantage of existing tools that allow you to run the same command on multiple machines, so, um, responsible things like that, uh, yeah, I think that would be the way to go. do it. that and that's the joy of Unix is ​​a lot of these problems, if you have a little Unix command, you can plug it in with an extra bash script and anable and kubernetes and God only knows what else I run occasionally within GitHub actions that works because it's just a command line, so yes, for that, I would look at existing tools that allow you to run commands in parallel on multiple machines.

It's amazing for everyone. Next step pip install llm or pipex install llm and tell us on Discord how I just want to thank everyone for joining in once again and thank you Simon for all your experience and wisdom and it's always fun to chat. Thanks for inviting me and I will send a markdown document with all the links. demos and stuff on Discord at some point in the next six hours I'll get them, so I'll put them in the Discord channel. Fantastic, very good, see you all on Discord and thanks once again, Simon.