Post ASVyF6PrdIYjzQsCDQ by simon@fedi.simonwillison.net
(DIR) More posts by simon@fedi.simonwillison.net
(DIR) Post #ASUgjGpBVSvqcpzZpI by simon@fedi.simonwillison.net
2023-02-09T05:59:52Z
0 likes, 0 repeats
Weeknotes: A bunch of things I learned this week, plus datasette-explainhttps://simonwillison.net/2023/Feb/9/weeknotes/
(DIR) Post #ASUiFOGDUkbsuOW3Yu by pronoiac@mefi.social
2023-02-09T06:14:54Z
0 likes, 0 repeats
@simon heads up, Heroku error
(DIR) Post #ASUiRp5ZZdD7aeSwVs by justin@ramble.space
2023-02-09T06:18:42Z
0 likes, 0 repeats
@simon Hey Simon! I just got a Heroku application error when trying to load your site and thought I should let you know. 😊
(DIR) Post #ASUisRI2ALZzb6zC0u by simon@fedi.simonwillison.net
2023-02-09T06:24:07Z
0 likes, 0 repeats
@justin yeah I got a total denial of service attack from Mastodon there! I bumped up to a larger Heroku worker and it seems to have fixed the issue.I run Cloudflare in front of everything so normally spikes of traffic don't affect me like this, not sure why that didn't help this time
(DIR) Post #ASUjBqYGT2MdRopdpY by simon@fedi.simonwillison.net
2023-02-09T06:25:14Z
0 likes, 0 repeats
@justin @pronoiac @noname yeah I got a total denial of service attack from Mastodon there! I bumped up to a larger Heroku worker and it seems to have fixed the issue.I run Cloudflare in front of everything so normally spikes of traffic don't affect me like this, not sure why that didn't help this time
(DIR) Post #ASUp46PyC0kM42duLY by simon@fedi.simonwillison.net
2023-02-09T07:33:39Z
0 likes, 0 repeats
Also in my weeknotes: I trained a GPT language model from scratch on my blog!> Google is a group of software as it looks further reading as PHP’s enough for Linux, but I need it’s pretty much more for me to be served for me than Pingback. I’ll put out a background idea with a problem with a new tool (in group time accessible). I am getting a web page for that it’s great site in my HTML. I’m done using an entry interface for their mouse gestures (which is done) I have a few more of the site.
(DIR) Post #ASUpDeuFKqildNMRjE by simon@fedi.simonwillison.net
2023-02-09T07:35:04Z
0 likes, 0 repeats
I mean, it's total junk - it helps illustrate how langauge models really are just fancy Markov chain generators - but it does manage to capture my voice in a weird kind of junky way
(DIR) Post #ASUpYS30L4czpoN8q0 by caranatar@kolektiva.social
2023-02-09T07:37:41Z
0 likes, 0 repeats
@simon I too am done using an entry interface for their mouse gestures
(DIR) Post #ASUqq0atFUF1r89qO8 by ferrata@hachyderm.io
2023-02-09T07:53:15Z
0 likes, 0 repeats
@simon I think the mouse gestures part is genius!
(DIR) Post #ASUrRW7YzkqqLkKGbg by j2bryson@mastodon.social
2023-02-09T08:00:01Z
0 likes, 0 repeats
@simon there’s a great video of a musician playing with himself on bass and being freaked out by how it’s recognisably both him & crap. I have no idea how to find it again. He’s a major vlogger interviewing an AI startup who make the models. I came across it because they cited me in their ethics statement.
(DIR) Post #ASUrcWo0n22RetQnSK by glaforge@uwyn.net
2023-02-09T08:00:40Z
0 likes, 0 repeats
@simon How did you do that?
(DIR) Post #ASUs8Gc0X5mshYwzyK by simon@fedi.simonwillison.net
2023-02-09T08:07:52Z
0 likes, 0 repeats
@glaforge I should add that to the TIL! I basically ran the Shakespeare example but against a fresh text file that I created using content from my blog: https://til.simonwillison.net/llms/nanogpt-shakespeare-m2
(DIR) Post #ASUuBYQWxBtYjMHGlc by karga@mas.to
2023-02-09T08:30:00Z
0 likes, 0 repeats
@simon it's up for debate if human intelligence is just the fancy Markov chain generator
(DIR) Post #ASUuXkHZYWKBs2fXqC by glaforge@uwyn.net
2023-02-09T08:34:40Z
0 likes, 0 repeats
@simon neat, thank for the link and explanation. I guess the gibberish output is because it'd need an even bigger corpus of text?
(DIR) Post #ASUuhlKCXEPm01fuee by simon@fedi.simonwillison.net
2023-02-09T08:36:10Z
0 likes, 0 repeats
@glaforge yeah in order to produce actually meaningful sentences you would need to train it on TBs of varied English language data - there's no way it could ever get there just with 30MB of text from my blog
(DIR) Post #ASUvFNqsJUpL2qzQum by codepope@mastodon.org.uk
2023-02-09T08:42:40Z
0 likes, 0 repeats
@simon and so dawned the era of the bad impressionists
(DIR) Post #ASUvS5O4HA03XhZYPo by glaforge@uwyn.net
2023-02-09T08:45:00Z
0 likes, 0 repeats
@simon Makes sense.NanoGPT starts from scratch or has some existing pre-trained weights to get started?
(DIR) Post #ASUvlJblcvDv9N9KG8 by simon@fedi.simonwillison.net
2023-02-09T08:48:37Z
0 likes, 0 repeats
@glaforge starts completely from scratch, at least the way I've been running it
(DIR) Post #ASV78RKgAm1dGNNzCi by aswath@mastodon.social
2023-02-09T10:55:56Z
0 likes, 0 repeats
@simon How many generations of infinite number of Markov chain generators (recursively) would it take to write a Shakespearean drama? I should ask ChatGPT!
(DIR) Post #ASVxmROgXykofBefAW by glaforge@uwyn.net
2023-02-09T20:45:41Z
0 likes, 0 repeats
@simon Just stumbled upon this article that replicates #GPT with #numpy and which says that they loaded existing GPT-2 model weights.So perhaps with such a pre-training, the output would be better on your corpus afterwards?I'm really thinking this is interesting for example for open source projects, as they could help developers and guide them through using the project. Really promising stuff.
(DIR) Post #ASVyF6PrdIYjzQsCDQ by simon@fedi.simonwillison.net
2023-02-09T20:51:01Z
0 likes, 0 repeats
@glaforge I'm pretty skeptical that fine-tuning an existing model is a good way to get decent results on things like "build a chat bot to help people use this open source project" - my hunch is that the fine-tuned data at the end would be overwhelmed by the data already in the modelInstead, I like this alternative pattern for building that same idea: https://simonwillison.net/2023/Jan/13/semantic-search-answers/
(DIR) Post #ASVyfmGqdo7pKRNLnM by glaforge@uwyn.net
2023-02-09T20:55:41Z
0 likes, 0 repeats
@simon Well, intuitively, it's like asking to create a poem without knowing the language. You need to know the language, before being able to write the poem? I'm gonna read the link, thanks.
(DIR) Post #ASW3KNK2hfO7drU3Rg by glaforge@uwyn.net
2023-02-09T20:56:40Z
0 likes, 0 repeats
@simon Your blog! I think I've read it already 😉
(DIR) Post #ASW3KNwKPK0rYbOd7Y by glaforge@uwyn.net
2023-02-09T20:58:06Z
0 likes, 0 repeats
@simon I think Supabase went with embeddings here:https://supabase.com/blog/chatgpt-supabase-docs
(DIR) Post #ASW3KORWXKxz7LzXkG by simon@fedi.simonwillison.net
2023-02-09T21:44:17Z
0 likes, 0 repeats
@glaforge That's effectively the same trick I used for my blog demo:- Get a user's question.- Query the Postgres database for the most relevant documents related to the question (using embeddings)- Inject these documents as context for GPT-3 to reference in its answer.