Posts by jedbrown@hachyderm.io
 (DIR) Post #AS91GJH09MJKMw3oWm by jedbrown@hachyderm.io
       2023-01-29T18:57:36Z
       
       0 likes, 0 repeats
       
       @simon This may reduce variability, but is still prone to interference if your workloads are memory bandwidth or IO intensive, since those are shared resources on a shared node and other jobs on the node can have a lot of temporal variability. This is a tough problem because dedicated nodes are expensive and CI infrastructure is rarely set up to do all the correctness testing and then spend 1 minute on a dedicated node running only performance regression tests.
       
 (DIR) Post #ASdBwoSKYnYsfSoJH6 by jedbrown@hachyderm.io
       2023-02-12T15:08:15Z
       
       0 likes, 4 repeats
       
       All the hype about souped up developer productivity using LLMs for coding reminds me of the original title of this 2014 paper, before it was milquetoasted in 2015 acceptance. LLMs can help you rapidly acquire semi-plagiarized fragments of well-traveled code instead of using a quality library with vision of the problem domain. Might be great for KPIs, but this debt will come back to bite you, unless you're already gone. Will be painful for orgs to adapt. https://research.google/pubs/pub43146/
       
 (DIR) Post #ASrJ9kdrTIA2Uml9RA by jedbrown@hachyderm.io
       2023-02-20T03:54:07Z
       
       0 likes, 0 repeats
       
       @simon Why would they want to prevent that? Conspiracies have been profitable for most of these companies and they've tried to walk the thinnest line between maximizing engagement and liability/brand unsafety. They fire ethicists who do more than provide PR cover. It's dark, but I have no confidence in internal controls.
       
 (DIR) Post #AW2DRDouGNxPs6JWNc by jedbrown@hachyderm.io
       2023-05-26T02:17:29Z
       
       0 likes, 0 repeats
       
       @schizanon @simon @grwster If humans produce verbatim paragraphs or similar amounts of code without attribution, that's plagiarism. When language models do it, we tie ourselves in knots to avoid acknowledging the obvious fact that it's also plagiarism. They'll get better at obfuscation so it's only going to get harder to "prove".This extends further: the central value proposition of the current hype cycle is ability to launder illegal and unethical behavior under the banner of so-called AI.
       
 (DIR) Post #AW2EyZodbYY7bHemki by jedbrown@hachyderm.io
       2023-05-26T04:05:17Z
       
       0 likes, 0 repeats
       
       @simon @schizanon @grwster What do you suppose this anecdote shows? Surely not that it can't recite from the training data, nor that performance claims are not significantly contaminated by the training set. > As further evidence for this hypothesis, we tested it on Codeforces problems from different times in 2021. We found that it could regularly solve problems in the easy category before September 5, but none of the problems after September 12.https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks
       
 (DIR) Post #AW2O4iZmqUOK5yiJ84 by jedbrown@hachyderm.io
       2023-05-26T05:47:23Z
       
       0 likes, 0 repeats
       
       @simon  @grwster I have a more cynical take that for the tasks to which observers ascribe skill/knowledge or {basket of anthropomorphizing terms}, LMs can only ever be right for the wrong reasons. If a human exhibited the sort of impressive performance on certain tasks followed by deranged responses on follow-up or problem variants, we'd infer that they were cheating and retract our praise for those moments. Yet we're so thirsty for LMs to be something they're not that we keep rationalizing it.
       
 (DIR) Post #Ab78rjikE89hX0Es64 by jedbrown@hachyderm.io
       2023-10-25T01:38:45Z
       
       1 likes, 0 repeats
       
       @ekuber @cliffle It's so important I use this tweet in academic talks that aren't about Rust at all.