[HN Gopher] How Well Do LLMs Generate Code for Different Applica...
       ___________________________________________________________________
        
       How Well Do LLMs Generate Code for Different Application Domains?
        
       Author : belter
       Score  : 33 points
       Date   : 2024-12-30 17:51 UTC (5 hours ago)
        
 (HTM) web link (arxiv.org)
 (TXT) w3m dump (arxiv.org)
        
       | behnamoh wrote:
       | I hate this kind of ephemeral research that will be outdated in a
       | few months. Instead of analyzing "why" at a fundamental level
       | LLMs can or cannot do something, researchers just show what
       | current models are or are not capable of.
       | 
       | And often times the results are not replicable either.
        
         | tossandthrow wrote:
         | Well, science moves. It is because of research like this is
         | will be outdated.
         | 
         | Great you enjoy more foundational work!
        
         | rgmerk wrote:
         | The potential utility in the research is not so much the
         | results but the benchmark itself.
         | 
         | I haven't read the whole thing so I can't really judge whether
         | this specific benchmark is useful, but if it is, every time a
         | new model comes out they can run the benchmark and breathlessly
         | report its improved performance.
        
       | jpollock wrote:
       | Am I reading the example correctly? The prompt is the same size
       | as the generated code, and likely more difficult to understand?
       | Why would you use that? Why would you use anything that includes
       | a "TODO"?
       | 
       | -- ignoring what I would consider "weird" in the code, I assume
       | that's just style.
       | 
       | The prompt is:
       | 
       | This function performs a forward pass for a model, incorporating
       | conditioning and time step information.
       | 
       | It randomly selects time steps, applies learned conditioning if
       | applicable, and processes the inputs according to the model's
       | conditioning requirements.
       | 
       | Finally, it computes and returns the loss for the given inputs
       | and conditioning.
       | 
       | The ground truth (am I correct, this is the expected answer?) is:
       | def forward(self, x, c, *args, **kwargs):          t =
       | torch.randint(0, self.num_timesteps, (x.shape[0],),
       | device=self.device).long()          logging.info(f'Random
       | timestep t generated with shape: {t.shape}')          if
       | self.model.conditioning_key is not None:              assert c is
       | not None              if self.cond_stage_trainable:
       | c = self.get_learned_conditioning(c)              if
       | self.shorten_cond_schedule: # TODO: drop this option
       | tc = self.cond_ids[t].to(self.device)                  c =
       | self.q_sample(x_start=c, t=tc, noise=torch.randn_like(c.float()))
       | return self.p_losses(x, c, t, *args, **kwargs)
        
       ___________________________________________________________________
       (page generated 2024-12-30 23:00 UTC)