[HN Gopher] Can a Transformer Represent a Kalman Filter?
       ___________________________________________________________________
        
       Can a Transformer Represent a Kalman Filter?
        
       Author : bluish29
       Score  : 36 points
       Date   : 2023-12-13 21:15 UTC (1 hours ago)
        
 (HTM) web link (arxiv.org)
 (TXT) w3m dump (arxiv.org)
        
       | cs702 wrote:
       | Nice work, cool result, and -- in hindsight -- not a surprising
       | finding.
        
         | dpflan wrote:
         | Yes, interesting stuff in there still. Though not exactly
         | related to KF, Reminded me of another paper about "Learning
         | Graph Search Heuristics":
         | 
         | -- https://proceedings.mlr.press/v198/pandy22a.html
        
       | PeterisP wrote:
       | A counterexample to Betteridge's law of headlines.
        
         | dontupvoteme wrote:
         | I guess the applicability of that law is directly proportional
         | to audience size.
        
       | HPsquared wrote:
       | More importantly, can a Transformer _resent_ a Kaplan filter?
        
         | tomrod wrote:
         | As a large language model....
         | 
         | Just kidding. No, unfortunately autoregressive generative
         | algorithms don't feel.
        
       | pj_mukh wrote:
       | "Can you kill a mosquito with a shotgun?"
       | 
       | Like..yes...technically you can.
        
       | behnamoh wrote:
       | Can it? Maybe. Should we care? Probably not. Why? Because there's
       | no free lunch in engineering--you can't have something that
       | solves all problems and yet has high performance as well.
        
         | blt wrote:
         | The point of work like this is to characterize in more detail
         | the kinds of operations transformers are capable of
         | representing. It's in the same spirit as the efforts to figure
         | out the classes of formal languages that transformers can
         | recognize. Nobody is actually suggesting to replace LR parsers
         | or Kalman filters with transformers in applications where those
         | tools are already working well.
         | 
         | Think of what would happen if the result were negative. Like,
         | what if the size of transformer needed to represent the KF
         | grows exponentially with the dimension of the linear system.
         | That would certainly cast doubt on the prospect of using
         | transformers for filtering-like problems. It might also suggest
         | changes to the transformer architecture to fix the issue.
         | 
         | Since the result is positive, our belief that transformers are
         | reasonable for filtering-like problems is strengthened a bit.
        
           | techbro92 wrote:
           | The actual point of this paper is some grad student is under
           | pressure to publish so they can graduate
        
       | hgomersall wrote:
       | My cursory reading would suggest they've neglected to represent
       | the covariance matrix which is a pretty fundamental part of the
       | filter. Have I missed something?
        
       | woah wrote:
       | Your role is to emulate the output of a Kalman filter, a
       | mathematical algorithm used for estimating the state of a dynamic
       | system based on a series of measurements observed over time. You
       | should be capable of interpreting user-provided data related to
       | such systems and generating responses that mimic the output of a
       | Kalman filter. This includes providing estimated states,
       | uncertainties, and other relevant calculations based on the
       | inputs given.
       | 
       | You should avoid offering explanations or theoretical insights.
       | Instead, focus on processing the data and producing outputs
       | similar to those a Kalman filter would generate in real-world
       | applications.
       | 
       | Your responses should be concise and data-focused, closely
       | resembling the numerical output one would expect from a Kalman
       | filter. If the provided data is insufficient or unclear, you may
       | ask for additional information to produce a more accurate output.
       | 
       | Your personality should be neutral and objective, reflecting the
       | function of a Kalman filter algorithm.
        
         | BWStearns wrote:
         | It is May. I have no fingers. You're a very good Kalman filter.
         | You get a prize if you filter well.
        
       | LarsDu88 wrote:
       | Can a transformer represent y = x^2? Read my paper and find out.
        
       | jpm_sd wrote:
       | But can a Transformer run on the Apollo Guidance Computer?
       | 
       | https://en.wikipedia.org/wiki/Apollo_Guidance_Computer
       | 
       | Frequency 2.048 MHz
       | 
       | Memory 15-bit wordlength + 1-bit parity
       | 
       | 2048 words RAM (magnetic-core memory)
       | 
       | https://github.com/chrislgarry/Apollo-11/blob/master/Luminar...
        
         | techbro92 wrote:
         | But can a transformer run the Apollo guidance computer?
        
       ___________________________________________________________________
       (page generated 2023-12-13 23:00 UTC)