[HN Gopher] Why tensors? A beginner's perspective
___________________________________________________________________
Why tensors? A beginner's perspective
Author : mfn
Score : 131 points
Date : 2022-03-09 02:20 UTC (1 days ago)
(HTM) web link (mfaizan.github.io)
(TXT) w3m dump (mfaizan.github.io)
| 725686 wrote:
| A wonderful little video to understand what tensors are, by
| Daniel Fleish:
|
| https://www.youtube.com/watch?v=f5liqUk0ZTw
|
| Very simple and basic.
|
| Edit: incorrectly wrote vectors instead of tensors.
| saberience wrote:
| This doesn't seem like it's for beginners.
| brummm wrote:
| Hmm, this is stuff physicists learn in their first year
| undergrad classes for mathematical foundations. Seems to me
| it's the very definition of beginner.
| VeninVidiaVicii wrote:
| > Most commonly, a tensor is defined as being anything that
| transforms like a tensor.
|
| Definitely not beginner level.
| ericphanson wrote:
| I was happy to see that this article is actually talking about
| tensors, not just multidimensional arrays (which for some reasons
| are often called tensors by machine learning folks).
| lalaithion wrote:
| This is mostly a semantic argument, but I find this to be a
| very annoying perspective. Given a basis, there is a natural
| isomorphism between tensors of a certain type and
| multidimensional arrays of certain dimensions.
| edflsafoiewq wrote:
| Of course there is, but if you perform an operation on a
| multidimensional array, there is no guarantee it corresponds
| to an operation on tensors, ie. the resulting tensor may
| depend on the basis.
| lalaithion wrote:
| Sure, if you perform an _arbitrary_ operation on a
| multidimensional array. But the same is true of any
| representation of any mathematical object. It makes no
| physical sense to take the sine of a mass, or two to the
| power of a length. But that doesn 't mean that whenever
| someone says "oh, the mass of an object is a real number" I
| need to nitpick them.
| kaashif wrote:
| It's not clear to me what you're annoyed about exactly. The way
| I see it, there are a few options:
|
| You're getting annoyed that people are confusing the map with
| the territory [1]. Multidimensional arrays with certain
| properties can be used to represent tensors, but aren't
| tensors. In the same way a diagram of torus isn't a topological
| space, or a multiplication table isn't a group, or a matrix is
| not a linear map. Isomorphic but not literally the thing.
|
| Or you're annoyed that people forget an array representing a
| tensor needs to satisfy some transformation law and can't just
| be any big array with some numbers in it.
|
| Or maybe you're a fan of basis-free linear algebra!
|
| Which one is it?
|
| 1: https://en.wikipedia.org/wiki/Map%E2%80%93territory_relation
| ummonk wrote:
| I personally am a fan of basis free linear algebra.
|
| More importantly though, "tensors" as commonly used in
| machine learning seem to rely on a single special basis, so
| they really are just multidimensional arrays. A machine
| learning algorithm isn't really invariant under a change of
| basis. For example, the ReLU activation function is not
| independent of a change of basis.
| jstx1 wrote:
| Same word, different contexts and meaning. In ML tensors _are_
| multidimensional arrays (and nothing more). Neither physicists
| nor ML researchers /developers are confused about what it
| means.
| lysecret wrote:
| I am a bit confused
| lanstin wrote:
| I'm pretty sure that a multidimensional array was the
| original implementation of tensors and the more fancy linear
| forms formalism came later in the twentieth century. Then the
| question was how do these multidimensional arrays operate on
| vectors and how do they transport around a manifold. (Source:
| first book I read on general relativity in the 1970s had a
| lot of pages about multidimensional arrays and also this good
| summary of the history:
| https://math.stackexchange.com/questions/2030558/what-is-
| the...
|
| (Sort of like how vectors kind of got going via a list of
| numbers and then they found the right axioms for vector
| spaces and then linear algebra shifted from a lot of
| computation to a sort of spare and elegant set of theorems on
| linearity).
| westurner wrote:
| AFAIU, Matrices are categorically subsets of Tensors where
| the product operator, at least, is not the tensor product
| but the Dot product.
|
| Dot product: https://en.wikipedia.org/wiki/Dot_product
|
| Matrix multiplication > Dot product, bilinear form and
| inner product: https://en.wikipedia.org/wiki/Matrix_multipl
| ication#Dot_prod...
|
| > _The dot product of two column vectors is the matrix
| product_
|
| Tensor > Geometric objects
| https://en.wikipedia.org/wiki/Tensor :
|
| > _The transformation law for a tensor behaves as a functor
| on the category of admissible coordinate systems, under
| general linear transformations (or, other transformations
| within some class, such as local diffeomorphisms.) This
| makes a tensor a special case of a geometrical object, in
| the technical sense that it is a function of the coordinate
| system transforming functorially under coordinate
| changes.[24] Examples of objects obeying more general kinds
| of transformation laws are jets and, more generally still,
| natural bundles.[25][26]_
| hnarayanan wrote:
| I was confused by this when I first got to ML.
| cloogshicer wrote:
| > Neither physicists nor ML researchers/developers are
| confused about what it means.
|
| I'm sure this has confused _a lot_ of people, especially
| beginners. Clashing terminology is one of the main
| difficulties in interdisciplinary work, in my experience. I
| don 't think it's good to shrug it off like that.
| russellbeattie wrote:
| I'm literally learning this, or at least being reminded of
| something I had totally forgotten, as I read this thread.
| So yeah.
|
| Words can have multiple meanings, but I think we can all
| agree it's preferable if they don't have multiple _slightly
| different depending on context_ meanings. That 's just
| confusing.
| The_rationalist wrote:
| What is the difference?
| kgwgk wrote:
| Something like the difference between a "vector" of length N
| which is just a collection of N numbers and one that is a
| representation of a N-dimensional geometric algebra object.
| icapybara wrote:
| Tensors have additional properties that arrays don't
| necessarily have. For example, the coordinate system
| transform rule that the author describes in the beginning of
| the post.
|
| One of my old physics professors taught us to think of
| tensors as "arrays with units." If it's a
| vector/matrix/higher dimensional array but has physical
| units, it's probably a tensor. The fact that it has units
| means it represents something physical which must obey
| additional constraints (like the coordinate system
| transformation rule).
| omarhaneef wrote:
| Since I also thought Tensors were just higher dimension
| arrays, isn't this really what ML folks think Tensors are,
| since they (we?) do attach units to the Tensors most of the
| time?
| ogogmad wrote:
| The physicist's approach is a bit non-conceptual. From a
| mathematical point of view, a tensor is essentially an
| arbitrary multi-linear map. Think of the dot product, the
| determinant of a matrix (which is linear on each _column_
| but is not linear on a matrix), the exterior product in
| exterior algebra (or geometric algebra), a linear map
| itself (which is obviously a special case of a
| multilinear map), etc.
|
| The coordinate change stuff that physicists talk about
| stems from observing that a matrix can be used to
| represent _some_ tensors, but the rule for changing basis
| changes along with the kind of tensor. So if M is a
| matrix which represents a linear map and P is a matrix
| whose columns are basis vectors, then PMP^{-1} is the
| same linear map as M but in basis P; if on the other hand
| the matrix M represents a bilinear form as opposed to a
| linear map, then the basis change formula is actually
| PMP^T, where we use the matrix transpose. _Sylvester 's
| Law Of Inertia_ is then a non-trivial observation about
| matrix representations of bilinear forms.
|
| Physicists conflate a tensor with its representation in
| some coordinate system. Then they show how changing the
| coordinate system changes the coordinates. This point of
| view does provide some concrete intuition, though, so
| it's not all bad. By a coordinate system, I mean a linear
| basis.
|
| Hope that helps.
| cygx wrote:
| _Physicists conflate a tensor with its representation in
| some coordinate system_
|
| Rather, mathematicians that complain about the
| physicist's approach just haven't advanced far enough in
| their studies to understand how vector bundles are
| associated to the frame bundle ;)
| ummonk wrote:
| Not all physicists make that conflation - Wald's book in
| General Relativity emphasizes tensors as abstract
| concepts, and then details how they can be expressed in
| terms of a basis.
|
| The OP also emphasizes the abstract interpretation as
| providing more intuition than the coordinate
| transformation rule.
| ajkjk wrote:
| There are really just two things called tensors.
|
| Physicists' tensors = generalization of arrays with
| units; have to transform according to certain coordinate
| laws.
|
| Mathematicians' tensors = generalization of arrays,
| transformation rules don't matter.
| red_trumpet wrote:
| But they are not totally different! The Physicists'
| tensors are actually Mathematicians' tensors, but
| parametrized by some parameters (coordinates). Then you
| have special laws what happens if you make a (possibly
| non-linear) change of the coordinates. See
| https://en.wikipedia.org/wiki/Tensor#Tensor_fields
| klodolph wrote:
| > Mathematicians' tensors = generalization of arrays,
| transformation rules don't matter.
|
| That's definitely inaccurate, at least it doesn't match
| what I think of as tensors in mathematics.
|
| In mathematics, tensors are the most general result of a
| bilinear opteration. This does imply that they transform
| according to certain laws: if you represent the tensor
| using some particular basis, that basis can be expressed
| in the original vector spaces you multiplied, and
| choosing a different basis for your vector spaces results
| in a different basis for your tensors.
|
| By "most general bilinear operation" I am talking about
| what is expressed in category theory as a universal
| property... with a morphism that preserves bilinear maps.
|
| Tensors can be over multiple vector spaces or a single
| vector space (in which case it's typically implied that
| it's over the vector space and its dual). When you use a
| vector space and its dual, I believe you get the kind of
| tensor that physicists deal with, and all of the same
| properties. Note that while vector spaces and their dual
| may seem to be equivalent at first glance (and they are
| isomorphic in finite-dimensional cases), both
| mathematicians and physicists must know that they have
| different structure and transform differently.
|
| Something that will throw you off is that mathematicians
| often like to use category theory and "point free"
| reasoning where you talk about vector spaces and tensor
| products in terms of things like objects and morphisms,
| and often avoid talking about actual vectors and tensors.
| Physicists talk about tensors using much more concrete
| terms and specify coordinate systems for them. It can
| require some insight in order to figure out that
| mathematicians and physicists are actually talking about
| the same thing, and figure out how to translate what a
| physicist says about a tensor to what a mathematician
| says.
| uoaei wrote:
| This is a pretty tired line, I gotta say.
|
| Obviously they are not. One is a linear operator, the other
| is a data structure for implementing computations using
| that operator. This description extends to all tensors.
|
| It's like saying "queues are not just lists". That is true
| and also neither insightful nor helpful.
|
| I don't see it as mystifying or complicated, what am I
| missing?
| ummonk wrote:
| Sure but if you're working with lists you shouldn't call
| them queues. Similarly if you're working with mere
| multidimensional arrays don't call them tensors.
| Beldin wrote:
| The way I think of it: you have 0-dimensional arrays of numbers
| (plain numbers or scalars). You have 1-dimensional arrays of
| numbers (a list of N numbers or an N-vector). You have
| 2-dimensional arrays of numbers (an NxM matrix). We can extend
| this concept to 3- and 4-dimensional arrays and even further.
|
| The kicker? _All_ of them are tensors. Tensor is just a
| generalisation of the concept.
|
| I am no licensed mathematician, so this could be off. However,
| every time I dive into this topic, I have to wade through way too
| complex mathnobabble to arrive at that notion. So let's keep it
| simple: tensors are a mathematician's template for arrays of any
| dimension.
| bmitc wrote:
| Anyone interested in a visual exploration should checkout
| _Geometrical Vectors_ by Gabriel Weinreich.
|
| https://www.maa.org/press/maa-reviews/geometrical-vectors
| billfruit wrote:
| Is there any book that treats whole off geometry using vectors?
| bmitc wrote:
| I'm not sure I understand the question enough to answer. Do
| you mean something like differential geometry? There, the
| theory is built upon vectors and covectors (i.e.,
| differential forms) that are associated with tangent spaces
| and cotangent spaces, respectively. But that is modern
| differential geometry and not classical geometry.
| Koshkin wrote:
| Here is a _really_ good resource for a beginner:
|
| https://grinfeld.org/books/An-Introduction-To-Tensor-Calculu...
| ok123456 wrote:
| Is that you Pavel?
| beaconstudios wrote:
| OK that helps me to understand why tensorflow is called what it
| is - if a tensor turns a set of vectors into a scalar that's
| exactly what an artificial neuron does with weights and inputs,
| and they are linked up to form a data flow graph.
| chobytes wrote:
| My version is just: Tensors allow us to write data and operations
| on data in a way which does not depend on how we chose to
| represent them.
|
| For example, if I have a vector x in V and a map T from V to W,
| then I would like the truth of T(x)=y to be independent of how I
| represent T and x.
| zardo wrote:
| I like the concrete example from when I first used tensors in
| school. Stress in a block of concrete. You can choose any basis
| you like to represent the stresses and transform between them.
|
| Whether or not the concrete block breaks under that stress
| obviously does not depend on your choice of basis or units, so
| your transformation rules had better reflect that reality.
| xyzzyz wrote:
| That was explanation from a perspective of someone acquainted
| with modern physics. As such, it will make sense to physicist,
| but no sense to most everyone else, including mathematicians who
| don't know modern physics.
|
| For example, in the beginning, author describes tensors as things
| behaving according to tensor transformation formula. This is
| already very much a physicist kind of thinking: it assumes that
| there is some object out there, and we're trying to understand
| what it is in terms of how it behaves. It also uses the summation
| notation which is rather foreign to non-physicist mathematicians.
| Then, when it finally reaches the point where it is all related
| to tensors in TensorFlow sense, we find that there is no
| reference made to the transformation formula, purportedly so
| crucial to understanding tensors. How comes?
|
| The solution here is quite simple: what author (and physicists)
| call tensors is not what TensorFlow (and mathematicians) call
| tensors. Instead, author describes what mathematicians call "a
| tensor _bundle_ ", which is a correspondence that assigns each
| point of space a unique _tensor_. That's where the transformation
| rule comes from: if we describe this mapping in terms of some
| coordinate system (as physicist universally do), the
| transformation rule tells you how to this description changes in
| terms of change of the coordinates. This setup, of course, has
| little to do with TensorFlow, because there is no space that its
| tensors are attached to, they are just standalone entities.
|
| So what are the mathematician's (and TensorFlow) tensors? They're
| actually basically what the author says, after very confusing and
| irrelevant introduction talking about change of coordinates of
| underlying space -- irrelevant, because TensorFlow tensors are
| not attached as a bundle to some space (manifold) as they are on
| physics, so no change of space coordinates ever happens. Roughly,
| tensors are a sort of universal objects representing multi linear
| maps: bilinear maps V x W -> R correspond canonically one-to-one
| to regular linear maps V (x) W -> R, where V (x) W is a vector
| space called tensor product of V and W, and tensors are simply
| vectors in this tensor product space.
|
| Basically, the idea is to replace weird multi linear objects with
| normal linear objects (vectors), that we know how to deal with,
| using matrix multiplication and stuff. That's all there is to it.
| mr_mitm wrote:
| Thanks for this summary.
|
| Even as a physicist I found it highly confusing when I got told
| in physics classes that a tensor is "just a thing (or object)
| that behaves like so under coordinate transformation". Like,
| what do you mean by "thing"? I have no intuition to this yet, I
| need it concise definitions! Fortunately I took a differential
| geometry class at the same time, which was really helpful.
| ABeeSea wrote:
| > Instead, author describes what mathematicians call "a tensor
| bundle", which is a correspondence that assigns each point of
| space a unique tensor.
|
| Technically, that's a tensor field which is a section of the
| tensor bundle. Similarly, a vector field is a section of the
| tangent bundle (the collection of all the tangent spaces of the
| points on the manifold). A vector field is just a choice of a
| tangent vector for each point from that point's tangent space.
| hinkley wrote:
| > author describes tensors as things behaving according to
| tensor transformation formula
|
| In grade school it drove me nuts when the homework required us
| to describe a word without using the word (or it's Latinate
| siblings). And yet as an adult there are few enough weeks that
| go by where some grownup doesn't try to pull that same trick.
|
| If you think developers are guilty of circular logic, check out
| some of the math pages on Wikipedia. You can get lost in
| moments.
| jhokanson wrote:
| Speaking of math pages on Wikipedia ... and math text more
| generally
|
| Is it just me or are we horrible at teaching advanced math?
| Where are the examples (with actual numbers)? Where is the
| motivation? Where are the pictures?
| GoatOfAplomb wrote:
| > Where are the examples (with actual numbers)?
|
| In upper-level undergraduate math, I made a game of seeing
| how many pages I would go before seeing 7 printed anywhere.
| It was usually 10 pages, if I included the page numbers.
| chobytes wrote:
| This is definitely a problem! Having a large set of
| interests and problems to draw examples and intuition from
| are how I deal with it. I suspect this is why so many
| mathematicians are also into physics.
| exdsq wrote:
| 100%! For those of us who need to learn from practical
| examples through to generalized intuition maths can be
| really really hard to learn depending on the source. Wish
| I was one of those people who finds it easier to learn
| from abstract first through to implementations second.
| joppy wrote:
| Wikipedia is a terrible place to learn advanced
| mathematics, for the reasons you raise (and more). There
| are lots of terrific short books, and many terrific
| lectures online.
| hinkley wrote:
| Randall Monroe has a comic about how most people need
| enough math to be able to handle a birthday dinner where
| the guests split the bill for the birthday boy/girl evenly
| and pay for their meals and tip separately.
|
| That's a pretty good bar and I wonder if we could just cut
| to that chase earlier. But I also believe that people need
| enough math to see when they're being cheated, and I feel
| like you could just tell middle schoolers that and they
| would pay attention. Maybe even primary school.
|
| You told Billy he could have three apples, and now there
| are two left. Did Billy take more apples than he should
| have?
|
| It's always how do you share your cookies fairly with your
| friends and if they're my cookies why do I have to share
| them at all? Screw "fairly" I'm keeping the extras at
| least. That sort of sharing is a socially advanced concept
| they don't entirely get just yet.
| chobytes wrote:
| I think a lot of circularity occurs in mathematics because we
| don't typically qualify our utterances when it can be
| implicitly understood.
|
| Eg "Numbers (formal) are those objects which behave like
| numbers (informal)."
| dboreham wrote:
| Bertrand Russell turns over in grave.
| hansen wrote:
| To be a bit pedantic: the identification of tensors with
| multilinear forms requires finite dimensions (or reflexive
| topological spaces).
| tagrun wrote:
| It has nothing to do with tensor fields, uniform/constant
| tensors still obey the proper coordinate transformations,
| that's the defining property of any tensor. (With non-uniform
| tensor fields, covariant derivatives also pick up a correction,
| but that's a separate thing.)
|
| TensorFlow "tensor"(and most other use of "tensor" in
| programmer jargon) is not a tensor at all, it's just a
| multidimensional array.
| catgary wrote:
| What do you think the tensor product of finite dimensional
| vector spaces looks like?
| contravariant wrote:
| Mathematicians would disagree with you there. There are no
| coordinates to transform in an ordinary tensor space and
| therefore no way for a tensor to be affected by such a
| transformation.
|
| Matrices (or linear transformations in general) are important
| examples of tensors. There's a nice adjunction between tensor
| spaces A(x)B and the space of linear transformations B=>C
| given by:
|
| Hom(A(x)B, C) = Hom(A, B=>C)
|
| In the case of Tensorflow I think they do actually still talk
| about linear transformations of some kind so it's perfectly
| fine to call them tensors.
| cygx wrote:
| _There are no coordinates to transform in an ordinary
| tensor space and therefore no way for a tensor to be
| affected by such a transformation._
|
| Sure there are: Any basis of the underlying vector space(s)
| induces a basis of the tensor space. Components respective
| to some basis are coordinates. You can then investigate
| what happens to the induced basis (or rather, the
| respective components) under a basis transformation of the
| underlying vector space(s), which is where the
| "physicist's" definition of tensors originates.
| contravariant wrote:
| The components of a vector aren't the same as the
| coordinates physicists talk about when they're dealing
| with tensors. The components would be something like the
| value of the magnetic potential, or the local wind speed.
| The coordinates would be the location where that
| particular vector is 'anchored'.
|
| A change of coordinates does indeed induce a change of
| basis, but a change of basis isn't really a change of
| coordinates. And strictly speaking some vector spaces
| don't really have an obvious basis (without invoking
| choice), so having a basis be a prerequisite for the
| definition is not ideal.
|
| The whole requirement that a tensor is 'something that
| transforms like [...] under a coordinate transformation'
| is just how physicists have chosen to phrase that a
| vector bundle is only well defined if it's definition
| isn't dependent on some arbitrary choice of coordinates.
| In my opinion this requirement is more easily apparent in
| the mathematical definition where _there is no choice of
| coordinates in the first place_ , rather than the
| physicists way of working with some choice of coordinates
| and checking how things transform.
| cygx wrote:
| I'm aware. Though if we want to be more precise, that's
| about tensor fields, where the basis transformations of
| the underlying vector bundles (the tangent and cotangent
| bundle) are in turn induced by coordinate transformations
| of the base manifold.
|
| However, physicists get introduced to tensors far earlier
| than any excursions into differential geometry when
| discussing rigid bodies.
| contravariant wrote:
| Yes I'd also call those tensor fields. The main point I'm
| trying to make is that the tensor transformation law
| _only_ makes sense for such fields.
| cygx wrote:
| The terms co- and contravariant make sense on a purely
| algebraic basis, with components of tensors transforming
| 'the same as' or 'opposite to' the basis vectors. That
| the basis transformation is induced by transformations of
| some base manifold is incidental.
| ummonk wrote:
| Exactly. The fact that the bases are related to
| coordinates on the manifold is a property of differential
| geometry but the laws for transformation between bases
| are more general.
| kaashif wrote:
| > author describes tensors as things behaving according to
| tensor transformation formula
|
| Yeah, the idea that there are pre-existing things that we're
| trying to describe is somewhat weird to me when we're trying to
| come up with a definition of a tensor. The whole point of
| mathematics is that you come up with the definitions and
| theorems fall out.
|
| In particular, this comment is funny and speaks to some
| difference in how I and the author view what we're doing when
| defining a tensor:
|
| > But why that specific transformation law - why must tensors
| transform in that way in order to preserve whatever object the
| tensor represents?
|
| Because we defined it like that! When you make the definition
| "a tensor is a thing that follows X laws", you don't get to ask
| _why_ , you just defined it!
|
| Just a funny bit of phrasing, I get what is meant :)
| edflsafoiewq wrote:
| > The whole point of mathematics is that you come up with the
| definitions and theorems fall out.
|
| That's just how it's presented in textbooks. It's obviously
| not math is actually done.
| ummonk wrote:
| The post was also a poor explanation for someone doing modern
| physics. [edit: not true actually I should have read the rest
| of the post - it's a good post]
|
| Wald's approach in General Relativity is much better - he
| treats Tensors as a multilinear map from vectors and dual
| vectors to scalars.
|
| He then derives the underlying coordinate transformaton rules,
| for the vector spaces used in differential geometry. But
| mfn wrote:
| That's the approach I used as well in the second half of the
| article - I just mentioned the transformation law in the
| beginning since that's what most physics students encounter
| first.
|
| Most of the article tries to provide some intuition behind
| why multilinear maps, which sound like a fairly abstract
| concept, might be relevant in physics. The key link being the
| importance of coordinate invariance.
|
| I didn't go into deriving the coordinate transforms from the
| multilinear map definition as I didn't feel that it'd provide
| much better intuition, but I did mention the equivalence near
| the end.
| ummonk wrote:
| Yeah sorry you're right - I should have read the rest of
| your post, which is excellent and describes precisely why
| the coordinates/transformations focused definition is bad
| for one's intuition.
| mbbutler wrote:
| Why are you complaining that the author didn't talk about
| tensors as they are used in tensorflow? Tensorflow is never
| even mentioned in the piece.
|
| The author is perfectly clear in the first sentence that the
| piece's focus is about the usefulness of tensors in a physics
| context.
| xyzzyz wrote:
| Huh, you're right, not sure why I thought it's TensorFlow
| related.
___________________________________________________________________
(page generated 2022-03-10 23:00 UTC)