[HN Gopher] Swin Transformer: Hierarchical Vision Transformer Us...
___________________________________________________________________
Swin Transformer: Hierarchical Vision Transformer Using Shifted
Windows
Author : jonbaer
Score : 39 points
Date : 2023-04-09 16:12 UTC (6 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| sroussey wrote:
| Should add (2021)
| nharada wrote:
| Researchers love the ViT and all its varieties, but a reminder
| for those following at home that ConvNets still work and scale
| fine, depending on your requirements. For example, in "A ConvNet
| for the 2020s"[1], the authors are able to scale up ConvNets to
| the sizes of ViTs.
|
| From the abstract: "Constructed entirely from standard ConvNet
| modules, ConvNeXts compete favorably with Transformers in terms
| of accuracy and scalability, achieving 87.8% ImageNet top-1
| accuracy and outperforming Swin Transformers on COCO detection
| and ADE20K segmentation"
|
| [1] https://arxiv.org/abs/2201.03545
| godelski wrote:
| It is also important to remember that there are different
| inductive biases for different networks. Sutton's "Bitter
| Lesson"[0] argues for scale + more arbitrary complexity. ViTs
| scale really well because some of this arbitrary complexity.
| But I agree with your point that what to use is not obvious and
| this gets really complicated real quickly.
|
| For example, Swin loses some of the properties of self-
| attention but Neighborhood Attention doesn't[1] (these are both
| considered reductive attention types). Does this have a large
| effect? Might depend on your task.
|
| Looking at non-classification tasks is important. This is
| especially important since ImageNet has a lot of issues with
| redundancy (e.g. there's a label "sunglass" (836) and
| "sunglasses" (837)), images with multiple labels that are
| valid, and more. I'd argue that once ImageNet accuracy is over
| 80% then the accuracy no longer strongly correlates with
| downstream tasks[2] (segmentation, detection) or even other
| tasks like generation. This is why it is really important for
| researchers to pay closer attention now and we can't just look
| at benchmarks. Doing so will hinder research. We could
| previously get away with this because classification previously
| strongly correlated with downstream performance and that the
| error rate in ImageNet was much larger than the improvements on
| accuracy.
|
| Worse than that, some of the main benchmarks we use are highly
| effected by these biases. For example, convolutions learn
| texture and so using something like FID[3,4] can have plenty of
| issues that might not give an actual depiction of how good a
| network actually is at its task. This is even true for non-deep
| model based metrics[5] and so you have to be REALLY careful
| about how you evaluate things.
|
| TLDR: be careful with evaluating benchmarks and evaluate things
| holistically.
|
| ===== Minimal Bib =====
|
| [0] http://www.incompleteideas.net/IncIdeas/BitterLesson.html
|
| [1] https://arxiv.org/abs/2204.07143
|
| [2] If you're wondering how this can happen, it is
| memorization. E.g. if the target image has both rabbits, cars,
| and other things (real example) in it but the label is "car
| wheel" (479) then the network has to learn to ignore "car
| mirror" (475) and rabbits (330,331,332). But downstream tasks
| like detection and segmentation perform multiple
| classifications on a single image and using a over-fit backbone
| can hinder performance.
|
| [3] https://arxiv.org/abs/2203.06026
|
| [4] It is worth noting that FID is calculated form InceptionV3
| weights, which was only trained on ImageNet-1k (full dataset is
| 22k) and had an accuracy <80% (top-5 < 95%). It is a bit weird
| this is used for datasets like FFHQ because there is no person
| label. Or even most LSUN classes because ImageNet likely has a
| texture bias itself (with animals and plants composing the
| majority of the dataset and texture being an important feature
| there). Evaluating models is fucking hard and a lot of this
| isn't internalized my many, especially outside the field.
|
| [5] https://arxiv.org/abs/1511.01844
| panabee wrote:
| given the focus on transformers in recent years, modernizing
| and adapting CNNs to hardware advances seems like an under-
| researched area. it's interesting and something we're
| exploring.
|
| FAIR did great work with this paper.
| macrolocal wrote:
| Nb. There are plenty of specialized architectures for
| convolutions: Nervana, Habana, Groq, etc.
| thatsadude wrote:
| You should see MobileOne and FastVIT from Apple
| pstoll wrote:
| Not paying too close attention, but this sounds a lot like the
| Gabor filter stacking from days of yore.
|
| Ah looks like people are still looking at them.
| https://bihy.medium.com/improving-convolutional-neural-netwo...
| logophobia wrote:
| Reminds me of the ideas behind google's multi-axis transformer:
| https://arxiv.org/abs/2204.01697
|
| Both using a hierarchical transformer, adapting the transformer
| network architecture to vision tasks more efficiently.
| heyitsguay wrote:
| Something I've noticed is that there's a ton of work going into
| better vision transformers, but whenever there's a big new
| multimodal result or something, everyone's just using ViT. What
| gives? Like in CLIP or PaLM-E, they very easily could have used
| another vision architecture, it's not like they were running all
| modalities through the same encoder.
___________________________________________________________________
(page generated 2023-04-09 23:01 UTC)