Neighborhood Attention Transformer
Clicks: 133
ID: 282328
2022
Article Quality & Performance Metrics
Overall Quality
Not rated
Combines reader engagement with the AI quality analysis. This
article has not been analysed, so there is no overall score —
reader engagement is measured and shown alongside.
Reader Engagement
Steady Performance
30.0
/100
133 views
50 readers
AI Quality Assessment
Not analyzed
Readership in this journal
SteadyRanked #24 of 803 articles by views in arXiv
Most read
Least read
Bar heights use a square-root scale. Only the 120 most-read articles are drawn; the journal has 803 in total.
Mint this article as an NFT
Not yet mintedCreate a permanent, verifiable on-chain record of this article on the Scimatic Network. The NFT is held in your Journament account, and you can withdraw it to your own wallet at any time.
5
SUSD
one-off · no wallet required
Abstract
We present Neighborhood Attention (NA), the first efficient and scalable
sliding-window attention mechanism for vision. NA is a pixel-wise operation,
localizing self attention (SA) to the nearest neighboring pixels, and therefore
enjoys a linear time and space complexity compared to the quadratic complexity
of SA. The sliding-window pattern allows NA's receptive field to grow without
needing extra pixel shifts, and preserves translational equivariance, unlike
Swin Transformer's Window Self Attention (WSA). We develop NATTEN (Neighborhood
Attention Extension), a Python package with efficient C++ and CUDA kernels,
which allows NA to run up to 40% faster than Swin's WSA while using up to 25%
less memory. We further present Neighborhood Attention Transformer (NAT), a new
hierarchical transformer design based on NA that boosts image classification
and downstream vision performance. Experimental results on NAT are competitive;
NAT-Tiny reaches 83.2% top-1 accuracy on ImageNet, 51.4% mAP on MS-COCO and
48.4% mIoU on ADE20K, which is 1.9% ImageNet accuracy, 1.0% COCO mAP, and 2.6%
ADE20K mIoU improvement over a Swin model with similar size. To support more
research based on sliding-window attention, we open source our project and
release our checkpoints at:
https://github.com/SHI-Labs/Neighborhood-Attention-Transformer .
| Reference Key |
shi2022neighborhood
Use this key to autocite in the manuscript while using
SciMatic Manuscript Manager or Thesis Manager
|
|---|---|
| Authors | Ali Hassani; Steven Walton; Jiachen Li; Shen Li; Humphrey Shi |
| Journal | arXiv |
| Year | 2022 |
| DOI |
DOI not found
|
| URL | |
| Keywords |
Citations
No citations found. To add a citation, contact the admin at info@scimatic.org
Comments
No comments yet. Be the first to comment on this article.