https://pytorch.org/tutorials/prototype/vulkan_workflow.html

 

  * Get Started
  * Ecosystem
  * Mobile
  * Blog
  * Tutorials
  * Docs
    PyTorch

    torchaudio

    torchtext

    torchvision

    torcharrow

    TorchData

    TorchRec

    TorchServe

    TorchX

    PyTorch on XLA Devices

  * Resources
    About

    Learn about PyTorch's features and capabilities

    PyTorch Foundation

    Learn about the PyTorch foundation

    Community

    Join the PyTorch developer community to contribute, learn, and
    get your questions answered.

    Community Stories

    Learn how our community solves real, everyday machine learning
    problems with PyTorch.

    Developer Resources

    Find resources and get questions answered

    Events

    Find events, webinars, and podcasts

    Forums

    A place to discuss PyTorch code, issues, install, research

    Models (Beta)

    Discover, publish, and reuse pre-trained models

  * GitHub

 
Table of Contents  
1.13.1+cu117
[                    ] 
PyTorch Recipes

  * See All Recipes
  * See All Prototype Recipes

Introduction to PyTorch

  * Quickstart
  * Tensors
  * Datasets & DataLoaders
  * Transforms
  * Build the Neural Network
  * Automatic Differentiation with torch.autograd
  * Optimizing Model Parameters
  * Save and Load the Model

Introduction to PyTorch on YouTube

  * Introduction to PyTorch - YouTube Series
  * Introduction to PyTorch
  * Introduction to PyTorch Tensors
  * The Fundamentals of Autograd
  * Building Models with PyTorch
  * PyTorch TensorBoard Support

Learning PyTorch

  * Deep Learning with PyTorch: A 60 Minute Blitz
  * Learning PyTorch with Examples
  * What is torch.nn really?
  * Visualizing Models, Data, and Training with TensorBoard

Image and Video

  * TorchVision Object Detection Finetuning Tutorial
  * Transfer Learning for Computer Vision Tutorial
  * Adversarial Example Generation
  * DCGAN Tutorial
  * Spatial Transformer Networks Tutorial
  * Optimizing Vision Transformer Model for Deployment

Audio

  * Audio I/O
  * Audio Resampling
  * Audio Data Augmentation
  * Audio Feature Extractions
  * Audio Feature Augmentation
  * Audio Datasets
  * Speech Recognition with Wav2Vec2
  * Speech Command Classification with torchaudio
  * Forced Alignment with Wav2Vec2

Text

  * Language Modeling with nn.Transformer and TorchText
  * Fast Transformer Inference with Better Transformer
  * NLP From Scratch: Classifying Names with a Character-Level RNN
  * NLP From Scratch: Generating Names with a Character-Level RNN
  * NLP From Scratch: Translation with a Sequence to Sequence Network
    and Attention
  * Text classification with the torchtext library

Reinforcement Learning

  * Reinforcement Learning (DQN) Tutorial
  * Train a Mario-playing RL Agent

Deploying PyTorch Models in Production

  * Introduction to TorchScript
  * Loading a TorchScript Model in C++
  * Real Time Inference on Raspberry Pi 4 (30 fps!)

Code Transforms with FX

  * (beta) Building a Simple CPU Performance Profiler with FX

Frontend APIs

  * (beta) Channels Last Memory Format in PyTorch
  * Forward-mode Automatic Differentiation (Beta)
  * Using the PyTorch C++ Frontend
  * Dynamic Parallelism in TorchScript
  * Autograd in C++ Frontend

Extending PyTorch

  * Double Backward with Custom Functions
  * Fusing Convolution and Batch Norm using Custom Function
  * Custom C++ and CUDA Extensions
  * Extending TorchScript with Custom C++ Operators
  * Extending TorchScript with Custom C++ Classes
  * Registering a Dispatched Operator in C++
  * Extending dispatcher for a new backend in C++

Model Optimization

  * PyTorch Profiler With TensorBoard
  * Optimizing Vision Transformer Model for Deployment
  * Pruning Tutorial
  * (beta) Dynamic Quantization on an LSTM Word Language Model
  * (beta) Dynamic Quantization on BERT
  * (beta) Quantized Transfer Learning for Computer Vision Tutorial
  * (beta) Static Quantization with Eager Mode in PyTorch
  * Grokking PyTorch Intel CPU performance from first principles
  * Grokking PyTorch Intel CPU performance from first principles
    (Part 2)
  * Getting Started - Accelerate Your Scripts with nvFuser
  * Multi-Objective NAS with Ax
  * torch.compile Tutorial

Parallel and Distributed Training

  * Distributed and Parallel Training Tutorials
  * PyTorch Distributed Overview
  * Distributed Data Parallel in PyTorch - Video Tutorials
  * Single-Machine Model Parallel Best Practices
  * Getting Started with Distributed Data Parallel
  * Writing Distributed Applications with PyTorch
  * Getting Started with Fully Sharded Data Parallel(FSDP)
  * Advanced Model Training with Fully Sharded Data Parallel (FSDP)
  * Customize Process Group Backends Using Cpp Extensions
  * Getting Started with Distributed RPC Framework
  * Implementing a Parameter Server Using Distributed RPC Framework
  * Distributed Pipeline Parallelism Using RPC
  * Implementing Batch RPC Processing Using Asynchronous Executions
  * Combining Distributed DataParallel with Distributed RPC Framework
  * Training Transformer models using Pipeline Parallelism
  * Distributed Training with Uneven Inputs Using the Join Context
    Manager

Mobile

  * Image Segmentation DeepLabV3 on iOS
  * Image Segmentation DeepLabV3 on Android

Recommendation Systems

  * Introduction to TorchRec
  * Exploring TorchRec sharding

Multimodality

  * TorchMultimodal Tutorial: Finetuning FLAVA

  * Tutorials >
  * PyTorch Vulkan Backend User Workflow
  * [view-page-]

Shortcuts
prototype/vulkan_workflow
[pytorch-co]
Run in Google Colab
Colab
[pytorch-do]
Download Notebook
Notebook
[pytorch-gi]
View on GitHub
GitHub

PyTorch Vulkan Backend User WorkflowP

Author: Ivan Kobzarev

IntroductionP

PyTorch 1.7 supports the ability to run model inference on GPUs that
support the Vulkan graphics and compute API. The primary target
devices are mobile GPUs on Android devices. The Vulkan backend can
also be used on Linux, Mac, and Windows desktop builds to use Vulkan
devices like Intel integrated GPUs. This feature is in the prototype
stage and is subject to change.

Building PyTorch with Vulkan backendP

Vulkan backend is not included by default. The main switch to include
Vulkan backend is cmake option USE_VULKAN, that can be set by
environment variable USE_VULKAN.

To use PyTorch with Vulkan backend, we need to build it from source
with additional settings. Checkout the PyTorch source code from
GitHub master branch.

Optional usage of vulkan wrapperP

By default, Vulkan library will be loaded at runtime using the
vulkan_wrapper library. If you specify the environment variable 
USE_VULKAN_WRAPPER=0 libvulkan will be linked directly.

Desktop buildP

Vulkan SDKP

Download VulkanSDK from https://vulkan.lunarg.com/sdk/home and set
environment variable VULKAN_SDK

Unpack VulkanSDK to VULKAN_SDK_ROOT folder, install VulkanSDK
following VulkanSDK instructions for your system.

For Mac:

cd $VULKAN_SDK_ROOT
source setup-env.sh
sudo python install_vulkan.py

Building PyTorch:

For Linux:

cd PYTORCH_ROOT
USE_VULKAN=1 USE_VULKAN_SHADERC_RUNTIME=1 USE_VULKAN_WRAPPER=0 python setup.py install

For Mac:

cd PYTORCH_ROOT
USE_VULKAN=1 USE_VULKAN_SHADERC_RUNTIME=1 USE_VULKAN_WRAPPER=0 MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py install

After successful build, open another terminal and verify the version
of installed PyTorch.

import torch
print(torch.__version__)

At the time of writing of this recipe, the version is
1.8.0a0+41237a4. You might be seeing different numbers depending on
when you check out the code from master, but it should be greater
than 1.7.0.

Android buildP

To build LibTorch for android with Vulkan backend for specified 
ANDROID_ABI.

cd PYTORCH_ROOT
ANDROID_ABI=arm64-v8a USE_VULKAN=1 sh ./scripts/build_android.sh

To prepare pytorch_android aars that you can use directly in your
app:

cd $PYTORCH_ROOT
USE_VULKAN=1 sh ./scripts/build_pytorch_android.sh

Model preparationP

Install torchvision, get the default pretrained float model.

pip install torchvision

Python script to save pretrained mobilenet_v2 to a file:

import torch
import torchvision

model = torchvision.models.mobilenet_v2(pretrained=True)
model.eval()
script_model = torch.jit.script(model)
torch.jit.save(script_model, "mobilenet2.pt")

PyTorch 1.7 Vulkan backend supports only float 32bit operators. The
default model needs additional step that will optimize operators
fusing

from torch.utils.mobile_optimizer import optimize_for_mobile
script_model_vulkan = optimize_for_mobile(script_model, backend='vulkan')
torch.jit.save(script_model_vulkan, "mobilenet2-vulkan.pt")

The result model can be used only on Vulkan backend as it contains
specific to the Vulkan backend operators.

By default, optimize_for_mobile with backend='vulkan' rewrites the
graph so that inputs are transferred to the Vulkan backend, and
outputs are transferred to the CPU backend, therefore, the model can
be run on CPU inputs and produce CPU outputs. To disable this, add
the argument optimization_blocklist=
{MobileOptimizerType.VULKAN_AUTOMATIC_GPU_TRANSFER} to 
optimize_for_mobile. (MobileOptimizerType can be imported from 
torch.utils.mobile_optimizer)

For more information, see the torch.utils.mobile_optimizer API
documentation.

Using Vulkan backend in codeP

C++ APIP

at::is_vulkan_available()
auto tensor = at::rand({1, 2, 2, 3}, at::device(at::kCPU).dtype(at::kFloat));
auto tensor_vulkan = t.vulkan();
auto module = torch::jit::load("$PATH");
auto tensor_output_vulkan = module.forward(inputs).toTensor();
auto tensor_output = tensor_output.cpu();

at::is_vulkan_available() function tries to initialize Vulkan backend
and if Vulkan device is successfully found and context is created -
it will return true, false otherwise.

.vulkan() function called on Tensor will copy tensor to Vulkan
device, and for operators called with this tensor as input - the
operator will run on Vulkan device, and its output will be on the
Vulkan device.

.cpu() function called on Vulkan tensor will copy its data to CPU
tensor (default)

Operators called with a tensor on a Vulkan device as an input will be
executed on a Vulkan device. If an operator is not supported for the
Vulkan backend the exception will be thrown.

List of supported operators:

_adaptive_avg_pool2d
_cat
add.Scalar
add.Tensor
add_.Tensor
addmm
avg_pool2d
clamp
convolution
empty.memory_format
empty_strided
hardtanh_
max_pool2d
mean.dim
mm
mul.Scalar
relu_
reshape
select.int
slice.Tensor
transpose.int
transpose_
unsqueeze
upsample_nearest2d
view

Those operators allow to use torchvision models for image
classification on Vulkan backend.

Python APIP

torch.is_vulkan_available() is exposed to Python API.

tensor.to(device='vulkan') works as .vulkan() moving tensor to the
Vulkan device.

.vulkan() at the moment of writing of this tutorial is not exposed to
Python API, but it is planned to be there.

Android Java APIP

For Android API to run model on Vulkan backend we have to specify
this during model loading:

import org.pytorch.Device;
Module module = Module.load("$PATH", Device.VULKAN)
FloatBuffer buffer = Tensor.allocateFloatBuffer(1 * 3 * 224 * 224);
Tensor inputTensor = Tensor.fromBlob(buffer, new int[]{1, 3, 224, 224});
Tensor outputTensor = mModule.forward(IValue.from(inputTensor)).toTensor();

In this case, all inputs will be transparently copied from CPU to the
Vulkan device, and model will be run on Vulkan device, the output
will be copied transparently to CPU.

The example of using Vulkan backend can be found in test application
within the PyTorch repository: https://github.com/pytorch/pytorch/
blob/master/android/test_app/app/src/main/java/org/pytorch/testapp/
MainActivity.java#L133

Building android test app with VulkanP

cd $PYTORCH_ROOT
USE_VULKAN=1 sh ./scripts/build_pytorch_android.sh

Or if you need only specific abi you can set it as an argument:

cd $PYTORCH_ROOT
USE_VULKAN=1 sh ./scripts/build_pytorch_android.sh $ANDROID_ABI

Add prepared model mobilenet2-vulkan.pt to test applocation assets:

cp mobilenet2-vulkan.pt $PYTORCH_ROOT/android/test_app/app/src/main/assets/

cd $PYTORCH_ROOT
gradle -p android test_app:installMbvulkanLocalBaseDebug

After successful installation, the application with the name 'MBQ'
can be launched on the device.

Testing models without uploading to android deviceP

Software implementations of Vulkan (e.g. https://
swiftshader.googlesource.com/SwiftShader ) can be used to test if a
model can be run using PyTorch Vulkan Backend (e.g. check if all
model operators are supported).

---------------------------------------------------------------------
Rate this Tutorial
---------------------------------------------------------------------

(c) Copyright 2022, PyTorch.

Built with Sphinx using a theme provided by Read the Docs.

  * PyTorch Vulkan Backend User Workflow
      + Introduction
          o Building PyTorch with Vulkan backend
              # Optional usage of vulkan wrapper
              # Desktop build
              # Vulkan SDK
              # Android build
      + Model preparation
      + Using Vulkan backend in code
      + C++ API
      + Python API
      + Android Java API
      + Building android test app with Vulkan
      + Testing models without uploading to android device

* *

Docs

Access comprehensive developer documentation for PyTorch

View Docs

Tutorials

Get in-depth tutorials for beginners and advanced developers

View Tutorials

Resources

Find development resources and get your questions answered

View Resources
 

  * PyTorch
  * Get Started
  * Features
  * Ecosystem
  * Blog
  * Contributing

  * Resources
  * Tutorials
  * Docs
  * Discuss
  * Github Issues
  * Brand Guidelines

  * Stay up to date
  * Facebook
  * Twitter
  * YouTube
  * LinkedIn

  * PyTorch Podcasts
  * Spotify
  * Apple
  * Google
  * Amazon

  * Terms
  * |
  * Privacy

(c) Copyright The Linux Foundation. The PyTorch Foundation is a project
of The Linux Foundation. For web site terms of use, trademark policy
and other policies applicable to The PyTorch Foundation please see
www.linuxfoundation.org/policies/. The PyTorch Foundation supports
the PyTorch open source project, which has been established as
PyTorch Project a Series of LF Projects, LLC. For policies applicable
to the PyTorch Project a Series of LF Projects, LLC, please see
www.lfprojects.org/policies/.

To analyze traffic and optimize your experience, we serve cookies on
this site. By clicking or navigating, you agree to allow our usage of
cookies. As the current maintainers of this site, Facebook's Cookies
Policy applies. Learn more, including about available controls:
Cookies Policy.

[pytorch-x]
  

  * Get Started
  * Ecosystem
  * Mobile
  * Blog
  * Tutorials
  * Docs
      + PyTorch
      + torchaudio
      + torchtext
      + torchvision
      + torcharrow
      + TorchData
      + TorchRec
      + TorchServe
      + TorchX
      + PyTorch on XLA Devices
  * Resources
      + About
      + PyTorch Foundation
      + Community
      + Community Stories
      + Developer Resources
      + Events
      + Forums
      + Models (Beta)
  * Github