https://www.infoq.com/news/2023/01/hinton-forward-algorithm/ BT Live Webinar and Q&A: Popular Patterns for Distributed Streaming Data Environments (Live Webinar February 9th, 2023) Save Your Seat Close Toggle Navigation Facilitating the Spread of Knowledge and Innovation in Professional Software Development English edition * English edition * Chinese edition * Japanese edition * French edition Write for InfoQ Search [ ] [Search] Sign Up / Login Email [ ] Password [ ] [Login] Forgot password ? InfoQ Account Email [ ] [Send Recovery Email] Back to login Resend Activation [ ] [Resend] Back to login Login with: Google Microsoft Twitter Facebook Don't have an InfoQ account? Sign Up Logo - Back to homepage 1,804,321 Dec unique visitors News Articles Presentations Podcasts Guides Topics Development * Java * Kotlin * .Net * C# * Swift * Go * Rust * JavaScript Featured in Development * The Service and the Beast: Building a Windows Service that Does Not Fail to Restart Windows Services play a key role in the Microsoft Windows operating system, and support the creation and management of long-running processes. When "Fast Startup" is enabled and the PC is started after a regular shutdown, though, services may fail to restart. The aim of this article is to create a persistent service that will always run and restart after Windows restarts, or after shutdown. The Service and the Beast: Building a Windows Service that Does Not Fail to Restart All in development Architecture & Design * Architecture * Enterprise Architecture * Scalability/Performance * Design * Case Studies * Microservices * Service Mesh * Patterns * Security Featured in Architecture & Design * API Evolution Without Versioning with Brandon Byars Everyone likes the idea of building something new. So much freedom. But what about making changes after you have users? In this episode, Thomas Betts talks with Brandon Byars about how you can evolve your API without versioning, a topic he spoke about at QCon San Francisco. API Evolution Without Versioning with Brandon Byars All in architecture-design AI, ML & Data Engineering * Big Data * Machine Learning * NoSQL * Database * Data Analytics * Streaming Featured in AI, ML & Data Engineering * The Next Decade of Software is about Climate - What is the Role of ML? Sara Bergman introduces the field of green software engineering, showing options to estimate the carbon footprint and discussing ideas on how to make Machine Learning greener. The Next Decade of Software is about Climate - What is the Role of ML? All in ai-ml-data-eng Culture & Methods * Agile * Diversity * Leadership * Lean/Kanban * Personal Growth * Scrum * Sociocracy * Software Craftmanship * Team Collaboration * Testing * UX Featured in Culture & Methods * How to Measure the Energy Consumption of Bugs Software engineers should accept their responsibility to take energy consumption and carbon dioxide emissions into account when developing software, they have a big responsibility towards nature, our environment and sustainability. This article sheds light on how software engineers can this perspective into account, zooming in on energetic shortcomings or bottlenecks of bugs. How to Measure the Energy Consumption of Bugs All in culture-methods DevOps * Infrastructure * Continuous Delivery * Automation * Containers * Cloud * Observability Featured in DevOps * Lessons Learned from Enterprise Usage of GitHub Actions GitHub Actions is an effective CI tool. However, integrating it into enterprise organizations can be challenging. This article looks at best practices for GitHub Actions in the enterprise. Lessons Learned from Enterprise Usage of GitHub Actions All in devops EventsNew Helpful links * About InfoQ * InfoQ Editors * Write for InfoQ * About C4Media * Diversity Choose your language * En * Zhong Wen * Ri Ben * Fr InfoQ Live InfoQ Live January Learn how to achieve high-level observability without picking and choosing which logs to collect. Register Now. QCon London QCon London Adopt the right emerging trends to solve your complex engineering challenges. Register Now. The Software Architects' Newsletter The Software Architects' Newsletter Your monthly guide to all the topics, technologies and techniques that every professional needs to know about. Subscribe for free. InfoQ Homepage News Deep Learning Pioneer Geoffrey Hinton Publishes New Deep Learning Algorithm AI, ML & Data Engineering Panel: 2023 Data Engineering Trends and Predictions (Live Webinar January 19, 2023) - Save Your Seat Deep Learning Pioneer Geoffrey Hinton Publishes New Deep Learning Algorithm Like Print Bookmarks Jan 10, 2023 2 min read by * Anthony Alford Write for InfoQ Join a community of experts. Increase your visibility. Grow your career.Learn more Geoffrey Hinton, professor at the University of Toronto and engineering fellow at Google Brain, recently published a paper on the Forward-Forward algorithm (FF), a technique for training neural networks that uses two forward passes of data through the network, instead of backpropagation, to update the model weights. Hinton's motivation for the algorithm is to address some of the shortcomings of standard backpropagation training which requires full knowledge of the computation in the forward pass to compute derivatives and storing activation values during training. Hinton's insight was to use two forward passes of input data---one positive and one negative---which have opposite objective functions to be optimized. Hinton showed that networks trained with FF could perform computer vision (CV) tasks about as well as those trained using backpropagation. According to Hinton, The Forward-Forward algorithm (FF) is comparable in speed to backpropagation but has the advantage that it can be used when the precise details of the forward computation are unknown. It also has the advantage that it can learn while pipelining sequential data through a neural network without ever storing the neural activities or stopping to propagate error derivatives....The two areas in which the forward-forward algorithm may be superior to backpropagation are as a model of learning in cortex and as a way of making use of very low-power analog hardware without resorting to reinforcement learning. Although artificial neural networks (ANN) are based on a mathematical model of the brain, the standard backpropagation algorithm used to train these networks is not based on any known biological process. Besides being biologically implausible, backpropagation also has some computational drawbacks as noted above. Hinton points out that ANNs can be trained using reinforcement learning (RL) without backpropagation, but this technique "scales badly...for large networks containing many millions or billions of parameters." In 2021, InfoQ covered a biologically-plausible alternative to backpropagation called zero-divergence inference learning (Z-IL) which can exactly reproduce the results of backpropagation. Hinton's FF algorithm replaces the forward-backward passes of backpropagation training with two forward passes that "operate in exactly the same way as each other." The first forward pass operates on positive data from a training set, and the network weights are adjusted to cause this input to increase a layer's goodness value. In the second forward pass, the network is given a generated negative example that is not taken from the dataset. The network weights are adjusted such that this input decreases a layer's goodness. Hinton used FF to train several neural networks to perform CV tasks on the MNIST and CIFAR datasets. The networks were relatively small, containing two or three hidden convolutional layers, and were trained for less than 100 epochs. When evaluated on test datasets, the FF-trained networks performed "only slightly worse" than those trained using backpropagation. Diego Fiori, CTO at Nebuly, implemented Hinton's FF algorithm and discussed his results on Twitter: Hinton's paper proposed 2 different Forward-Forward algorithms, which I called Base and Recurrent. Let's see why, despite the name, Base is actually the most performant algorithm....the Base FF algorithm can be much more memory efficient than the classical backprop, with up to 45% memory savings for deep networks. Fiori open-sourced his implementation of FF on GitHub. Mohammad Pezeshki, a PhD student at Universite de Montreal, also open-sourced his own implementation. About the Author Anthony Alford Show moreShow less Inspired by this content? Write for InfoQ. Becoming an editor for InfoQ was one of the best decisions of my career. It has challenged me and helped me grow in so many ways. We'd love to have more people join our team. [ccccd59] Thomas BettsLead Editor, Software Architecture and Design @InfoQ; Senior Principal Engineer Write for InfoQ Rate this Article Adoption ( ) ( ) ( ) ( ) ( ) Style ( ) ( ) ( ) ( ) ( ) [ ] [Submit] [ ] [Submit] [ ] Author Contacted This content is in the AI, ML & Data Engineering topic Related Topics: * AI, ML & Data Engineering * Neural Networks * Deep Learning * Machine Learning * Related Editorial * Related Sponsored Content + SQL to NoSQL: Architecture Differences and Considerations for Migration * Related Sponsor [ScyllaDBLo] ScyllaDB is the database for data-intensive apps requiring high performance + low latency. Achieve extreme scale with the lowest TCO. Learn More. Related Content * Researchers Publish Survey of Algorithmically-Efficient Deep Learning * Meta's CICERO AI Wins Online Diplomacy Tournament * Google Publishes Technique for AI Language Model Self-Improvement * ML.NET 2.0 Release Contains New NLP APIs and AutoML Updates * NVIDIA Kubernetes Device Plug-in Brings Temporal GPU Concurrency * AutoML: the Promise vs. Reality According to Practitioners AutoML: the Promise vs. Reality According to Practitioners * Improving Retrospective Effectiveness with End-of-Year and Focus Retrospectives * Google Kubernetes Engine Adds Multishares for Filestore Enterprise * SourceBuddy Compiles Dynamically Created Java Source Code * How to Measure the Energy Consumption of Bugs How to Measure the Energy Consumption of Bugs * Android Extension SDK Aims to Simplify the Use of Modular System Components * Traefik Hub Enables Simple and Secure Container Publishing * Lessons Learned from Enterprise Usage of GitHub Actions Lessons Learned from Enterprise Usage of GitHub Actions * Waymo Developed Collision Avoid Test to Evaluate Its Autonomous Driver * SBOM Quality and Availability Varies Greatly across Projects * Six Essential Tracks at QCon London, March 27-29, 2023: Architecture, FinTech, ML, and More! * Increasing Team and Individual Motivation with the Motivational Diagnostic Increasing Team and Individual Motivation with the Motivational Diagnostic * HashiCorp Terraform Plugin Framework Now Generally Available * AWS Introduces Step Functions Distributed Map for Large-Scale Parallel Data Processing * Just, a New CLI for Spring Boot Applications * Java News Roundup: Ideal Graph Visualizer Open-Sourced, TomEE MicroProfile 5.0 Certification * Report Finds Heavy Use of Open-Source Solutions for Kubernetes Security * PyTorch-Nightly Struck by Supply Chain Attack Exfiltrating Data and Files * Amazon ECS Adds Automated Rollbacks * The Scientific Method for Testing System Resilience The Scientific Method for Testing System Resilience * Microfrontends Anti-Patterns: Seven Years in the Trenches Microfrontends Anti-Patterns: Seven Years in the Trenches * Zero Trust Access to Corporate Applications with AWS Verified Access * Google Cloud Introduces Sensitive Actions to Improve Security for Premium Accounts * GlassFish 7.0 Delivers Support for JDK 17 and Jakarta EE 10 * Sustainability for Software Companies: Reducing Impact by Deciding What Not to Do * A Skeptic's Guide to Software Architecture Decisions A Skeptic's Guide to Software Architecture Decisions * Amazon Releases Fortuna, an Open-Source Library for ML Model Uncertainty Quantification * Generating Text Inputs for Mobile App Testing Using GPT-3 * Blue-Green Deployment from the Trenches Blue-Green Deployment from the Trenches * Driving Employee Retention and Performance through Recruiting Driving Employee Retention and Performance through Recruiting * Java News Roundup: Jakarta Data Initial Release, Micronaut 3.8, JReleaser 1.4, Gradle 8.0-RC1 * Using Code Instrumentation for Fault Injection at the Application Level at eBay * Learnings from Spotify Mobile Engineering's Recent Platform Migration * Microsoft's New Memory Optimized Ebsv5 VM Sizes in Preview Offer More Performance * AWS Announces GA of DocumentDB Elastic Clusters The InfoQ Newsletter A round-up of last week's content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example Enter your e-mail address [ ] [ ] Select your country [Select a country] [ ] I consent to InfoQ.com handling my data as explained in this Privacy Notice. [Subscribe] We protect your privacy. Hello stranger! You need to Register an InfoQ account or Login or login to post comments. But there's so much more behind being registered. Get the most out of the InfoQ experience. Tell us what you think [ ] [ ] Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p [ ] Email me replies to any of my messages in this thread [Post Message] Community comments Watch thread Close Your Reply Quote original message [ ] [ ] Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p [ ] Email me replies to any of my messages in this thread [Post Message] Cancel Close Your Reply [ ] [ ] Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p [ ] Email me replies to any of my messages in this thread [Post Message] Cancel Close OK Discuss * Development + The Service and the Beast: Building a Windows Service that Does Not Fail to Restart + OCaml 5 Brings Support for Concurrency and Shared Memory Parallelism + The InfoQ eMag - The InfoQ Trends Reports 2022 * Architecture & Design + Six Essential Tracks at QCon London, March 27-29, 2023: Architecture, FinTech, ML, and More! + API Evolution Without Versioning with Brandon Byars + The Scientific Method for Testing System Resilience * Culture & Methods + Improving Retrospective Effectiveness with End-of-Year and Focus Retrospectives + How to Measure the Energy Consumption of Bugs + Increasing Team and Individual Motivation with the Motivational Diagnostic * AI, ML & Data Engineering + Deep Learning Pioneer Geoffrey Hinton Publishes New Deep Learning Algorithm + Waymo Developed Collision Avoid Test to Evaluate Its Autonomous Driver + Amazon Releases Fortuna, an Open-Source Library for ML Model Uncertainty Quantification * DevOps + Google Kubernetes Engine Adds Multishares for Filestore Enterprise + Traefik Hub Enables Simple and Secure Container Publishing + Lessons Learned from Enterprise Usage of GitHub Actions The InfoQ Newsletter A round-up of last week's content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example * Get a quick overview of content published on a variety of innovator and early adopter technologies * Learn what you don't know that you don't know * Stay up to date with the latest information from the topics you are interested in Enter your e-mail address [ ] [ ] Select your country [Select a country] [ ] I consent to InfoQ.com handling my data as explained in this Privacy Notice. [Subscribe] We protect your privacy. QCon International Software Development Conference Real-world technical talks. No product pitches. Practical ideas to inspire you and your team. March 27-29, 2023. Attend in-person or online. QCon London brings together the world's most innovative senior software engineers across multiple domains to share their real-world implementation of emerging trends and practices. Level-up on 15 major software and leadership topics including Modern Frontend Development and Architecture, Enhancing Developer Productivity and Experience, Remote and Hybrid Work, Debugging Production, AI/ML Trends, Data Engineering Innovations, Architecture in 2025, and more. SAVE YOUR SPOT NOW Home Create account Login QCon Conferences Events Write for InfoQ InfoQ Editors About InfoQ About C4Media Media Kit InfoQ Developer Marketing Blog Diversity Events * InfoQ Live JANUARY 25, 2023 * QCon London MARCH 27-29, 2023 * QCon San Francisco OCTOBER 2-6, 2023 Follow us on Youtube223K Followers Linkedin21K Followers RSS19K Readers Twitter53.4k Followers Facebook21K Likes AlexaNew Stay in the know The InfoQ PodcastThe InfoQ Podcast Engineering Culture Podcast Engineering Culture Podcast The Software Architects' NewsletterThe Software Architects' Newsletter General Feedback feedback@infoq.com Advertising sales@infoq.com Editorial editors@infoq.com Marketing marketing@infoq.com InfoQ.com and all content copyright (c) 2006-2023 C4Media Inc. Privacy Notice, Terms And Conditions, Cookie Policy BT *