Post AVH6sGSaMZdexIyEYi by thserra@sigmoid.social
(DIR) More posts by thserra@sigmoid.social
(DIR) Post #AVH6HvlTpX3kLTU0iO by thserra@sigmoid.social
2023-05-03T10:18:26Z
0 likes, 0 repeats
In case you missed Calvin Tsay's tweet, we have just released a survey along with Gonzalo Munoz and Joey Huchette on polyhedral theory in deep learning: https://arxiv.org/abs/2305.00241This thread covers some of the main points, why we did this, and why you should care about it. Read along!
(DIR) Post #AVH6KgAAIHoepGL7nE by thserra@sigmoid.social
2023-05-03T10:18:57Z
0 likes, 0 repeats
First, this is a topic that we the authors have all explored in recent years, in one way or another.Polyhedral theory can help us understand what neural networks with ReLU activations can model, and also how to train them and to optimize over trained networks more efficiently. 2/N
(DIR) Post #AVH6OodmiRcUHGAOVE by thserra@sigmoid.social
2023-05-03T10:19:42Z
0 likes, 0 repeats
Second, we have learned a lot in process of writing about neural networks from the very basics. The introduction starts from the most basic models, includes a historical perspective, and argues about why we are where we are with these networks today.
(DIR) Post #AVH6TOOhiOIwKjIMQS by thserra@sigmoid.social
2023-05-03T10:20:31Z
0 likes, 0 repeats
(I recall Mike Trick saying that you do a new PhD every 5 years in academia, and indeed this is how I feel about this survey; especially because the idea was born in 2019 and developed very slowly!) 4/N
(DIR) Post #AVH6Um80yC3QDs9FLs by thserra@sigmoid.social
2023-05-03T10:20:44Z
0 likes, 0 repeats
Third, this survey allows us to go back to our own research and related scholarship and talk about them in a longer and more didactic format.For me, that means explaining the concept of linear regions and what we make out of them in much more detail than in research papers.5/N
(DIR) Post #AVH6YFWE9YAgFE87RQ by thserra@sigmoid.social
2023-05-03T10:21:22Z
0 likes, 0 repeats
We are talking about neural networks that model piecewise linear functions, and the number of these pieces can quickly grow very large. Now I had room to talk about that in a more accessible way. 6/N
(DIR) Post #AVH6be9r0AeBWwHjMW by thserra@sigmoid.social
2023-05-03T10:21:59Z
0 likes, 0 repeats
Not to mention the geometry of these pieces - or linear regions - produced by each layer of the neural network and how they affect the number of pieces that can be produced by the next layers as well.7/N
(DIR) Post #AVH6csspjpmhURuj1U by thserra@sigmoid.social
2023-05-03T10:22:15Z
0 likes, 0 repeats
Ultimately, we can think about the union of these pieces as a disjunctive program, which is indeed one of the topics from my PhD that ended up attracting me to do theoretical work in deep learning.8/N
(DIR) Post #AVH6hZLDp4o0KTQUXA by thserra@sigmoid.social
2023-05-03T10:23:05Z
0 likes, 0 repeats
Conveniently, that takes us to the next topic of the survey: optimization over a trained neural network. This has many relevant applications, including robustness against adversarial attacks (nicely illustrated with a picture of Calvin Tsay's dog)9/N
(DIR) Post #AVH6lhfymBWJKz6gQi by thserra@sigmoid.social
2023-05-03T10:23:47Z
0 likes, 0 repeats
Strengthening these MILP formulations is a topic in which both Joey Huchette and @CalvinTsay have done great contributions, which they contextualize in the survey with the current state of this topic.10/N
(DIR) Post #AVH6r4aqzuHbrWGnAm by thserra@sigmoid.social
2023-05-03T10:24:48Z
0 likes, 0 repeats
Up to this point, the survey is about ways in which discrete optimization and related theoretical tools can complement deep learning.In the last stride, Gonzalo Munoz discussed at length how we may (and sometimes should) train neural networks using discrete optimization!11/N
(DIR) Post #AVH6sGSaMZdexIyEYi by thserra@sigmoid.social
2023-05-03T10:25:01Z
0 likes, 0 repeats
This was a long and ambitious undertaking, and it is nevertheless very likely that we may have missed important work besides the 329 references we cite. Hence, your feedback to any of us is deeply appreciated!12/12