Datar Kostler 2022a - Revision history

http://www.colloquiam.com/wd/index.php?action=history&feed=atom&title=Datar_Kostler_2022a Datar Kostler 2022a - Revision history 2026-05-11T11:49:07Z Revision history for this page on the wiki MediaWiki 1.27.0-wmf.10 http://www.colloquiam.com/wd/index.php?title=Datar_Kostler_2022a&diff=262416&oldid=prev Move page script: Move page script moved page Datar Kostler 1970a to Datar Kostler 2022a 2022-11-25T15:06:07Z

<p>Move page script moved page <a href="/public/Datar_Kostler_1970a" class="mw-redirect" title="Datar Kostler 1970a">Datar Kostler 1970a</a> to <a href="/public/Datar_Kostler_2022a" title="Datar Kostler 2022a">Datar Kostler 2022a</a></p> <table class="diff diff-contentalign-left" data-mw="interface"> <tr style='vertical-align: top;' lang='en'> <td colspan='1' style="background-color: white; color:black; text-align: center;">← Older revision</td> <td colspan='1' style="background-color: white; color:black; text-align: center;">Revision as of 15:06, 25 November 2022</td> </tr><tr><td colspan='2' style='text-align: center;' lang='en'><div class="mw-diff-empty">(No difference)</div> </td></tr></table>

Move page script http://www.colloquiam.com/wd/index.php?title=Datar_Kostler_2022a&diff=261845&oldid=prev JSanchez: JSanchez moved page Draft Sanchez Pinedo 569355817 to Datar Kostler 1970a 2022-11-23T11:26:34Z

<p>JSanchez moved page <a href="/public/Draft_Sanchez_Pinedo_569355817" class="mw-redirect" title="Draft Sanchez Pinedo 569355817">Draft Sanchez Pinedo 569355817</a> to <a href="/public/Datar_Kostler_1970a" class="mw-redirect" title="Datar Kostler 1970a">Datar Kostler 1970a</a></p> <table class="diff diff-contentalign-left" data-mw="interface"> <tr style='vertical-align: top;' lang='en'> <td colspan='1' style="background-color: white; color:black; text-align: center;">← Older revision</td> <td colspan='1' style="background-color: white; color:black; text-align: center;">Revision as of 11:26, 23 November 2022</td> </tr><tr><td colspan='2' style='text-align: center;' lang='en'><div class="mw-diff-empty">(No difference)</div> </td></tr></table>

JSanchez http://www.colloquiam.com/wd/index.php?title=Datar_Kostler_2022a&diff=261844&oldid=prev JSanchez at 11:26, 23 November 2022 2022-11-23T11:26:30Z

<p></p> <table class="diff diff-contentalign-left" data-mw="interface"> <col class='diff-marker' /> <col class='diff-content' /> <col class='diff-marker' /> <col class='diff-content' /> <tr style='vertical-align: top;' lang='en'> <td colspan='2' style="background-color: white; color:black; text-align: center;">← Older revision</td> <td colspan='2' style="background-color: white; color:black; text-align: center;">Revision as of 11:26, 23 November 2022</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l6" >Line 6:</td> <td colspan="2" class="diff-lineno">Line 6:</td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>== Abstract ==</div></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>== Abstract ==</div></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div><pdf>Media:Draft_Sanchez Pinedo_569355817455_abstract.pdf</pdf></div></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div><pdf>Media:Draft_Sanchez Pinedo_569355817455_abstract.pdf</pdf></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"></ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">== Full Paper ==</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"><pdf>Media:Draft_Sanchez Pinedo_569355817455_paper.pdf</pdf></ins></div></td></tr> </table>

JSanchez http://www.colloquiam.com/wd/index.php?title=Datar_Kostler_2022a&diff=261842&oldid=prev JSanchez at 11:26, 23 November 2022 2022-11-23T11:26:28Z

<p></p> <table class="diff diff-contentalign-left" data-mw="interface"> <col class='diff-marker' /> <col class='diff-content' /> <col class='diff-marker' /> <col class='diff-content' /> <tr style='vertical-align: top;' lang='en'> <td colspan='2' style="background-color: white; color:black; text-align: center;">← Older revision</td> <td colspan='2' style="background-color: white; color:black; text-align: center;">Revision as of 11:26, 23 November 2022</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l3" >Line 3:</td> <td colspan="2" class="diff-lineno">Line 3:</td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>Replacing the traditional forward and backward passes in a residual network with a Multigrid-Reduction-in-Time (MGRIT) algorithm paves the way for exploiting parallelism across the layer dimension. In this paper, we evaluate the layer-parallel MGRIT algorithm with respect to convergence, scalability, and performance on regression problems. Specifically, we demonstrate that a few MGRIT iterations solve the systems of equations corresponding to the forward and backward passes in ResNets up to reasonable tolerances. We also demonstrate that the MGRIT algorithm breaks the scalability barrier created by the sequential propagation of data during the forward and backward passes. Moreover, we show that ResNet training using the layer-parallel algorithm significantly reduces the training time compared to the layer-serial algorithm on two non-linear regression tasks. We observe much more efficient training loss curves using layer-parallel ResNets as compared to the layer-serial ResNets on two regression tasks. We hypothesize that the error stemming from approximately solving the forward and backward pass systems using the MGRIT algorithm helps the optimization algorithm escape flat saddle-point-like plateaus or local minima on the optimization landscape. We validate this by illustrating that artificially injecting noise in a typical forward or backward propagation, allows the optimizer to escape a saddle-point-like plateau at network initialization.</div></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>Replacing the traditional forward and backward passes in a residual network with a Multigrid-Reduction-in-Time (MGRIT) algorithm paves the way for exploiting parallelism across the layer dimension. In this paper, we evaluate the layer-parallel MGRIT algorithm with respect to convergence, scalability, and performance on regression problems. Specifically, we demonstrate that a few MGRIT iterations solve the systems of equations corresponding to the forward and backward passes in ResNets up to reasonable tolerances. We also demonstrate that the MGRIT algorithm breaks the scalability barrier created by the sequential propagation of data during the forward and backward passes. Moreover, we show that ResNet training using the layer-parallel algorithm significantly reduces the training time compared to the layer-serial algorithm on two non-linear regression tasks. We observe much more efficient training loss curves using layer-parallel ResNets as compared to the layer-serial ResNets on two regression tasks. We hypothesize that the error stemming from approximately solving the forward and backward pass systems using the MGRIT algorithm helps the optimization algorithm escape flat saddle-point-like plateaus or local minima on the optimization landscape. We validate this by illustrating that artificially injecting noise in a typical forward or backward propagation, allows the optimizer to escape a saddle-point-like plateau at network initialization.</div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"></ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">== Abstract ==</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"><pdf>Media:Draft_Sanchez Pinedo_569355817455_abstract.pdf</pdf></ins></div></td></tr> </table>

JSanchez http://www.colloquiam.com/wd/index.php?title=Datar_Kostler_2022a&diff=261840&oldid=prev JSanchez at 11:26, 23 November 2022 2022-11-23T11:26:26Z

<p></p> <table class="diff diff-contentalign-left" data-mw="interface"> <col class='diff-marker' /> <col class='diff-content' /> <col class='diff-marker' /> <col class='diff-content' /> <tr style='vertical-align: top;' lang='en'> <td colspan='2' style="background-color: white; color:black; text-align: center;">← Older revision</td> <td colspan='2' style="background-color: white; color:black; text-align: center;">Revision as of 11:26, 23 November 2022</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l1" >Line 1:</td> <td colspan="2" class="diff-lineno">Line 1:</td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">                                </ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">==Summary==</ins></div></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">Replacing the traditional forward and backward passes in a residual network with a Multigrid-Reduction-in-Time (MGRIT) algorithm paves the way for exploiting parallelism across the layer dimension. In this paper, we evaluate the layer-parallel MGRIT algorithm with respect to convergence, scalability, and performance on regression problems. Specifically, we demonstrate that a few MGRIT iterations solve the systems of equations corresponding to the forward and backward passes in ResNets up to reasonable tolerances. We also demonstrate that the MGRIT algorithm breaks the scalability barrier created by the sequential propagation of data during the forward and backward passes. Moreover, we show that ResNet training using the layer-parallel algorithm significantly reduces the training time compared to the layer-serial algorithm on two non-linear regression tasks. We observe much more efficient training loss curves using layer-parallel ResNets as compared to the layer-serial ResNets on two regression tasks. We hypothesize that the error stemming from approximately solving the forward and backward pass systems using the MGRIT algorithm helps the optimization algorithm escape flat saddle-point-like plateaus or local minima on the optimization landscape. We validate this by illustrating that artificially injecting noise in a typical forward or backward propagation, allows the optimizer to escape a saddle-point-like plateau at network initialization.</ins></div></td></tr> </table>

JSanchez http://www.colloquiam.com/wd/index.php?title=Datar_Kostler_2022a&diff=261839&oldid=prev JSanchez: Created blank page 2022-11-23T11:26:25Z

<p>Created blank page</p> <p><b>New page</b></p><div></div>

JSanchez