Answered step by step
Verified Expert Solution
Link Copied!

Question

1 Approved Answer

Consider an optimization problem, where you want to train a neural network that is robust to small perturbations of its parameters 0 (weights and biases).

image text in transcribedimage text in transcribed
Consider an optimization problem, where you want to train a neural network that is robust to small perturbations of its parameters 0 (weights and biases). In other words, rather than minimizing your loss function L with respect to parameters 0, you want to minimize some of its "smoothened" version obtained by averaging over parameters from a local neighborhood of 0. There are many different ways of defining the "smoothened" version and one of the most popular approaches is to give smooth as (even though you are not constrained to use that smoothing in this problem !): "smooth = Eg~N(0,1) [C(0 + og)], (2) where o > 0 is a small positive hyperparameter defining the size of the local neighborhood. Propose Monte Carlo based algorithm for optimizing parameters of the neural network with respect to the smoothened metric. Assume that the loss function L is a blackbox, i.e. you can query it but you do not have its explicit form. Your algorithm does not need to be efficient in terms of time complexity.Can you think about scenarios, where such robust neural networks might be particularly useful ? Provide justification why your algorithm works. Hint: Think about most naive way of approximationadients of blackbox differentiable func- tions (see: definition of the gradient and finite difference approach) and improve that method by proving that VLsmooth (0) = Eg~N(0,D) [C(0 + og)g]. IMPORTANT: It is also fine if you propose your own smoothing mechanism and prove the analogue of formula: VCo "smooth (0) = =Eg~N(0,I) [C(0 + og)g] for or accurate approximate version of it that you can use with Monte Carlo techniques

Step by Step Solution

There are 3 Steps involved in it

Step: 1

blur-text-image

Get Instant Access to Expert-Tailored Solutions

See step-by-step solutions with expert insights and AI powered tools for academic success

Step: 2

blur-text-image

Step: 3

blur-text-image

Ace Your Homework with AI

Get the answers you need in no time with our AI-driven, step-by-step assistance

Get Started

Recommended Textbook for

Modeling the Dynamics of Life Calculus and Probability for Life Scientists

Authors: Frederick R. Adler

3rd edition

840064187, 978-1285225975, 128522597X, 978-0840064189

More Books

Students also viewed these Mathematics questions

Question

What does conveying empathy mean in delivering apologies?

Answered: 1 week ago