Answered step by step
Verified Expert Solution
Question
1 Approved Answer
We'll need the indicator function def indicator _ function ( x ) : x _ in = np . array ( x ) x _
We'll need the indicator function def indicatorfunctionx: xin nparrayx xinxin xinxin return xin # Main backward pass routine def backwardpassallweights, allbiases, allf allh y: # We'll store the derivatives dldweights and dldbiases in lists as well alldldweights NoneK alldldbiases NoneK # And we'll store the derivatives of the loss with respect to the activation and preactivations in lists alldldf NoneK alldldh NoneK # Again for convenience we'll stick with the convention that allh is the net input and allfk in the net output # Compute derivatives of the loss with respect to the network output alldldfK nparraydlossdoutputallfKy # Now work backwards through the network for layer in rangeK: # TODO Calculate the derivatives of the loss with respect to the biases at layer from alldldflayereq # NOTE! To take a copy of matrix X use ZnparrayX # REPLACE THIS LINE alldldbiaseslayer npzeroslikeallbiaseslayer # TODO Calculate the derivatives of the loss with respect to the weights at layer from alldldflayer and allhlayereq # Don't forget to use npmatmul # REPLACE THIS LINE alldldweightslayer npzeroslikeallweightslayer # TODO: calculate the derivatives of the loss with respect to the activations from weight and derivatives of next preactivations second part of last line of eq # REPLACE THIS LINE alldldhlayer npzeroslikeallhlayer if layer : # TODO Calculate the derivatives of the loss with respect to the preactivation f use derivative of ReLu function, first part of last line of eq # REPLACE THIS LINE alldldflayer npzeroslikeallflayer return alldldweights, alldldbiases alldldweights, alldldbiases backwardpassallweights, allbiases, allf allh y npsetprintoptionsprecision # Make space for derivatives computed by finite differences alldldweightsfd NoneK alldldbiasesfd NoneK # Let's test if we have the derivatives right using finite differences deltafd # Test the dervatives of the bias vectors for layer in rangeK: dldbias npzeroslikealldldbiaseslayer # For every element in the bias for row in rangeallbiaseslayershape: # Take copy of biases We'll change one element each time allbiasescopy nparrayx for x in allbiases allbiasescopylayerrow deltafd networkoutput computenetworkoutputnetinput, allweights, allbiasescopy networkoutput computenetworkoutputnetinput, allweights, allbiases dldbiasrowleastsquareslossnetworkoutput y leastsquareslossnetworkoutputydeltafd alldldbiasesfdlayer nparraydldbias print printBias d derivatives from backprop:"layer printalldldbiaseslayer printBias d derivatives from finite differences"layer printalldldbiasesfdlayer if npallclosealldldbiasesfdlayeralldldbiaseslayerrtole atole equalnanFalse: printSuccess Derivatives match." else: printFailure Derivatives different." # Test the derivatives of the weights matrices for layer in rangeK: dldweight npzeroslikealldldweightslayer # For every element in the bias for row in rangeallweightslayershape: for col in rangeallweightslayershape: # Take copy of biases We'll change one element each time allweightscopy nparrayx for x in allweights allweightscopylayerrowcol deltafd networkoutput computenetworkoutputnetinput, allweightscopy, allbiases networkoutput computenetworkoutputnetinput, allweights, allbiases dldweightrowcolleastsquareslossnetworkoutput y leastsquareslossnetworkoutputydeltafd alldldweightsfdlayer nparraydldweight print printWeight d derivatives from backprop:"layer printalldldweightslayer printWeight d derivatives from finite differences"layer printalldldweightsfdlayer if npallclosealldldweightsfdlayeralldldweightslayerrtole atole equalnanFalse: printSuccess Derivatives match." else: printFailure Derivatives different." finish the TODO. make sure the outputs match
We'll need the indicator function
def indicatorfunctionx:
xin nparrayx
xinxin
xinxin
return xin
# Main backward pass routine
def backwardpassallweights, allbiases, allf allh y:
# We'll store the derivatives dldweights and dldbiases in lists as well
alldldweights NoneK
alldldbiases NoneK
# And we'll store the derivatives of the loss with respect to the activation and preactivations in lists
alldldf NoneK
alldldh NoneK
# Again for convenience we'll stick with the convention that allh is the net input and allfk in the net output
# Compute derivatives of the loss with respect to the network output
alldldfK nparraydlossdoutputallfKy
# Now work backwards through the network
for layer in rangeK:
# TODO Calculate the derivatives of the loss with respect to the biases at layer from alldldflayereq
# NOTE! To take a copy of matrix X use ZnparrayX
# REPLACE THIS LINE
alldldbiaseslayer npzeroslikeallbiaseslayer
# TODO Calculate the derivatives of the loss with respect to the weights at layer from alldldflayer and allhlayereq
# Don't forget to use npmatmul
# REPLACE THIS LINE
alldldweightslayer npzeroslikeallweightslayer
# TODO: calculate the derivatives of the loss with respect to the activations from weight and derivatives of next preactivations second part of last line of eq
# REPLACE THIS LINE
alldldhlayer npzeroslikeallhlayer
if layer :
# TODO Calculate the derivatives of the loss with respect to the preactivation f use derivative of ReLu function, first part of last line of eq
# REPLACE THIS LINE
alldldflayer npzeroslikeallflayer
return alldldweights, alldldbiases
alldldweights, alldldbiases backwardpassallweights, allbiases, allf allh y
npsetprintoptionsprecision
# Make space for derivatives computed by finite differences
alldldweightsfd NoneK
alldldbiasesfd NoneK
# Let's test if we have the derivatives right using finite differences
deltafd
# Test the dervatives of the bias vectors
for layer in rangeK:
dldbias npzeroslikealldldbiaseslayer
# For every element in the bias
for row in rangeallbiaseslayershape:
# Take copy of biases We'll change one element each time
allbiasescopy nparrayx for x in allbiases
allbiasescopylayerrow deltafd
networkoutput computenetworkoutputnetinput, allweights, allbiasescopy
networkoutput computenetworkoutputnetinput, allweights, allbiases
dldbiasrowleastsquareslossnetworkoutput y leastsquareslossnetworkoutputydeltafd
alldldbiasesfdlayer nparraydldbias
print
printBias d derivatives from backprop:"layer
printalldldbiaseslayer
printBias d derivatives from finite differences"layer
printalldldbiasesfdlayer
if npallclosealldldbiasesfdlayeralldldbiaseslayerrtole atole equalnanFalse:
printSuccess Derivatives match."
else:
printFailure Derivatives different."
# Test the derivatives of the weights matrices
for layer in rangeK:
dldweight npzeroslikealldldweightslayer
# For every element in the bias
for row in rangeallweightslayershape:
for col in rangeallweightslayershape:
# Take copy of biases We'll change one element each time
allweightscopy nparrayx for x in allweights
allweightscopylayerrowcol deltafd
networkoutput computenetworkoutputnetinput, allweightscopy, allbiases
networkoutput computenetworkoutputnetinput, allweights, allbiases
dldweightrowcolleastsquareslossnetworkoutput y leastsquareslossnetworkoutputydeltafd
alldldweightsfdlayer nparraydldweight
print
printWeight d derivatives from backprop:"layer
printalldldweightslayer
printWeight d derivatives from finite differences"layer
printalldldweightsfdlayer
if npallclosealldldweightsfdlayeralldldweightslayerrtole atole equalnanFalse:
printSuccess Derivatives match."
else:
printFailure Derivatives different."
finish the TODO. make sure the outputs match
Step by Step Solution
There are 3 Steps involved in it
Step: 1
Get Instant Access with AI-Powered Solutions
See step-by-step solutions with expert insights and AI powered tools for academic success
Step: 2
Step: 3
Ace Your Homework with AI
Get the answers you need in no time with our AI-driven, step-by-step assistance
Get Started