Consider the first convolutional layer of AlexNet, which uses a 7 7 convolutional kernel, with an

Consider the first convolutional layer of AlexNet, which uses a 7 х 7 convolutional kernel, with an input feature depth of 3 and an output feature depth of 48. The original image width is 220220.

a. Ignore the 7 х 7 convolutional kernel for the moment, and consider just the center element of that kernel. A 1 х 1 convolutional kernel is mathematically equivalent to a matrix multiplication, using a weight matrix that is input_depthoutput_depth in dimensions. With these depths, and using a standard matrix multiplication, what fraction of the TPU’s 65,536 ALUs can be used?

b. For convolutional neural networks, the spatial dimensions are also sources of weight reuse, since the convolutional kernel gets applied to many different (x,y) coordinate positions. Suppose that the TPU reaches balanced compute and memory at a batch size of 1400 (as you might have computed in exercise 1d). What is the smallest square image size that the TPU can process efficiently at a batch size of 1?

c. The first convolutional layer of AlexNet implements a kernel stride of 4, which means that rather than moving by one X or Y pixel at each application the 7 х 7 kernel moves by 4 pixels at a time. This striding means that we can permute the input data from 220 х 220 х 3 to be 55 х 55 х 48 (dividing the X and Y dimensions by 4 and multiplying the input depth by 16), and simultaneously we can restack the 7 х 7 х 3 х 48 convolutional weights to be 2 х 2 х 48 х 48 (just as the input data gets restacked by 4 in X and Y, we do the same to the 7 х 7 elements of the convolutional kernel, ending up with ceiling(7/4)=2 elements in each of the X and Y dimensions). Because the kernel is now 2 х 2, we need to perform only four matrix multiplication operations, using weight matrices of size 4848.What is the fraction of the 65,536 ALUs that can be used now?