4. Maximum Likelihood Estimation Let X1, . . . , X... 6 Rd be 31 sample points drawn independently from a multivariate normal distribution N01, 2). (a) Suppose the normal distribution has an unknown diagonal covariance matrix 0.2 l 2 z: 0% 0.2 d and an unknown mean p. Derive the maximum likelihood estimates, denoted ii and 67, for ,u and 03-. Show all your work. (b) Suppose the normal distribution has a known covariance matrix 2 and an unknown mean An, where Z and A are known d x d matrices, E is positive denite, and A is invertible. Derive the maximum likelihood estimate, denoted ii, for n. ( ) ~ , ( | , 1 )= ( ) 2 ( , ,........, | , , ,...., , = 1,2,3, ... . , 1 ) = ( 2 = ( , 1 = 1 ln( ) = ln | , ,........, ...... + 2 ...... ( ) .... ...... exp 2 ) , 1 2 ,...., 2 2 + 1 2 ) , + 1 2 2 2 1 ln( ) 2 = ln( ) => = ln 2 ...... = 00+ = 1 1 2 + 1 => = + 2 2 = + => 1 2 = 1 2 ( ) =0 1 + 1 1 =0 =0 1 1 + = ( ( ) ) =0 ( ) => = =| 1 | ( ) ( ~ , , ) , ,1 . ( | , 1 )= ( ) 2 ( , | ,........, , , ,...., = ( 1 = ) = , 1 2 1 2 + ...... | ,........, ...... ( ) ( ) ( .... ...... exp 2 ln( ) = ln , = 1,2,3, ... . , , 1 2 + 1 2 , ,...., 2 + ) , 2 2 ( 2 ) 1 2 ( ) 1 ln( ) 2 = ln( ) => = ln = 00+ 2 ...... 1 2 + 1 2 2 = ( =0 ( ) 1 2 ) 1 ) => 2 ( 1 ) =0 2 =1 =1 => 2 1 2 =1 =0 =1 [ 0] . => 2 1 => =0 2 =1 =1 2 1 = 2 =1 =1 => = , 1 . ( ) ~ , ( | , 1 )= ( ) 2 ( , ,........, | , , ,...., , = 1,2,3, ... . , 1 ) = ( 2 = ( , 1 = 1 ln( ) = ln | , ,........, ...... + 2 ...... ( ) .... ...... exp 2 ) , 1 2 ,...., 2 2 + 1 2 ) , + 1 2 2 2 1 ln( ) 2 = ln( ) => = ln 2 ...... = 00+ = 1 1 2 + 1 => = + 2 2 = + => 1 2 = 1 2 ( ) =0 1 + 1 1 =0 =0 1 1 + = ( ( ) ) =0 ( ) => = =| 1 | ( ) ( ~ , , ) , ,1 . ( | 1 )= , ( ) 2 ( , ,........, | , , ,...., = ( 1 = 1 ) = , 2 + ...... ) ( ) .... | , 1 2 , + 1 2 ,...., 2 2 + ) , 2 2 1 2 ( 1 2 ) ( ) ln( ) = ln( ) => = ln 2 ...... = 00+ ( ...... ,........, ...... 1 ( 2 exp 2 ln( ) = ln , = 1,2,3, ... . , ( 1 2 ) + 2 2 = =0 ( 1 2 ( ) ) ) => ( 2 =1 ( = 2 ) 2 =1 2 = =1 => ) ( = =1 => =0 2 =1 => ) 2 =1 2 = =1 2 . =1 . => = , . (a) Let X i N ( , 2i ) thenthe pdf of the random variable X i is given by 1 1 f ( x i , i ) = e 2 2 2 i 2 i ( xi ) 2 Since X i are independent random variables so the Joint PDF of the random variables X i ,i=1,2,3, ... . , d is f ( x 1 , x 2 , ... ..... , x d| , 1 , 2 , ... . , d = 1 2 2 1 2 2 2 ...... d e 1 2 1 2 1 2 x ) 2 ( x2 ) ... .... 2 ( x d ) 2( 1 21 2 2 2 d Let L=f ( x1 , x2 , ... ... .. , x d| , 1 , 2 , .... , d ,Then d x 2i 2 d x i 2 d 1 1 1 L= exp + 2 2 2 2 2 2 i=1 i i=1 i i=1 i 2 21 22 ... ... 2d ( ) 2 d x i 2 d xi 2 d 1 1 1 ln ( L )=ln + + ln ( e ) 2 21 22 ... ... 2d 2 i =1 2i 2 i=1 2i 2 i=1 2i ( )( Also Let H=ln ( L ) then > H=ln ( 2 21 22 ... ... 2d ) d x 2i 2 d x i 2 d 1 1 + 2 i=1 2i 2 i =1 2i 2 i=1 2i d d d d x x H 1 1 =00+ i2 2 = 2i 2 i=1 i i=1 i i=1 i i=1 i x ( i)2 3i 2 H 1 xi 2 x i + 1 = + = + i i i 3i ) For MLE we have H 0 H = =0 i x ( i)2 =0 3i d > i=1 d > i=1 xi 2 i i=1 1 01 = + i 2i x ( i)2 3i d xi 1 = 2 1 2 i i=1 i = i d > = d x 2i i=1 d i 12 i=1 i i=|xi | Here Two unknown itwo equation s Solve them we get MLE for i Solving this is vary lengthy so you leave these two equation together for MLE of i (b) Let The multivariate normal distribution is N ( A , ) Where A is d d invertible Let A i be therow vectors of A , 1 i d . Let X i N ( A i , 2i ) then the pdf of the normal random variable X i is given by f ( x i A i , i )= 1 2 2 i e 1 2 x Ai ) 2( i 2i Since X i are independent random variables so the Joint PDF of the random variables X i ,i=1,2,3, ... . , d is f ( x 1 , x 2 , ... ..... , x d| A i , 1 , 2 , ... . , d = 1 2 2 1 2 2 2 ... ... d e 1 2 1 2 1 2 x A1 ) 2 ( x2 A 2 ) ... .... 2 ( x d A d ) 2( 1 2 1 2 2 2 d Let L=f ( x1 , x2 , ... ... .. , x d| A i , 1 , 2 , ... . , d , Then L= 2 2 d x i 2 d x i Ai 1 d ( Ai ) 1 1 exp + 2 i=1 2i 2 i=1 2i 2 i=1 2i 2 21 22 ... ... 2d ( ln ( L )=ln ( ) d x 2i 2 d x i A i 1 d ( A i )2 1 1 + ln ( e ) + 2 21 22 ... ... 2d 2 i =1 2i 2 i=1 2i 2 i =1 2i )( ) Also Let H=ln ( L ) then 2 2 d x 2 d x A 1 d ( A ) 1 > H=ln ( 2 ... ... ) i2 + i 2i i 2 2 i=1 i 2 i=1 i 2 i=1 i 2 1 xi Ai 2i xi A i 2i 2 2 2 d A A d ( i ) A i 2i ( i ) A i i=1 d 2i i=1 d = i=1 d H =00+ i=1 For MLE we have xi Ai 2i H =0 d A ( i ) A i i=1 2i d > i=1 =0 xi Ai 2i A ( i ) A i d = 2i i=1 d i =1 A xi Ai 2 i d = Ai i=1 ( i ) 2 i d > i=1 d > i=1 d > i=1 T x i A i d Ai A i = 2 2 i i i=1 x i Ai 2 i d = i=1 T Ai Ai 2 i . . d > = i=1 d i=1 xi Ai ( ) 2 i T ( ) Ai Ai 2 i Thisis required MLE for , Here A i i are known givenquestion . ( ) ~ , ( | , 1 )= ( ) 2 ( , ,........, | , , ,...., , = 1,2,3, ... . , 1 ) = ( 2 = ( , 1 = 1 ln( ) = ln | , ,........, ...... + 2 ...... ( ) .... ...... exp 2 ) , 1 2 ,...., 2 2 + 1 2 ) , + 1 2 2 2 1 ln( ) 2 = ln( ) => = ln 2 ...... = 00+ = 1 1 2 + 1 => = + 2 2 = + => 1 2 = 1 2 ( ) =0 1 + 1 1 =0 =0 1 1 + = ( ( ) ) =0 ( ) => = =| 1 | ( ) ( ~ , , ) , ,1 . ( | , 1 )= ( ) 2 ( , | ,........, , , ,...., = ( 1 = ) = , 1 2 1 2 + ...... | ,........, ...... ( ) ( ) ( .... ...... exp 2 ln( ) = ln , = 1,2,3, ... . , , 1 2 + 1 2 , ,...., 2 + ) , 2 2 ( 2 ) 1 2 ( ) 1 ln( ) 2 = ln( ) => = ln = 00+ 2 ...... 1 2 + 1 2 2 = ( =0 ( ) 1 2 ) 1 ) => 2 ( 1 ) =0 2 =1 =1 => 2 1 2 =1 =0 =1 [ 0] . => 2 1 => =0 2 =1 =1 2 1 = 2 =1 =1 => = , 1 . ( ) ~ , ( | , 1 )= ( ) 2 ( , ,........, | , , ,...., , = 1,2,3, ... . , 1 ) = ( 2 = ( , 1 = 1 ln( ) = ln | , ,........, ...... + 2 ...... ( ) .... ...... exp 2 ) , 1 2 ,...., 2 2 + 1 2 ) , + 1 2 2 2 1 ln( ) 2 = ln( ) => = ln 2 ...... = 00+ = 1 1 2 + 1 => = + 2 2 = + => 1 2 = 1 2 ( ) =0 1 + 1 1 =0 =0 1 1 + = ( ( ) ) =0 ( ) => = =| 1 | ( ) ( ~ , , ) , ,1 . ( | 1 )= , ( ) 2 ( , ,........, | , , ,...., = ( 1 = 1 ) = , 2 + ...... ) ( ) .... | , 1 2 , + 1 2 ,...., 2 2 + ) , 2 2 1 2 ( 1 2 ) ( ) ln( ) = ln( ) => = ln 2 ...... = 00+ ( ...... ,........, ...... 1 ( 2 exp 2 ln( ) = ln , = 1,2,3, ... . , ( 1 2 ) + 2 2 = =0 ( 1 2 ( ) ) ) => ( 2 =1 ( = 2 ) 2 =1 2 = =1 => ) ( = =1 => =0 2 =1 => ) 2 =1 2 = =1 2 . =1 . => = , . (a) Let X i N ( , 2i ) thenthe pdf of the random variable X i is given by 1 1 f ( x i , i ) = e 2 2 2 i 2 i ( xi ) 2 Since X i are independent random variables so the Joint PDF of the random variables X i ,i=1,2,3, ... . , d is f ( x 1 , x 2 , ... ..... , x d| , 1 , 2 , ... . , d = 1 2 2 1 2 2 2 ...... d e 1 2 1 2 1 2 x ) 2 ( x2 ) ... .... 2 ( x d ) 2( 1 21 2 2 2 d Let L=f ( x1 , x2 , ... ... .. , x d| , 1 , 2 , .... , d ,Then d x 2i 2 d x i 2 d 1 1 1 L= exp + 2 2 2 2 2 2 i=1 i i=1 i i=1 i 2 21 22 ... ... 2d ( ) 2 d x i 2 d xi 2 d 1 1 1 ln ( L )=ln + + ln ( e ) 2 21 22 ... ... 2d 2 i =1 2i 2 i=1 2i 2 i=1 2i ( )( Also Let H=ln ( L ) then > H=ln ( 2 21 22 ... ... 2d ) d x 2i 2 d x i 2 d 1 1 + 2 i=1 2i 2 i =1 2i 2 i=1 2i d d d d x x H 1 1 =00+ i2 2 = 2i 2 i=1 i i=1 i i=1 i i=1 i x ( i)2 3i 2 H 1 xi 2 x i + 1 = + = + i i i 3i ) For MLE we have H 0 H = =0 i x ( i)2 =0 3i d > i=1 d > i=1 xi 2 i i=1 1 01 = + i 2i x ( i)2 3i d xi 1 = 2 1 2 i i=1 i = i d > = d x 2i i=1 d i 12 i=1 i i=|xi | Here Two unknown itwo equation s Solve them we get MLE for i Solving this is vary lengthy so you leave these two equation together for MLE of i (b) Let The multivariate normal distribution is N ( A , ) Where A is d d invertible Let A i be therow vectors of A , 1 i d . Let X i N ( A i , 2i ) then the pdf of the normal random variable X i is given by f ( x i A i , i )= 1 2 2 i e 1 2 x Ai ) 2( i 2i Since X i are independent random variables so the Joint PDF of the random variables X i ,i=1,2,3, ... . , d is f ( x 1 , x 2 , ... ..... , x d| A i , 1 , 2 , ... . , d = 1 2 2 1 2 2 2 ... ... d e 1 2 1 2 1 2 x A1 ) 2 ( x2 A 2 ) ... .... 2 ( x d A d ) 2( 1 2 1 2 2 2 d Let L=f ( x1 , x2 , ... ... .. , x d| A i , 1 , 2 , ... . , d , Then L= 2 2 d x i 2 d x i Ai 1 d ( Ai ) 1 1 exp + 2 i=1 2i 2 i=1 2i 2 i=1 2i 2 21 22 ... ... 2d ( ln ( L )=ln ( ) d x 2i 2 d x i A i 1 d ( A i )2 1 1 + ln ( e ) + 2 21 22 ... ... 2d 2 i =1 2i 2 i=1 2i 2 i =1 2i )( ) Also Let H=ln ( L ) then 2 2 d x 2 d x A 1 d ( A ) 1 > H=ln ( 2 ... ... ) i2 + i 2i i 2 2 i=1 i 2 i=1 i 2 i=1 i 2 1 xi Ai 2i xi A i 2i 2 2 2 d A A d ( i ) A i 2i ( i ) A i i=1 d 2i i=1 d = i=1 d H =00+ i=1 For MLE we have xi Ai 2i H =0 d A ( i ) A i i=1 2i d > i=1 =0 xi Ai 2i A ( i ) A i d = 2i i=1 d i =1 A xi Ai 2 i d = Ai i=1 ( i ) 2 i d > i=1 d > i=1 d > i=1 T x i A i d Ai A i = 2 2 i i i=1 x i Ai 2 i d = i=1 T Ai Ai 2 i . . d > = i=1 d i=1 xi Ai ( ) 2 i T ( ) Ai Ai 2 i Thisis required MLE for , Here A i i are known given