45-733 PROBABILITY AND STATISTICS I Notes #7A

February 2000

Maximum Likelihood Estimation

1. In parametric estimation we assume that we know the type of distribution (e.g, Normal, Poisson, Bernoulli, etc.) from which our random sample is drawn, and on the basis of that random sample we must infer the values of the parameters of the distribution. For example, we take a random sample from a Poisson distribution and on the basis of the random sample, we decide what the value of l is. In addition, we use the random sample to make statements about how confident we are in our guess about the values of the parameters.
2. An Estimator is a formula, or a rule, that we use to get values for the parameters. For example, we have an urn with a large number of balls in it. There are only two colors of balls -- green and red -- in the urn. We draw 10 balls with replacement and note their color. Clearly, the best guess about the proportion of green balls in the urn is the number of green balls drawn divided by 10. However, note that this is just the sample mean.
   
   Technically, an estimator is a real valued funtion of the sample.
   
   
3. Maximum Likelihood Method of Obtaining Estimators
   We need a systematic way of getting estimators. The method of maximum likelihood is a very powerful method of doing so and is strongly intuitive. It is not fool-proof, but for almost all important distributions of interest, it provides us with plausible and useful estimators of the underlying parameters.
4. The Maximum Likelihood Method has three steps. With respect to the Bernoulli distribution these are:
   
   1. First: Form the Joint distribution of the sample (the Likelihood function of the sample)
      
      f(x₁ , x₂ , ... , x_n | p) = f₁(x₁)f₂(x₂) f₃(x₃)...f_n(x_n) = P_i=1,n f_i(x_i) =
      p^x₁(1 - p)^{(1 - x₁)} p^x₂(1 - p)^{(1 - x₂)} p^x₃(1 - p)^{(1 - x₃)}... p^x_n(1 - p)^{(1 - x_n)} =
      p^{(å_i=1,n x_i)}(1 - p) ^{(n - å_i=1,n x_i)}
      
      
      
   2. Second: Take the natural log of the Likelihood function
      
      L(x₁ , x₂ , ... , x_n | p) = ln{f(x₁ , x₂ , ... , x_n | p)} =
      [å_i=1,n x_i]ln(p) + (n - å_i=1,n x_i)ln(1 - p)
      
      
      
   3. Third: Find the maximum by taking the first derivative of the log of the Likelihood function
      
      ¶L/ ¶p = [å_i=1,n x_i]/p - (n - å_i=1,n x_i)/(1 - p) = 0
      = å_i=1,n x_i - p[å_i=1,n x_i] - np + p[å_i=1,n x_i] = å_i=1,n x_i - np
      

                 ^   _
         Hence:  p = Xn
      
      

      
      
5. Example: Find the Maximum Likelihood Estimator for l in the Poisson Distribution.
   
   f(x₁ , x₂ , ... , x_n | l) = [(e^-ll^x₁)/x₁!] [(e^-ll^x₂)/x₂!] [(e^-ll^x₃)/x₃!]... [(e^-ll^x_n)/x_n!] =
   [(e^-nll ^{å_i=1,n x_i})/(P_i=1,n x_i!)]
   L(x₁ , x₂ , ... , x_n | l) = -nl + (å_i=1,n x_i) ln(l) - ln(P_i=1,n x_i!)
   ¶L/ ¶l = -n + (å_i=1,n x_i)/l = 0
   

              ^   _
      Hence:  l = Xn
   
   

   
   
6. Example: Find the Maximum Likelihood Estimator for m and s² in the Normal Distribution.
   
   f(x₁ , x₂ , ... , x_n | m, s²) = {1/[(2p)^1/2 s]} [e^{-(x₁ - m)2/ 2(s)2}]
   {1/[(2p)^1/2 s]} [e^{-(x₂ - m)2/ 2(s)2}] {1/[(2p)^1/2 s]} [e^{-(x₃ - m)2/ 2(s)2}]...
   {1/[(2p)^1/2 s]} [e^{-(x_n - m)2/ 2(s)2}] =
   {1/[(2p)^n/2 (s²)^n/2]} {e^{-[1/2(s)2] [å_i=1,n (x_i - m)2]}}
   
   L(x₁ , x₂ , ... , x_n | m, s²) = - (n/2)ln(2 p) - (n/2)ln (s²) - [1/2(s)²] [å_i=1,n (x_i - m)²]
   
   ¶L/ ¶ m = [1/(s²)] [å_i=1,n (x_i - m)] = 0
   ¶L/ ¶ s² = -[n/2(s²)] + [1/2(s⁴)] [å_i=1,n (x_i - m)²] = 0
   

   Hence: ^   _  
          m = Xn, Using ¶L/¶m
   and:   ^                 _  
          s2 = [åi=1,n (xi - Xn)2]/n