jagomart
digital resources
picture1_Types Of Lecture Method Pdf 88191 | Lecture 13


 154x       Filetype PDF       File size 0.20 MB       Source: web.stanford.edu


File: Types Of Lecture Method Pdf 88191 | Lecture 13
ee376a stats376a information theory lecture 13 02 20 2018 lecture 13 method of types lecturer tsachy weissman scribe fang cai rob jones yi sun can wang in last lecture we ...

icon picture PDF Filetype PDF | Posted on 15 Sep 2022 | 3 years ago
Partial capture of text on file.
                          EE376A/STATS376A Information Theory                                           Lecture 13 - 02/20/2018
                                                       Lecture 13: Method of Types
                          Lecturer: Tsachy Weissman                         Scribe: Fang Cai, Rob Jones, Yi Sun, Can Wang
                       In last lecture we framed the problem of lossy compression and gave the main theorem that characterizes
                   the tradeoff between the rate and the distortion in the context of lossy compression. With that as our
                   motivation, this week we are going to talk about the method of types, expand our tools related to typical
                   sequences, the notion of strong typicality and the notion of conditional types, which are not only interesting
                   in their own right, but also serve very well when we go back to establish the main result in lossy compression.
                   We are also going to talk about some concrete schemes for lossy compression and how they are related to
                   clustering and machine learning problems. Today we are going to talk about Method of Types.
                   1     Notation
                   Denote: xn = {x1,...,xn} with xi ∈ X = {1,...,r} and
                                                                                n
                                                                  N(a|xn) = XI{xi=a},
                                                                               i=i
                                                                                N(a|xn)
                                                                    P n(a) =             .
                                                                      x            n
                   2     Empirical distribution and type class
                   Definition 1 (Empirical distribution, type class). The empirical distribution of xn is the probability
                   vector (P n(1),...,P n(r)). P denotes the collection of all empirical distribution of sequences of length n,
                             x             x         n
                                  n    n      n                                                                  n     n
                   i.e. P  ={P :x ∈X }. For P∈P , the type class or type of P is T(P) = {x : P                            =P}. The type
                         n       x                            n                                                       x
                              n      n          n       n      n = P n}.
                   class of x   is T   =T(P )={x˜ :P
                                    x          x              x˜     x
                                                                      n−1 1 n−2 2                  	
                   Example 2. If X = {0,1}, then Pn = (1,0),             n , n ,     n , n ,...,(0,1)
                                                                  n                           n    3 1 1
                   Example 3. If X = {a,b,c}, n = 5 and x = (a,a,c,b,a), then P                 = , , ,
                                                                                            x     5 5 5
                                                                                            5        5!
                   T n = {(a,a,a,b,c),(a,a,a,c,b),...,(c,b,a,a,a)} and |T n| =                  =         =20.
                    x                                                             x       3 1 1    3! 1! 1!
                       In the following we show that the number of different type classes induced by xn, |Pn|, can be upper
                   bounded by something which is polynomial in n, which doesn’t increase exponentially with n.
                                                   r−1
                   Theorem 4. |Pn| ≤ (n+1)            .
                   Proof
                                                     n                                   n         n                  n                  n
                   EveryempiricaldistributionP         is determined by vector (N(1|x ),N(2|x ),··· ,N(r−1|x )), whereN(a|x )
                                                    x
                                                                                                    n                     n
                   means the number of times that the symbol a appears in the sequence x . Since 0 ≤ N(a|x ) ≤ n, each of
                          n
                   N(a|x ) can take on no more than n+1 values.
                   Thus we have a vector of length r−1 and each element can take no more than n+1 values. Therefore there
                   are at most (n+1)r−1possibilities.
                    Note that for the case r = 2, the bound is tight. But the bound is not tight for the cases r ≥ 3 because we
                                                              P
                   didn’t incorporate the constraint that        r−1 N(a|xn) must be less than or equal to n when calculating the
                                                                 a=1
                   upper bound.
                                                                              1
                   Further notation
                   - For probability mass function(PMF) Q = {Q(x)}                  , we write H(Q) for H(X) when X is distributed
                   according to Q.                                             x∈X
                   - Q(xn) = Qn Q(x ). For S ⊂ Xn, Q(S) = P n                  Q(xn)
                                  i=1     i                              x ∈S
                                       n      n      −n[H(P n)+D(P n||Q)]                                                                  n
                   Theorem 5. ∀x ,Q(x ) = 2                 x        x       , where H(P n) is referred to as empirical entropy of x .
                                                                                           x
                   Proof
                                                                     n
                                                              n     Y
                                                         Q(x ) =        Q(x )
                                                                            i
                                                                    i=1
                                                                     Pn logQ(xi)
                                                                 =2 i=1
                                                                     P           n
                                                                 =2 a∈χN(a|x )logQ(a)
                                                                        h           n         i
                                                                     −n P      N(a|x ) log  1
                                                                 =2        a∈χ    n       Q(a)
                                                                        h                           i
                                                                         P                  1  P n(a)
                                                                     −n        P n(a)log        x
                                                                 =2        a∈χ x          Q(a) Pxn(a)
                                                                     −n[H(P n)+D(P nkQ)]
                                                                 =2          x        x
                    The next result is about the size of the type class associated with the empirical distribution P.
                                                   1      nH(P)                nH(P)
                   Theorem 6. ∀P∈P ,                 r−12        ≤|T(P)| ≤ 2         .
                                            n (n+1)
                       Note: We could calculate the size of type class |T(P)| exactly, which is
                                                                                    n                
                                                        |T(P)| =     n·P(1),n·P(2),··· ,n·P(r) .
                   But for our purposes, what we care about are (1) the behavior of this quantity for n large on an exponential
                   scale and (2) how it is related to quantities that are familiar and important to us, such as entropy.
                   Proof of upper bound in Theorem 6:
                          1 ≥ P(T(P))
                                 X n
                            =           P(x )
                               xn∈T(P)
                                 X −n[H(P n)+D(P n||P)]
                            =           2        x       x                                              (by Theorem 5, with Q=P)
                               xn∈T(P)
                                 X −n[H(P)+D(P||P)]                                      n
                            =           2                               (all elements x ∈ T(P) have empirical distribution P)
                               xn∈T(P)
                            =|T(P)|·2−nH(P)
                   Thus by simple algebraic manipulation we have:
                                                                                   nH(P)
                                                                       |T(P)| ≤ 2
                   Before proving the lower bound, we prove two lemmas.
                                                                                2
                   Lemma 7. For non-negative integers m,n, m! ≥ nm−n.
                                                                      n!
                       Proof:
                       If m ≥ n,
                                                            m! =m(m−1)···(n+1)≥nm−n.
                                                            n!    |          {z         }
                                                                   (m−n) factors, each≥n
                       If m < n,
                                                      m! =              1            ≥ 1 =nm−n.
                                                       n!     n(n−1)···(m+1)            nn−m
                                                              |        {z         }
                                                              (n−m) factors, each≤n
                                                                                                                                          
                   Lemma 8. ∀P,Q∈P ,P(T(P))≥P(T(Q)).
                                             n
                       Proof:
                                                       Q            nP(a)
                                  P(T(P))      |T(P)|(    a∈X P(a)        )
                                 P(T(Q)) = |T(Q)|(Qa∈X P(a)nQ(a))
                                                         n            Y
                                            = nP(1),nP(2),···,nP(r)       P(a)nP(a)−nQ(a)
                                                          n
                                                nQ(1),nQ(2),···,nQ(r) a∈X
                                            = Y (nQ(a))!P(a)n[P(a)−Q(a)]
                                               a∈X (nP(a))!
                                            ≥ Y(nP(a))nQ(a)−nP(a)P(a)n[P(a)−Q(a)]                             (by Lemma 7)
                                               a∈X
                                               Y n[Q(a)−P(a)]
                                            =      n
                                               a∈X
                                                nP      (Q(a)−P(a))
                                            =n a∈X                  =1
                                                                                                                                          
                   Proof of lower bound in Theorem 6:
                                        1 = P(Xn)
                                          = X P(T(Q))
                                             Q∈Pn
                                          ≤|P |· maxP(T(Q))
                                               n   Q∈P
                                                       n
                                          =|P |·P(T(P))                                                (by Lemma 8)
                                               n
                                                             −n[H(P)+D(P||P)]
                                          =|P |·|T(P)|·2                                             (by Theorem 5)
                                               n
                                                     r−1            −nH(P)
                                          ≤(n+1)         · |T(P)| · 2                                (by Theorem 4)
                   Thus by simple algebraic manipulation we have:
                                                                     1         nH(P)
                                                                        r−1 · 2       ≤|T(P)|
                                                                (n+1)
                                                                               3
                    Noting that by Theorem 5, we have that, for any probability mass function Q and any empirical distri-
                 bution P ∈ Pn,                                       −n[H(P)+D(P||Q)]
                                                   Q(T(P)) = |T(P)|2                  .
                 Together with Theorem 6, we obtain the following theorem.
                                                      1    −nD(P||Q)                 −nD(P||Q)
                 Theorem 9. ∀PMF Q,∀P∈Pn,(n+1)r−12                    ≤Q(T(P))≤2              .
                    This shows that up to an insignificant polynomial factor (    1   ), on an exponential scale, the prob-
                                                                              (n+1)r−1
                 ability that the sequence looks like it came from source P, if the data is generated i.i.d. from distribution
                 Q, is exponentially unlikely. The farther away P is from Q, the more unlikely it is. Note in the expression
                 above, the relative entropy D(P||Q) is between P, the “wrong” source, and Q, the true source, unlike in the
                 cost of mismatch in lossless compression D(p||q) (see lecture 6), p is the true source while q is the “wrong”
                 source.
                                                                    4
The words contained in this file might help you see if this file matches what you are looking for:

...Eea statsa information theory lecture method of types lecturer tsachy weissman scribe fang cai rob jones yi sun can wang in last we framed the problem lossy compression and gave main theorem that characterizes tradeo between rate distortion context with as our motivation this week are going to talk about expand tools related typical sequences notion strong typicality conditional which not only interesting their own right but also serve very well when go back establish result some concrete schemes for how they clustering machine learning problems today notation denote xn x xi r n a i p empirical distribution type class denition is probability vector denotes collection all length e or t example if then pn b c following show number dierent classes induced by be upper bounded something polynomial doesn increase exponentially proof everyempiricaldistributionp determined wheren means times symbol appears sequence since each take on no more than values thus have element therefore there at mos...

no reviews yet
Please Login to review.