ch06 1 2 5 6.pdf (Peebles)

6.1 Random Process Concept

• Def. of a random process as a collection of infinitely many jointly distributed random variables

• 2 types of infinity: countable, uncountable

• 2 types of random processes

• n-th order CDF of a random process

• existence and consistency of CDFs

• Def. of a vector (-valued) random process = Def. of jointly distributed random processes

• a random field, a random polynomial, a random graph, etc.

• measure-theoretic definition of a random XX

• Kolmogorov's existence/extension theorem

• a sample path/function and an underlying probability space

• a deterministic random process

• CT time-limited RP vs. CT time-unlimited RP

• a DT RP vs. a sequence of RVs

• a random vector with variable length vs. a sequence of RVs

• Engineering/practical meaning of infinity

• Kolmogorov's theorem and its implications: a deterministic world vs. parallel universes

• statistical average vs. center of mass

• sample path/function vs. mean path/function and the 1st-order PDF/CDF

• CT/DT continuous/discrete RPs

• Definition; Existence, Uniqueness; Consequences

6.2 Stationarity and Independence

• stationary vs. non-stationary

• strict-sense stationary to order 1, 2, ..., N

• the strict-sense stationarity

• a wide-sense stationarity, a Venn diagram

• auto-correlation function

• One, two, many, ...

• Multiple random processes are independent.

• Multiple random processes are jointly stationary.

• Multiple random processes are WSS.

• vector-valued RP

• Gaussian RVectors

6.5 Gaussian RPs

6.6 Poisson (counting) RPs

• a DT/CT counting process

• counting extended to Z, R

• an arrival process as a DT process from a DT/CT counting process

• an inter-arrival (time) process as a DT process derived from an arrival process

ch06 6.pdf (Peebles)

6.6 Poisson (counting) RPs

• a DT/CT counting process

• counting extended to Z, R

• an arrival process as a DT process from a DT/CT counting process

• an inter-arrival (time) process as a DT process derived from an arrival process

• As a CT discrete RP: a counting process; a Levy process,

• As a DT RP: an arrival process, an interarrival process, a renewal process

6.6 Poisson (counting) RPs

• 2nd-order properties

6.7 Complex random variables, vector, and processes

• complex random variable

  • CDF, PDF

  • mean, variance, pseudo-variance

  • circularity, propriety

  • proper-complex Gaussian random variable

• complex random vector

  • CDF, PDF

  •  mean vector, covariance and pseudo-covariance matrices

  • propriety vs. impropriety

  • proper-complex Gaussian random vector

    • PDF

    • CDF of norm
      1) Marcum's Q function
      2) generalized Marcum's Q function
      3) cf) chi-square distribution

• complex random process

  • why?

  • complex envelope process

• complex-valued vector random process

• widely linear processing

ch06_2b.pdf

6.2 Time Average and Ergodicity

• inference and statistical inference

• detection and estimation

• random observation and ensemble average

• single sample path and ergodic assumption

• Example: Poisson process with unknown arrival rate

• time average

• energy vs. power

• time average and power

• time average and time-autocorrelation function

• time average of a random process

• MS convergence of a sequence of RVs

• Example of a stationary non-ergodic process

• ergodicity in a wide sense

• ergodicity in a narrow sense and stationarity

• nonstationary but ergodic process: arrival rate estimation for a Poisson Process

• ergodicity in XX sense

• mean ergodic in the MS sense

• Markov, Chebychev inequalities

• stationary and ergodic process: mean and autocorrelation estimation for a WSS Gaussian RP

• stationarity and ergodicity of Gaussian and Poisson RPs

6.3 Correlation Functions

6.4 Measurement of Correlation Functions

9.1 Definitions

• positive semi-definiteness of a correlation function

• alpha-dependent random process

• correlation alpha-dependent random process

• white noise in terms of correlation function

• a mean-square periodic random process

• a wide-sense cyclostationary random process

ch07-4 7-5.pdf (Papoulis)

• Equality of two real/complex numbers

• Approximate equality of two real/complex numbers and the convergence of a sequence of real/complex numbers

• Def. of a convergent sequence

• Def. of a Cauchy sequence

• Equality of two real-valued functions

  • Everywhere (e) equality of two real/complex functions

  • Almost everywhere (a.e.) equality of two real/complex functions

• 5 modes of convergence of a sequence of real numbers

  • uniform

  • everywhere

  • almost everywhere

  • in the p-th mean

  • in measure

• Equality of two real random variables

  • almost sure equality

  • equality in distribution

• 4 modes of convergence of a sequence of real random variables

  • almost surely

  • in the p-th mean

  • in probability

  • in distribution

• a Law of Large Numbers

  • almost sure convergence and a Strong Law of Large Numbers (SLLN)

  • convergence in probability and a Weak Law of Large Numbers (WLLN)

• a Central Limit Theorem

  • convergence in distribution to a standard normal random variable

• the Gaussian approximation to binomial vs. the Poisson approximation to binomial random variables

• BER of synchronous CDMA with random spreading sequences

• a limit point of a sequence of real numbers

• limsup and liminf of a sequence of real numbers

• limsup and liminf of a sequence of events

• The Borel-Cantelli lemma and the convergence a.s.

• Monte-Carlo integration and the LLN

• Summary of 4 modes of convergence of a sequence of random variables

Appendix 9A Continuity, Differentiation

Riemann vs. Lebesgue Integrals

Integration

9.2 Systems with Stochastic Inputs

9.3 Power Spectrum

A spectral characterization of a random process

ch07.pdf (Peebles)

PSD of a random process

Wiener-Khinchine Theorme

• Supplemental Video#01
• OneNote#01

note#1

quiz#01sol

Ch 1. The Meaning  of Probability

note#2

Random Experiment, Outcome, Event, Set function,

• Supplemental Video#02
• OneNote#02

note#3

Matlab homework#01

Ch 2. The Axioms of Probability

• Supplemental Video#03
• OneNote#03
• HW#1
• Study https://en.m.wikipedia.org/wiki/Set-theoretic_limit)

• Supplemental Video#04

• OneNote#04

• HW#2 (Watch Lecture Videos 01--03 of 18, 01-06 of 14)

• Supplemental Video#05
• OneNote#05
• HW#3

note#4

• Supplemental Video#06
• OneNote#06

note#5 

Ch 3. Repeated Trials

Monty Hall problem with unequal P(B_1)

note#6

• Supplemental Video#07
• OneNote#07

lecture#7  7.asx

note#7

• Supplemental Video#08
• OneNote#08

note#8

note#9

note#10

Supplementary note_pch05 

• Supplemental Video#09
• OneNote#09

note#11

note#12 

exam01sol

exam01solution

• Supplemental Video#10
• OneNote#10
• HW#1:
   

  Please solve all the problems of Quiz 1.In particular, now you justify your answers in Problem 1. 

  
   

note#13  

• Supplemental Video#11
• OneNote#11

note#14  

Matlab homework#02 

6-1 Two Random Variables and Their Joint CDF, Joint Probability Density Function, Bivariate Gaussianity,

• Circular symmetry

• Circularly symmetrical and Independent => Gaussian

• Line masses

• Supplemental Video#12
• OneNote#12a
• OneNote#12b
• HW#2

note#15  

Supplementary note_pch06

6-2 A deterministic function of two random variables:

• direct method,

• linear (dep, indep),

• nonlinear (max statistic, min statistic, etc.);

6-3 Two functions of two random variables

• direct method: Example 6-21

• Jacobi's method: derivation, cartesian coordinate -> polar coordinate

• linear TF

• auxiliary variable method, 

• Supplemental Video#13
• OneNote#13

note#16  

6-4 Joint Moments:

• Expected Value and Correlation Coefficient (Correlation Coefficient, Covariance of Two R.V.'s)

• covariance

• correlation coefficient and equality condition,

• uncorrelated, orthogonal,

• variance of sum of two random variables,

6-5 Joint CF: independence and convolution; Normal random vector,

• Supplemental Video#14
• OneNote#14
• HW#3

17_1.asx  17_2.asx

note#17  

6-6 Conditional Distribution and Density Functions

• conditional density, total density, Bayes' theorem, independence

6-7 Conditional expectation:

• conditional mean, conditional variance

• joint Gaussian case

• Definition of E{X|Y}

• total expectation theorem

(covers upto Ch. 6)

• Supplemental Video#15
• OneNote#15a
• OneNote#15b

note#18  

7-1 Random Vector

• joint pdf, cdf, CF, MGF

• Transform: direct method, Jacobi's method

• Independence, order statistics, Group Independence, conditional independence

• Expectation: mean, covariance, correlation

4/15

16

• Supplemental Video#16
• OneNote#16a
• OneNote#16b

• linear independence

  • positive semi-definiteness of Rxx, Cxx

  • Equality of two random variables

  • linear independence and positive definiteness of Rxx

7-2 Conditional Density, CF, Multivariate Gaussian R.V.'s(N-variate Gaussian R.V.'s),

• Chain rule,

• total conditional pdf, conditional expectation, total expectation,

• Gaussian random vector CF and pdf

• Goodman's theorem for a proper-complex Gaussian random vector

  • (complex random vector: complementary covariance, proper, improper)

exam02solution

7-3 Mean Square Estimation

• MMSE estimator = Conditional mean estimator

• AMMSE estimator and conditional pdf of normal vector

• LMMSE estimator and orthogonality principle

• Z vector and orthonormal data transformation (whitening)

  • square-root of a covariance matrix

  • Cholesky factorization of a covariance matrix

• Supplemental Video#17
• OneNote#17
• HW#4: Solve the Midterm Exam Problems (due on Sun 4/18, 18:00)

20_1.asx 20_2.asx

note#20

7-4 Stochastic convergence and Limit Theorems

• a random sequence

• Convergence of a sequence of real numbers, a sequence real functions

• A random sequence converges to what?

• Convergence of a sequence of real functions

• Modes of convergence of a sequence of real random variables, SLLN, WLLN, CLT

  • Almost Sure Convergence, Convergence in Probability, Convergence in Quadratic Mean, Convergence in Distribution

exam02sol

• Supplemental Video#18
• OneNote#18
• HW#5

21_1.asx 21_2.asx

note#21 

•   Proofs

  • a.s. =>p

  • in the pth mean => p

  • a.s. does not imply p

  • p does not imply a.s.

  • p => d

• Convergence in distribution revisited

  • characteristic function

  • bounded continuous function

• Supplemental Video#19
• OneNote#19
• HW#6

lecture#22_1 lecture#22_2

22_1.asx 22_2.asx

note#22

Supplementary note_pch07   

• • Law of large numbers: strong, weak

  • approximation to Binomial: Gaussian (CLT), Poisson

7-5 Random numbers: meaning and generation

• Law of Large Numbers and Monte-Carlo integration

• Generation of a pseudo-random number

• Generation of random variable and vector

Video#21 note#21

 23_2.asx

• Supplemental Video#20
• OneNote#20

Ch. 8. Statistics

(Detection and Estimation Theory)

 23_3.asx

note#23

Supplementary note_ch6

Definition and Classification of RPs

• DT vs. CT random processes
• discrete vs. continuous random processes
• 2 viewpoints on a RP

How to characterize a RP and Kolmogorov's extension/consistency theorem

• a Poisson RP

• Supplemental Video#21
• OneNote#21

30_1.asx

note Ch. 6

24.asx

note#24

• a Wiener RP

nth-order distribution

• nth-order distribution of a RP
• 2nd-order moments of a RP

Stationarity

• stationary of order n
• SSS
• WSS, covariance stationary

Correlation Functions and Properties

25.asx

note#25

• Supplemental Video#22
• OneNote#22

26_1.asx 26_2.asx

note#26  

auto-correlation and auto-covariance functions and properties

• Positive semi-definiteness
• correlation/covariace alpha/m-dependent RP
• non-trivial and trivial white noise processes
• MS periodic RP
• Gaussian/Normal RPs
• Wiener RP

30_2.asx

note Ch. 6

27.asx

note#27 

28.asx

note#28

29.asx

note#29  

Two RPs:

• Jointly distributed, equal, independent
• jointly SSS, jointly WSS
• orthogonal RPs
• uncorrelated RPs
• Joint Gaussian random processes

• Supplemental Video#23
• OneNote#23

• Supplemental Video#24
• OneNote#24

• continuity
• differentiation
• integration

• Supplemental Video#25
• OneNote#25

6/4 Note 1, 2, 3, 4, 5, 6 Video a

9.2 Systems with Stochastic Inputs

• Memoryless system with stochastic inputs
• LTV vs. LTI systems
• LTI systems with stochastic inputs
• LTI systems with WSS inputs
• LTI systems with I/O in LCCDE with WSS inputs
• LTI MIMO systems with stochastic inputs

• Supplemental Video#26
• OneNote#26

31_1.asx
31_2.asx

note Ch. 7

lecture#31_3
31_3.asx

PSD & Its Properties, Relationship b/w PSD & Autocorrelation Function, xPSD & Its Properties, Relationship b/w xPSD & Cross-Correlation Function, 

u2009e2p08 (pdf)

u2009e2p10 (pdf)

32_1.asx 32_2.asx 32_3.asx 32_4.asx

• Supplemental Video#27
• OneNote#27

• Supplemental Video#28
• OneNote#28

• Spectral Characteristics of System Response
  Bandpass Processes
  Sampling of Processes (10.5 Bandlimited Processes and Sampling Theory)
  Discrete-Time Processes (12.2 Spectrum Estimation)

• Supplemental Video#29
• OneNote#29

10.1 Random Walks

• Brownian motion process

10.3 Modulation

10.4 Cyclostationary RP

• rational spectra

11.2 finite-order systems and state variables

• differential equations

11.3 Fourier series and KL expansions

11.4 Bi-frequency spectrum

• Supplemental Video#30

• mean-ergodic processes
• covariance-ergodic processes
• distribution-ergodic processes

12.3 Extrapolation and System Identification

12.4 General class of extrapolating spectra and Youla's parameterization

(Detection and Estimation Theory)

(Introduction to Information Theory)

(covers upto Ch. 9)

exam03sol