近似Bayes计算前沿研究进展及应用

朱万闯; 季春霖; 邓柯

doi:10.21656/1000-0887.400245

近似Bayes计算前沿研究进展及应用

doi: 10.21656/1000-0887.400245

¹清华大学工业工程系统计学研究中心, 北京 100084;²光启高等理工研究院, 广东深圳 518000

基金项目: 国家自然科学基金（11771242）

详细信息

作者简介:
朱万闯(1988—)，男，博士(E-mail: zwchuang@tsinghua.edu.cn)；季春霖(1981—)，男，正高级工程师，博士(E-mail: chunlin.ji@kuang_chi.org)

中图分类号: O357.41
计量
- 文章访问数: 1445
- HTML全文浏览量: 208
- PDF下载量: 413
- 被引次数: 0
出版历程
- 收稿日期: 2019-08-26
- 修回日期: 2019-09-01
- 刊出日期: 2019-11-01

Recent Progress of Approximate Bayesian Computation and Its Applications

¹Center for Statistical Science, Department of Industrial Engineering, Tsinghua University, Beijing 100084, P.R.China;²KuangChi Institute of Advanced Technology, Shenzhen, Guangdong 518000, P.R.China

Funds: The National Natural Science Foundation of China（11771242）

摘要

摘要: 在大数据和人工智能时代，建立能够有效处理复杂数据的模型和算法，以从数据中获取有用的信息和知识是应用数学、统计学和计算机科学面临的共同难题.为复杂数据建立生成模型并依据这些模型进行分析和推断是解决上述难题的一种有效手段.从一种宏观的视角来看，无论是应用数学中常用的微分方程和动力系统，或是统计学中表现为概率分布的统计模型，还是机器学习领域兴起的生成对抗网络和变分自编码器，都可以看作是一种广义的生成模型.随着所处理的数据规模越来越大，结构越来越复杂，在实际问题中所需要的生成模型也变得也越来越复杂，对这些生成模型的数学结构进行精确地解析刻画变得越来越困难.如何对没有精确解析形式（或其解析形式的精确计算非常困难）的生成模型进行有效的分析和推断，逐渐成为一个十分重要的问题.起源于Bayes统计推断，近似Bayes计算是一种可以免于计算似然函数的统计推断技术，近年来在复杂统计模型和生成模型的分析和推断中发挥了重要作用.该文从经典的近似Bayes计算方法出发，对近似Bayes计算方法的前沿研究进展进行了系统的综述，并对近似Bayes计算方法在复杂数据处理中的应用前景及其和前沿人工智能方法的深刻联系进行了分析和讨论.
- 近似Bayes计算 /
- 生成模型 /
- 深度学习 /
- 不确定性推断
Abstract: In the era of big data and artificial intelligence, it is a common challenge for applied mathematics, statistics and computer science to extract valuable information and knowledge from complex data and models. Generative models are a class of powerful models which can potentially handle the above difficulty. From a macro point of view, the differential equations and dynamic systems in applied mathematics, the probability distribution in statistical models, and the typical generative models (generative adversarial networks and variational auto-encoders) in computer science could be considered as generalized generative models. Along with larger and larger-size data, the structure of data becomes more and more complicated simultaneously. Therefore, more powerful generative models are essential to process real problems. It is a challenge to describe mathematical structures of these generative models. It poses a natural question of how to analyze such generative models without analytic forms (or hard to obtain their analytic forms). Originated from the Bayesian inference, the approximate Bayesian computation, as a likelihood-free technique, plays an important role in processing complex statistical models and generative models. Based on the classic approximate Bayesian computation, the development and recent advance of approximate Bayesian computation were systematically reviewed. Finally, the application of the approximate Bayesian computation to complex data and the deep connection between the approximate Bayesian computation and cutting-edge artificial intelligence methods were discussed.
- approximate Bayesian computation /
- generative model /
- deep learning /
- uncertainty inference

HTML全文

参考文献(77)

[1]	STRANG G. Introduction to Applied Mathematics [M]. Wellesley, MA： Wellesley-Cambridge Press， 1986.
[2]	谷超豪, 李大潜, 陈恕行. 数学物理方法[M]. 上海: 上海科学技术出版社, 2002.(GU Chaohao, LI Daqian, CHEN Shuxing. Mathematical and Physical Methods [M]. Shanghai: Shanghai Scientific & Technical Publishers, 2002.(in Chinese))
[3]	BOARD A. Stochastic Modelling and Applied Probability [M]. Springer, 2005.
[4]	COURANT R, HILBERT D. Methods of Mathematical Physics: Partial Differential Equations [M]. John Wiley & Sons, 2008.
[5]	ARNOL’D V I. Mathematical Methods of Classical Mechanics [M]. Springer Science & Business Media, 2013.
[6]	NETER J, KUTNER M H, NACHTSHEIM C J, et al. Applied Linear Statistical Models [M]. Chicago, 1996.
[7]	CONGDON P. Bayesian Statistical Modelling [M]. John Wiley & Sons, 2007.
[8]	FAHRMEIR L, TUTZ G. Multivariate Statistical Modelling Based on Generalized Linearmodels [M]. Springer Science & Business Media, 2013.
[9]	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]// Advances in Neural Information Processing Systems 〖STBX〗27 (NIPS 2014).Montreal, Canada, 2014.
[10]	KINGMA D P, WELLING M. Auto-encoding variational Bayes[R/OL]. [2019-08-26]. https://arxiv.org/pdf/1511.06434.pdf.
[11]	RADFORD A, METZ L, CHINTALA S. Unsupervised representation learning with deep convolutional generative adversarial networks[R/OL]. [2019-08-26]. https://arxiv.org/pdf/1312.6114.pdf.
[12]	CHEN X, DUAN Y, HOUTHOOFT R, et al. Infogan: interpretable representation learning by information maximizing generative adversarial nets[R/OL]. [2019-08-26]. https://arxiv.org/pdf/1606.03657.pdf.
[13]	ZHANG Han, XU Tao, LI Hongsheng, et al. Stackgan: text to photo-realistic image synthesis with stacked generative adversarial networks[R/OL]. [2019-08-26]. https://arxiv.org/pdf/1612.03242v1.pdf.
[14]	MAO X D, LI Q, XIE H R, et al. Least squares generative adversarial networks[C]//2017 IEEE International Conference on Computer Vision (ICCV).2017.
[15]	ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-image translation with conditional adversarial networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR).2017: 1125-1134.
[16]	ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein GAN[R/OL]. [2019-08-26]. https://arxiv.org/pdf/1701.07875.pdf.
[17]	ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//2017 IEEE International Conference on Computer Vision (ICCV).Venice, Italy, 2017.
[18]	KARRAS T, LAINE S, AILA T. A style-based generator architecture for generative adversarial networks[C]//2019 IEEE Conference on Computer Vision and Pattern Recognition(CVPR).2019: 4401-4410.
[19]	TAO C Y, CHEN L Q, HENAO R, et al. Chi-square generative adversarial network[C]// International Conference on Machine Learning.2018: 4894-4903.
[20]	SNDERBY C K, RAIKO T, MAALE L, et al. Ladder variational autoencoders[C]// Advances in Neural Information Processing Systems 29 (NIPS 2016).2016: 3738-3746.
[21]	HIGGINS I, MATTHEY L, GLOROT X, et al. Early visual concept learning with unsupervised deep learning[R/OL]. [2019-08-26]. https://arxiv.org/pdf/1606.05579.pdf.
[22]	RUBIN D B. Bayesianly justifiable and relevant frequency calculations for the applies statistician[J]. The Annals of Statistics,1984,12(4): 1151-1172.
[23]	PRITCHARD J K, SEIELSTAD M T, PEREZ-LEZAUN A, et al. Population growth of human Y chromosomes: a study of Y chromosome microsatellites[J]. Molecular Biology & Evolution,1999,16(12): 1791-1798.
[24]	WILKINSON R D,TAVARB S. Estimating primate divergence times by using conditioned birth-and-death processes[J]. Theoretical Population Biology,2009,75(4): 278-285.
[25]	PETERS G W, SISSON S A, FAN Y. Likelihood-free Bayesian inference for Alpha-stable models[J]. Computational Statistics & Data Analysis,2012,56(11): 3743-3756.
[26]	NOTT D J, FAN Y, MARSHALL L, et al. Approximate Bayesian computation and Bayes’ linear analysis: toward high-dimensional ABC[J]. Journal of Computational and Graphical Statistics,2014,23(1): 65-86.
[27]	RATMANN O, ANDRIEU C, WIUF C,et al. Model criticism based on likelihood-free inference, with an application to protein network evolution[J]. Proceedings of the National Academy of Sciences of the United States of America,2009,106(26): 10576-10581.
[28]	KULKARNI T, YILDIRIM I, KOHLI P,et al. Deep generative vision as approximate Bayesian computation[C]// NIPS 2014 ABC Workshop.2014.
[29]	SHEEHAN S, SONG Y S. Deep learning for population genetic inference[J]. PLoS Computational Biology,2016,12(3): e1004845. DOI: 10.1371/journal.pcbi.1004845.
[30]	GAL Y, GHAHRAMANI Z B. Dropout as a Bayesian approximation: representing model uncertainty in deep learning[C]// Proceedings of the 〖STBX〗33rd International Conference on Machine Learning.New York, USA, 2016: 1050-1059.
[31]	FELIP J, AHUJA N, GOMEZ-GUTIERREZ D, et al. Real-time approximate Bayesian computation for scene understanding[R/OL]. [2019-08-26]. https://arxiv.org/pdf/1905.13307.pdf.
[32]	GHOSH J K, RAMAMOORTHI R V. Bayesian Nonparametrics [M]. New York: Springer, 2003.
[33]	ROBERT C. The Bayesian Choice: From Decision-Theoretic Foundations to Computational Implementation [M]. Springer Science, 2007.
[34]	BERNARDO J M, SMITH A F M. Bayesian Theory [M]. John Wiley & Sons, 2009.
[35]	GELMAN A, CARLIN J B, STERN H S, et al. Bayesian Data Analysis [M]. 3rd ed. Chapman and Hall/CRC, 2013.
[36]	LIU J S. Monte Carlo Strategies in Scientific Computing [M]. New York: Springer, 2001.
[37]	BROOKS S, GELMAN A, JONES G, et al. Handbook of Markov Chain Monte Carlo [M]. New York: CRC press, 2011.
[38]	CHEN M H, SHAO Q M, IBRAHIM J G. Monte Carlo Methods in Bayesian Computation [M]. Springer Science & Business Media, 2012.
[39]	GIUDICI P, GIVENS G H, MALLICK B K. Wiley Series in Computational Statistics [M]. Wiley Online Library, 2013.
[40]	MARJORAM P, MOLITOR J, PLAGNOL V, et al. Markov chain Monte Carlo without likelihoods[J]. Proceedings of the National Academy of Sciences of the United States of America,2003,100(26): 15324-15328.
[41]	SISSON S A, FAN Y, TANAKA M M. Sequential monte carlo without likelihoods[J]. Proceedings of the National Academy of Sciences of the United States of America,2007,104(6): 1760-1765.
[42]	FEARNHEAD P, PRANGLE D.Constructing summary statistics for approximate Bayesian computation: semi-automatic approximate Bayesian computation[J]. Journal of the Royal Statistical Society: Series B (Statistical Methodology),2012,74(3): 419-474.
[43]	DELMORAL P, DOUCET A, JASRA A. An adaptive sequential Monte Carlo method for approximate Bayesian computation[J]. Statistics and Computing,2012,22(5): 1009-1020.
[44]	MENGERSEN K L, PUDLO P, ROBERT C P. Bayesian computation via empirical likelihood[J]. Proceedings of the National Academy of Sciences of the United States of America,2013,110(4): 1321-1326.
[45]	SISSON S A, FAN Y, BEAUMONT M. Handbook of Approximate Bayesian Computation [M]. Chapman and Hall/CRC, 2018.
[46]	FRAZIER D T, MARTIN G M, ROBERT C P. On consistency of approximate Bayesian computation[R/OL]. [2019-08-26]. https://arxiv.org/pdf/1508.05178.pdf.
[47]	HEIN J, SCHIERUP M, WIUF C. Gene Genealogies, Variation and Evolution: a Primer in Coalescent Theory[M]. Oxford: Oxford University Press, 2004.
[48]	TANAKA M M, FRANCIS A R, LUCIANI F, et al. Using approximate Bayesian computation to estimate tuberculosis transmission parameters from genotype data[J].Genetics,2006,173(3): 1511-1520.
[49]	WOOD S N. Statistical inference for noisy nonlinear ecological dynamic systems[J]. Nature,2010,466(7310): 1102-1104.
[50]	ZHU W C, FAN Y. A novel approach for Markov random field with intractable normalizing constant on large lattices[J]. Journal of Computational and Graphical Statistics,2018,27(1): 59-70.
[51]	PRANGLE D. Summary Statistics [M]. Chapman and Hall/CRC, 2018.
[52]	JOYCE P, MARJORAM P. Approximately sufficient statistics and Bayesian computation[J]. Statistical Applications in Genetics and Molecular Biology,2008,7(1). DOI: 10.2202/1544-6115.1389.
[53]	HASTIE T, TIBSHIRANI R, FRIEDMAN J. The Elements of Statistical Learning: Data Mining, Inference, and Prediction [M]. 2nd ed. New York: Springer, 2009.
[54]	MARDIA K V, KENT J T, BIBBY J M. Multivariate analysis[M]//Probability and Mathematical Statistics.New York: Academic Press, 1979.
[55]	JIANG B, WU T Y, ZHENG C, et al. Learning summary statistic for approximate Bayesian computation via deep neural network[J]. Statistica Sinica,2017,27(4): 1595-1618.
[56]	CREEL M. Neural nets for indirect inference[J]. Econometrics and Statistics,2017,2: 36-49.
[57]	RAYNAL L, MARIN J M, PUDLO P, et al. ABC random forests for Bayesian parameter inference[J]. Bioinformatics,2018,〖STHZ〗 35(10): 1720-1728.
[58]	BEAUMONT M A. Approximate Bayesian computation[J]. Annual Review of Statistics and Its Application,2019,6(1): 379-403.
[59]	Gunning P W, Ghoshdastider U, Whitaker S, Popp D, Robinson R C. The evolution of compositionally and functionally distinct actin filaments[J].Journal of Cell Science,2015,128(11): 2009-2019.
[60]	BERNTON E, JACOB P E, GERBER M, et al. Approximate Bayesian computation with the wasserstein distance[J]. Journal of the Royal Statistical Society: Series B (Statistical Methodology),2019,81(2): 235-269.
[61]	PARK M, JITKRITTUM W, SEJDINOVIC D. K2-ABC: approximate Bayesian computation with kernel embeddings[C]// Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (AISTATS).Cadiz, Spain, 2016.
[62]	SISSON S A, FAN Y, TANAKA M M. Sequential monte carlo without likelihoods: errata[J]. Proceedings of the National Academy of Sciences of the United States of America,2009,106(39): 1760-1765.
[63]	BEAUMONT M A, CORNUET J M, MARIN J M, et al. Adaptive approximate Bayesian computation[J]. Biometrika,2009,96(4): 983-990.
[64]	TONI T, WELCH D, STRELKOWA N, et al. Approximate Bayesian computation scheme for parameter inference and model selection in dynamical systems[J]. Journal of the Royal Society Interface,2009,〖STHZ〗 6(31): 187-202.
[65]	DUANE S, KENNEDY A D, PENDLETON B J, et al. Hybrid Monte Carlo[J]. Physics letters B,1987,195(2): 216-222.
[66]	MEEDS E, LEENDERS R, WELLING M. Hamiltonian ABC[R/OL]. [2019-08-26]. https://arxiv.org/pdf/1503.01916.pdf.
[67]	GIANNOPOULOS P, GODSILL S J. Estimation of car processes observed in noise using Bayesian inference[C]//2001 IEEE International Conference on Acoustics, Speech, and Signal Processing.Salt Lake City, UT, USA, 2001.
[68]	JI C L, YANG L G, ZHU W C, et al. On Bayesian inference for continuous-time autoregressive models without likelihood[C]//2018 21st International Conference on Information Fusion (FUSION).2018: 2137-2142.
[69]	HARVEY A C. Forecasting, Structural Time Series Models and the Kalman Filter [M]. Cambridge: Cambridge University Press, 1991.
[70]	JONES R H. Fitting a continuous time autoregression to discrete data[C]// Proceedings of the Second Applied Time Series Symposium Held.Tulsa, Oklahoma, 1980.
[71]	SZKELY G J, RIZZO M L, BAKIROV N K. Measuring and testing dependence by correlation of distances[J]. The Annals of Statistics,2007,35(6): 2769-2794.
[72]	LI C L, CHANG W C, CHENG Y, et al. MMD GAN: towards deeper understanding of moment matching network[C]// Advances in Neural Information Processing Systems 30 (NIPS 2017).2017.
[73]	HOFFMAN M D, BLEI D M, WANGC, et al. Stochastic variational inference[J]. Journal of Machine Learning Research,2013,14(1): 1303-1347.
[74]	ZHAO S J, SONG J M, ERMON S. Infovae: information maximizing variational autoencoders[R/OL]. [2019-08-26]. https://arxiv.org/pdf/1706.02262.pdf.
[75]	MAKHZANI A, SHLENS J, JAITLY N, et al. Adversarial autoencoders[R/OL]. [2019-08-26]. https://arxiv.org/pdf/1511.05644.pdf.
[76]	MORENO A, ADEL T, MEEDS E, et al. Automatic variational ABC[R/OL]. [2019-08-26]. https://arxiv.org/pdf/1606.08549.pdf.
[77]	王小娥, 蔺小林, 李建全. 一类具有脉冲免疫治疗的HIV-1感染模型的动力学分析[J]. 应用数学和力学, 2019,40(7): 728-740.(WANG Xiaoe, LIN Xiaolin, LI Jianquan. Dynamic analysis of a class of HIV-1 infection models with pulsed immunotherapy[J]. Applied Mathematics and Mechanics,2019,40(7): 728-740.(in Chinese))