TitleProbabilistic Chinese word segmentation with non-local information and stochastic training
AuthorsSun, Xu
Zhang, Yaozhong
Matsuzaki, Takuya
Tsuruoka, Yoshimasa
Tsujii, Jun'ichi
AffiliationPeking Univ, Key Lab Computat Linguist, Minist Educ, Beijing 100871, Peoples R China.
Peking Univ, Sch EECS, Beijing 100871, Peoples R China.
Univ Tokyo, Dept Comp Sci, Tokyo, Japan.
Natl Inst Informat, Tokyo, Japan.
Univ Tokyo, Dept EEIS, Tokyo, Japan.
Microsoft Res Asia, Beijing, Peoples R China.
KeywordsWord segmentation
Natural language processing
Conditional random fields
Latent conditional random fields
Online training
Issue Date2013
Publisherinformation processing management
CitationINFORMATION PROCESSING & MANAGEMENT.2013,49,(3),626-636.
AbstractIn this article, we focus on Chinese word segmentation by systematically incorporating non-local information based on latent variables and word-level features. Differing from previous work which captures non-local information by using semi-Markov models, we propose an alternative method for modeling non-local information: a latent variable word segmenter employing word-level features. In order to reduce computational complexity of learning non-local information, we further present an improved online training method, which can arrive the same objective optimum with a significantly accelerated training speed. We find that the proposed method can help the learning of long range dependencies and improve the segmentation quality of long words (for example, complicated named entities). Experimental results demonstrate that the proposed method is effective. With this improvement, evaluations on the data of the second SIGHAN CWS bakeoff show that our system is competitive with the state-of-the-art systems. (C) 2012 Elsevier Ltd. All rights reserved.
Appears in Collections:计算语言学教育部重点实验室

Web of Science®

Checked on Last Week


Checked on Current Time


Checked on Current Time

Google Scholar™

License: See PKU IR operational policies.