由于中文语言的复杂性，给中文分词系统带来了较大的困难，不论哪种分词系统都不能百分百的解决分词问题。针对目前中文分词存在的困难与问题，主要探讨了几种常见的中文分词算法及各自的优缺点。

Because of the complexity of Chinese language it brought the major difficulty to Chinese word segmentation system. No matter what kind of word segmentation system can not solve the problems perfectly. In view of the difficulties and problems existing in Chinese word segmentation, this paper mainly discussed several common Chinese word segmentation algorithm and the advantages and disadvantages of each.

分词和词性标注是中文语言处理的重要技术，广泛应用于语义理解、机器翻译、信息检索等领域。在搜集整理当前分词和词性标注研究与应用成果的基础上，对中文分词和词性标注的基本方法进行了分类和探讨。首先在分词方面，对基于词典的和基于统计的方法进行了详细介绍，并且列了三届分词竞赛的结果；其次在词性标注方面，分别对基于规则的方法和基于统计的方法进行了阐述；接下来介绍了中文分词和词性标注一体化模型相关方法。此外还分析了各种分词和词性标注方法的优点和不足，在此基础上，为中文分词和词性标注的进一步发展提供了建议。

Word segmentation and Part-Of-Speech ( POS) tagging are the basic task of the CLP ( Chinese Language Processing) and are widely applied in the semantic understanding,machine translation,information retrieval and other fields. In this paper,based on collecting current research and application results of word segmentation and part-of-speech tagging,analyze and classify the basic methods of Chi-nese Word Segmentation ( CWS) and POS tagging. First in terms of word segmentation,dictionary-based segmentation method and sta-tistics-based segmentation method were introduced in detail and some word segmentation results of the competition were also listed. Sec-ondly in terms of POS tagging,rule-based method and statistics-based method were expounded. Next,the main methods of building the model for joint CWS and POS tagging were presented. In this paper,also analyze the advantages and disadvantages for methods of CWS and POS tagging,based on which suggestions for the further development

词语粗分是分词后续处理的基础和前提，直接影响到分词系统最终的准确率和召回率。针对目前常用分词方法单一使用时存在的不足，综合机械分词的高效性和统计分词的灵活性，设计一种基于最短路径的二元语法中文词语粗分模型。实验结果表明，此粗分模型无论在封闭测试和开放测试中，还是在不同粗分模型对比测试和不同领域的开放测试中，都有较好的句子召回率。

The words rough segmentation is the foundation and premise of the segmentation following pro-cessing, which directly influences to the final rate of accuracy and recall of the segmentation sys-tem.In view of the limitation of present segmentation methods when they are used singly, and to synthesize efficiency of mechanical segmentations and flexibility of the statistical segmentations, designs a Chinese words rough segmentation model which is based on shortest-path and Bi-gram model.Experimental results show that the model has good effect of coarse points and sentence re-call rate, solve the most of ambiguity problems and decrease the problems of sparse data.

该文设计了一个基于复杂形式最大匹配算法(MMSeg_Complex)的自定义中文分词器,该分词器采用四种歧义消除规则,并实现了用户自定义词库、自定义同义词和停用词的功能,可方便地集成到Lucene中,从而有效地提高了Lucene的中文处理能力。通过实验测试表明,该分词器的分词性能跟Lucene自带的中文分词器相比有了极大的提高,并最终构建出了一个高效的中文全文检索系统。

This paper designed a custom Chinese word analyzer that based on a complex form of maximum matching algorithm (MMSEG_Complex). This analyzer use four kinds of disambiguation rules, and has achieved user-defined thesaurus、custom func-tion of synonyms and stop words , which can be easily integrated into Lucene, thus effectively improving the Chinese processing capabilities of Lucene. Through experiments we found that this analyzer''s performance of Chinese word segmentation has been greatly improved compared to the Chinese word analyzer which built-in Lucene, and then we can eventually build an effective Chinese full-text retrieval system.

汉语分词是中文信息处理的一项基础性工作。为避免人工阅读或机器处理时的分词歧义和未登录词难以识别的问题，有专家建议写作时在汉语词之间添加空格。文章从语言学本体研究、语言使用以及语言工程等不同角度对传统观念下的汉语分词存在的困难进行探讨，指出汉语分词在词的定义、群众语感以及分词规范、词表确定及工程应用等方面都存在不确定及不一致等因素。近年汉语自动分词处理不纠缠于词的确切定义，以字组词，针对标注语料和网络上带有丰富结构信息的海量文本，利用机器学习方法对汉语“切分单位”的标注取得了较好的进展。针对基础性的汉语分词规范，从语言规划的政策性、科学性及引导性角度提出建议，最后指出结合语言学指导和数据驱动的机器学习策略，可望为实现汉语自动分词的准确性和适应性提升服务。

Chinese word segmentation is fundamental for Chinese information processing.To a-void ambiguity and out-of-vocabulary word,there was a proposal for adding a manual space between Chinese words,which we disagree with it.This paper first elaborates difficulties in word segmenta-tion from the point of linguistic studies,language performance and language engineering,and then it discusses some uncertain factors in definition of the word,language awareness,word segmentation specification,construction of word list and its application in automatic text information processing. Not dwelling on exact definition of the word,the paper lists recent advances in character-based tag-ging with massive manually annotated recourses,which show an inspiring progress.At the end of the paper,we put forward a word segmentation guideline from a stance of language policy strategies. Guided by linguistic theory and data-driven machine learning algorithms,a practical word segmenta-tion system can achieve