| 基于长度递减与串频统计的文本切分算法 |
| |
| 引用本文： | 姜韶华,党延忠.基于长度递减与串频统计的文本切分算法[J].情报学报,2006,25(1):74-79. |
| |
| 作者姓名： | 姜韶华 党延忠 |
| |
| 作者单位： | 大连理工大学系统工程研究所,大连,116024 |
| |
| 摘 要： | 提出了一种基于汉字串频度及串长度递减的中文文本自动切分算法。采用长串优先匹配法,不需要词典,不需要事先估计字之间的搭配概率,不需要建立字索引,利用串频信息可以自动切分出文本中有意义的汉字串。该算法能够有效地切分出文本中新涌现的通用词、专业术语及专有名词,并且能够有效避免具有包含关系的长、短汉字串中的短汉字串的错误统计。实验表明,在无需语料库学习的情况下,该算法能够快速、准确地切分出中文文档中出现频率大于等于支持度阈值的汉字串。
|
| 关 键 词： | 汉字 自动切分 串频 长串优先匹配 |
| 修稿时间： | 2005年3月23日 |
Segmentation Algorithm for Chinese Text Based on Length Descending and String Frequency Statistics |
| |
| Jiang Shaohua,Dang Yanzhong.Segmentation Algorithm for Chinese Text Based on Length Descending and String Frequency Statistics[J].Journal of the China Society for Scientific andTechnical Information,2006,25(1):74-79. |
| |
| Authors: | Jiang Shaohua Dang Yanzhong |
| |
| Abstract: | This paper puts forward a new algorithm about automatic Chinese text segmentation based on Chinese characters string frequency and length descending.It can automatically distinguish meaningful Chinese characters string in text based on processing longer string first and string frequency information without the help of dictionary,without the help of acquiring the probability between words in advance and without the help of Chinese character index.This algorithm can effectively distinguish new universal words,specialized terms and proper nouns,and it can effectively avoid statistical error about the shorter string which is belonged to a longer one.Experiment results show that this algorithm can rapidly and exactly distill words which frequency is larger than predefined value without the help of text corpus studying beforehand and dictionary. |
| |
| Keywords: | automatic segmentation string frequency matching longer string first |
| 本文献已被 CNKI 万方数据 等数据库收录！ |
|
