摘要
本文介绍了一种汉语自动分词软件系统，该系统对原文进行三遍扫描：第一遍，利用切分标记将文本切分成汉字短串的序列；第二遍，根据各短串的每个子串在上下文中的频度计算其权值，权值大的子串视为候选词；第三遍，利用候选词集和一部常用词词典对汉字短串进行切分。实验表明，该分词系统的分词精度在１．５％左右，能够识别大部分生词。
This paper presents a software system on Chinese automatic word segmentation.The original text is scanned three times:first,the text is cut into short Chinese character string sequence by cut marks;second,every short sting is weighted by its frequency in context,and the short strings weighted heavy are regarded as candidate words;third,short strings are segmented by candidate word set and everyday words.Experiments results shows that the segmentation precision of this word segmentation system is aboue 1.5%,and a large part of new words can be recognized correctly.This system is very suitable to document retrieval and other areas.
出处
《中文信息学报》
CSCD
北大核心
1998年第1期17-25,共9页
Journal of Chinese Information Processing
关键词
中文信息处理
自动分词
汉语
串频统计
词形匹配
hinese Information Processing, Automatic Word Segmentation, Software System
