What a single sentence consist of? How to name it?
-
23-09-2019 - |
문제
I'm designing architecture of a text parser. Example sentence: Content here, content here.
Whole sentence is a... sentence, that's obvious. The
, quick
etc are words; ,
and .
are punctuation marks. But what are words and punctuation marks all together in general? Are they just symbols? I simply don't know how to name what a single sentence consists of in the most reasonable abstract way (because one may write it consists of letters/vowels etc).
Thanks for any help :)
해결책
What you're doing is technically lexical analysis ("lexing"), which takes a sequence of input symbols and generates a series of tokens or lexemes. So word, punctuation and white-space are all tokens.
In (E)BNF terms, lexemes or tokens are synonymous with "terminal symbols". If you think of the set of parsing rules as a tree the terminal symbols are the leaves of the tree.
So what's the atom of your input? Is it a word or a sentence? If it's words (and white-space) then a sentence is more akin to a parsing rule. In fact the term "sentence" can itself be misleading. It's not uncommon to refer to the entire input sequence as a sentence.
A semi-common term for a sequence of non-white-space characters is a "textrun".
다른 팁
은 동일한 변수 이름과 2 개 이상의 작업을 사용하는 것으로 보입니다.다른 변수 이름을 사용해보십시오.
def index
@reports = Report.page(params[:page])
end
def list
@report_list = Report.list.page(params[:page])
end
.
변수를 간단히 변경하십시오.그것은 일해야합니다!
Depending on what stage of your lexical analysis of input text you are looking at, these would be either "lexemes" or "tokens."