What a single sentence consist of? How to name it?

Question

I'm designing architecture of a text parser. Example sentence: Content here, content here.

Whole sentence is a... sentence, that's obvious. The, quick etc are words; , and . are punctuation marks. But what are words and punctuation marks all together in general? Are they just symbols? I simply don't know how to name what a single sentence consists of in the most reasonable abstract way (because one may write it consists of letters/vowels etc).

Thanks for any help :)

Answer 1

What you're doing is technically lexical analysis ("lexing"), which takes a sequence of input symbols and generates a series of tokens or lexemes. So word, punctuation and white-space are all tokens.

In (E)BNF terms, lexemes or tokens are synonymous with "terminal symbols". If you think of the set of parsing rules as a tree the terminal symbols are the leaves of the tree.

So what's the atom of your input? Is it a word or a sentence? If it's words (and white-space) then a sentence is more akin to a parsing rule. In fact the term "sentence" can itself be misleading. It's not uncommon to refer to the entire input sequence as a sentence.

A semi-common term for a sequence of non-white-space characters is a "textrun".

Answer 2

은 동일한 변수 이름과 2 개 이상의 작업을 사용하는 것으로 보입니다.다른 변수 이름을 사용해보십시오.

def index
  @reports = Report.page(params[:page])
end

def list
  @report_list = Report.list.page(params[:page])
end

.

변수를 간단히 변경하십시오.그것은 일해야합니다!

Answer 3

Depending on what stage of your lexical analysis of input text you are looking at, these would be either "lexemes" or "tokens."