MM: Collection continues to be a difficult word for me. Let us assume that a given MONK environment consists of a number of works. A work is something whose integrity is established by some combination of authorial and editorial decisions. 'Dubliners' and 'The Man with a Blue Guitar' are works in the sense that they were published as single books. 'The Dead' is not a work in that sense, but is part of a work. These distinctions have more to do with history or conventions than with logic. But there they are.

Some works are parts of collection that are recognized bibliographical entities of one kind or another. 'Great Expectations' enters MONK as part of NCF; 'Moby Dick' enters it as part of the Wright Archive. Some properties of the digital encoding can be inferred from the membership of a given text in a particular collection of that sort.

Some works in MONK are not part of collections in that sense. Stein's Making of Americans is not a collection, but a single work. The poems and letters of Emily Dickinson are distinct works (or are they parts of a single bibliographical entity, "The Works of Emily Dickinson?"). But they are not a collection in the sense of TCP or NCF.

We may be better off thinking in terms of a library. When I go to a library, I don't approach items as parts of a collection except in the trivial sense that all the items are part of the collection that is the library as a whole. I look for individual items in the library catalogue. Some of these items may be part of a collection (Rare Books, Music, Transportation), but that is incidental to my search.

To the extent that MONK is like a library it has a catalog, and from that catalog I choose items to work with, that is to say, a work set or "My Collection," a completely arbitrary construction of a set of items for a particular project. I think that MOA and Dickinson in the nora project are collections in that sense: project specific work sets.

My hunch is that users will have an easier with MONK if work at the bibliographical level works like it does in a library. The primary object is the bibliographical record for an individual item: Moby Dick. Whether this is part of Wright or the Library of the Americas is secondary (though it certainly should be there in the bibliographical record.

CP: Regarding to MM comment above: I think we can deal with this problem with an adequate UI. Users will be able to browse and serach both at the work level or a the work level (e.g. they can ask for a list of works written by Author X or by women) so we are OK I think... Amit made some strong point for keeping the TOC at the collection level to deal with subcollections etc.

MM: As above: a work may, but need not, be part of a collection. But you access it directly rather than through a collection
CP: this should be ok. Now that we know it could happen we will deal with it. A collection is only required to have one work and one paragraph! e.g. Stein could be alone in it's own monk collection... not a problem.

MM:
I have second thoughts about 'table of contents.' It may confuse users. A table of contents is typically found at the beginning of a book or single work. The word for the table of contents of a collection is 'catalog.' There certainly needs to be a MONK catalog, which is a list sortable by the major bibliographical criteria of author, date, genre, sex, place of origin. The bibliogrpahical collections status of an item should be a searchable criterion, but it will rarely be of primary interest to users. If I want to look for fiction written by women between 1600 and 1870, I am unlikely to care whether a given item comes from TCP, EAF, NCF, DocSouth, or Wright. Nor should I need information about the collection to identify items of a certain kind. The fact that an item is of a certain kind (novel, sermon, play, written by a woman, originating in the American South) needs to be recorded in the metadata for each work so that items can be directly retrieved from the Monk library in terms of their classifications.

Would it make sense to give to each item an LC catalogue number, which would allow us to leverage a lot of subject classification? The question arises in a slightly manner with the TCP collection. There, each record has an STC number, i.e. the catalog number of its bibliographical record in the Short Title Catalog

A table of contents is something that the user sees. Do we really envisage users seeing a representation of 'Bleak House' with its 7,000 distinct paragraphs? I find that use case very hard to imagine. That's different from saying that users should be able to identify and select a paragraph for a particular purpose (e.e.g adding it to a training set). But they are extremely unlikely to do this by choosing from a tree structure of some sort. It will be much more intuitive to read a chapter or similar section and select paragraphs or other text segments as you gao along.

It may be that 'tree structure' or 'chunk hierarchy' will be better terms for what we intend here. But these are ters for internal discussion.

—

CP: to discuss... I do like table of content better than tree structure. This also reveals that Workset also need a table of content as I WILL argue that yes you want to see the context of the text parts you are working with at least at the work level (may be not collection level): e.g. when your workset is the set of sentimental parts, then you do want to see a Table of Content of your workset.

MM Sentences are identified in the process of tokenization and sentence splitting. Their start and end points are explicitly marked, and they are identifiable objects with unique IDs just as word tokens are. They are not, however, part of the tree structure of a work, since they often cross XML element boundaries in certain forms of writing (poetry, drama).

CP: I guess this is just a clarification you give, no problem.

MM 'Unique word' is an ambiguous terms. 'Unique' in what context? Every work has a lot of words that occur only once in it. But the word may occur in other works. The ratio of lemmata that occur only once in a given work can be a useful measure of lexical density. You cannot sum the unique words of works. You can determine unique words in an author. That too may be useful and does in fact follow automatically from thhe aggregation of lexical data

CP; I was just using unique work as example, analytics has to define those metrics and publish a definition. I just know that # of unique word is useful to Tanya to get an idea of the amount of repetition... e.g. Green eggs and Ham has only 25 unique words if I remember correctly

MM If you imagine a Monk Dictionary, which is the aggregate of information about tokenization, there are kinds of information about token attributes that are context independent and can be stored at a dictionary level. The phonetic quality of a word or its reverse spelling belong. You can also encode information about the stop word status of certain words at that level

MM Lemmatization and stemming are different. See http://snowball.tartarus.org for what I think is a good reference site

MM For the distinction of main and side text you must make a distinction across all works in a Monk environment about elements whose content counts as main or side. It may be possible to vary these distinctions by sizable batches of texts from different collections in the bibliographical sense. Perhaps distinctions by genre are even more helpful. Side text in a play, for instance, can be identified with considerable precision.

Main text consists of the content of all child elements of the <body> element, except for <stage>, <speaker>, <table>, <note>.

Side text consist of the content of the <front> and <back> elements, plus the list of elements excluded from <body>.

Main text thus defined will only include words that readers will naively consider the 'real' text, but it may not include all those words. Side text is a miscellany of stuff. Whether side text by itself is a useful searchable body I don't know. I doubt it. So the point of side text is to filter out some stuff. There is no claim that the filtered-out stuff is in itself a useful thing.

Some users will find this distinction very useful, others will hate it. So you have to give users the option to ignore it. One useful feature of side text is that it is likely to filter out a lot of problematical sentences.

The <head> element is tricky in this regard. In certain genres, notably fiction, it will in 99.9% of cases consist of side text. In some TCP texts that have a question and answer structure (e.g a catechism), the question is encoded in a <head> element. There it is obviously part of the main text.

CP: as long as the ingestion process generate the data, the UI can give users the option to use or not the sidetext, may be with some level of granularity if the data is available. Should we assume it will be there is the question! (or ignore this problem for now)

MM Some discussions in the data cell suggest that this may actually be the simplest way for users to identify passages of arbitrary length for us in training corpora

MM
We may not get a clean nomenclature for features, criteria, attributes. 'Factor' may be another term that will come in. It is a well-established term in statistical analysis. For several potential analytics in MONK, sex, date, or genre will be 'factors' in the analysis.

A distinction for which we do not yet have a good vocabulary has to do with what the user knows in advance of an operation. Factors are always known in advance. Features are (partially) discovered: in text classification, I settle on some criteria in advance (lemmata, bigrams, POS tags, tag n-grams, separately or in combination) but I know nothing about the variables that make the difference.