training finder with xml documents #215
Description
I am trying to train anystyle with a set of xml documents. I do not find many information on how to do that and i have a couple of questions:
- the xml documents need to follow specific annotation guidelines?
- in the readme file you talk about xml document or folder containing TTX files. Does it mean that the training dataset if I want to use xml must consist of only 1 document with all my training data?
Thank you in advance.
Activity
inukshuk commentedon Oct 11, 2023
You can train the parser module with XML files (see the samples in res/parser). However, the finder module is trained using TTX files (see res/finder). TTX is a custom format that makes annotating long text documents relatively easy. Using XML for this would be cleaner but would require a lot of extra tooling to make it feasible (for training documents you need to label every line of a document after all).
Please note that the documents in res/finder are only one part of the sources used to train the default finder module. We can't publish the other documents due to copyrights.
olgagolgan commentedon Oct 11, 2023
thank you. So the xml must have that specific structure or i can use my own annotation?
And also, can i use input pdf documents with the parse command or should i provide txt files as input instead?
inukshuk commentedon Oct 11, 2023
The XML used for the parser must use the structure as in the sample files, i.e. one
<dataset>with any number of<sequence>records. The tag names used inside a sequence correspond to the labels known to the model, that means you can introduce your own labels there. However, I'd definitely stick with the labels of the default model (author, date, title, etc.) because the feature extraction and normalization helpers operate on those specifically. If you introduce new labels you might want to add your own normalizer code for example to process such segments.The parse command takes text input (one reference per line). But of course you can use finder and parser module in combination, for example with the CLI tool. The finder module would extract the references from a PDF or text document and pass it on to the parser module which would then segment and label each reference individually.
inukshuk commentedon Oct 11, 2023
Basically, the finder module takes entire documents; it splits the document into lines and operates on each line: every line is assigned a label; multiple lines with the same label are grouped together; reference groups are extracted; a heuristic based on regular expressions is applied to try and separate individual references.
The parser module takes one or more lines as input; each line is interpreted as a single reference; the line is split into word-tokens and each word is labeled; successive words with the same label are grouped together; normalizer routines are applied for specific labels.
olgagolgan commentedon Oct 11, 2023
thank you again.Does anystyle provide a converter for creating TTX files?
inukshuk commentedon Oct 11, 2023
Yes you can save documents as TTX. TTX is just plain text but with a certain prefix on each line; it was build for manual annotation using diff and simple text editors like nvim. You can also find more background info in some issue threads here.
olgagolgan commentedon Oct 11, 2023
So if i understood correctly:
I could train the parse model by annotating my xml files according to your model and use the trained parser as default when running the find command. Is it correct?
How can I set the trained parser as default?
inukshuk commentedon Oct 11, 2023
When using the CLI tool you can pass the model file as an argument from the command line. If you use the Ruby Gem you can set
modeloption of a given parser instance (or change the default setting).olgagolgan commentedon Oct 20, 2023
while trying to training the parser i got this error
error: undefined method 'strip' for nil:NilClassbut i cannot understand how to fix this. What could it be?
inukshuk commentedon Oct 23, 2023
I think this is probably a cryptic error message due to invalid training data. It's usually something like a blank segment (i.e., something like
<title></title>).