Mark entity spans in a text by highlighting them and selecting the respective labels. The model is used to tokenize the text to allow less sensitive highlighting, since the token boundaries are used to set the entity spans. The label set can be defined as a comma-separated list on the command line or as a path to a text file with one label per line. If no labels are specified, Prodigy will check if labels are present in the model. This recipe does not require an entity recognizer, and doesn’t do any active learning.

prodigy
ner.manual
dataset
spacy_model
source
--loader
--label
--patterns
--exclude
--highlight-chars

Example

prodigy
ner.manual
ner_news
en_core_web_sm
./news_headlines.jsonl
--label PERSON,ORG,PRODUCT
prodigy ner.manual ner_news en_core_web_sm ./news_headlines.jsonl --label PERSON,ORG,PRODUCT

Create gold-standard data for NER by correcting the model’s suggestions. The spaCy model will be used to predict entities contained in the text, which the annotator can remove and correct if necessary.

prodigy
ner.correct
dataset
spacy_model
source
--loader
--label
--exclude
--unsegmented

Example

prodigy
ner.correct
gold_ner
en_core_web_sm
./news_headlines.jsonl
--label PERSON,ORG
prodigy ner.correct gold_ner en_core_web_sm ./news_headlines.jsonl --label PERSON,ORG

Collect the best possible training data for a named entity recognition model with the model in the loop. Based on your annotations, Prodigy will decide which questions to ask next.

prodigy
ner.teach
dataset
spacy_model
source
--loader
--label
--patterns
--exclude
--unsegmented

Example

prodigy
ner.teach
ner_news
en_core_web_sm
./news_headlines.jsonl
--label PERSON,EVENT
prodigy ner.teach ner_news en_core_web_sm ./news_headlines.jsonl --label PERSON,EVENT

Take existing “silver” datasets with binary accept/reject annotations, merge the annotations to find the best possible analysis given the constraints defined in the annotations, and manually edit it to create a perfect and complete “gold” dataset.

prodigy
ner.silver-to-gold
dataset
silver_sets
spacy_model
--label

Load two models and a stream of text, compare their predictions and select which result you prefer. The outputs will be randomized, so you won’t know which model is which. When you stop the server, the results are calculated. This recipe is especially helpful if you’re updating an existing model or if you’re trying out a new strategy on the same problem. Even if two models achieve similar accuracy, one of them can still be subjectively “better”, so this recipe lets you analyze that.

prodigy
ner.eval-ab
dataset
model_a
model_b
source
--loader
--label
--exclude
--unsegmented

Example

prodigy
ner.eval-ab
eval_dataset
en_core_web_sm
./improved_ner_model
./news_headlines.jsonl
prodigy ner.eval-ab eval_dataset en_core_web_sm ./improved_ner_model ./news_headlines.jsonl

Manually annotate categories that apply to a text. If only one label is set, the classification interface is used. If more than one label is specified, the choice interface is used and categories are added as multiple choice options. If the --exclusive flag is set, categories become mutually exclusive, meaning that only one can be selected during annotation.

prodigy
textcat.manual
dataset
source
--loader
--label
--exclusive
--exclude

Example

prodigy
textcat.manual
news_topics
./news_headlines.jsonl
--label Technology,Politics,Economy,Entertainment
prodigy textcat.manual news_topics ./news_headlines.jsonl --label Technology,Politics,Economy,Entertainment

Collect the best possible training data for a text named entity recognition model with the model in the loop. Based on your annotations, Prodigy will decide which questions to ask next. All annotations will be stored in the database. If a patterns file is supplied via the --patterns argument, the matches will be included in the stream and the matched spans are highlighted, so you’re able to tell which words or phrases the selection was based on. Note that the exact pattern matches have no influence when updating the model – they’re only used to help pre-select examples for annotation.

prodigy
textcat.teach
dataset
spacy_model
source
--loader
--label
--patterns
--long-text
--init-tok2vec
--exclude
--unsegmented

Example

prodigy
textcat.teach
news_topics
en_core_web_sm
./news_headlines.jsonl
--label Technology,Politics,Economy,Entertainment
prodigy textcat.teach news_topics en_core_web_sm ./news_headlines.jsonl --label Technology,Politics,Economy,Entertainment

Create gold-standard data for part-of-speech tagging by correcting the model’s suggestions. The spaCy model will be used to predict part-of-speech tags, which the annotator can remove and correct if necessary. It’s often more efficient to focus on a few labels at a time, instead of annotating all labels jointly. The --fine-grained flag enables annotation of the fine-grained tags, i.e. Token.tag_ instead of Token.pos_. Note that this can lead to unexpected results and very long tags for some language models that use fine-grained tags composed of morphological features, like spaCy’s default Italian or Dutch models.

prodigy
pos.correct
dataset
spacy_model
source
--loader
--label
--exclude
--unsegmented
--fine-grained

Example

prodigy
pos.correct
news_pos
en_core_web_sm
./news_headlines.jsonl
--label NOUN,VERB,PROPN
prodigy pos.correct news_pos en_core_web_sm ./news_headlines.jsonl --label NOUN,VERB,PROPN

Collect the best possible training data for a part-of-speech tagging model with the model in the loop. Based on your annotations, Prodigy will decide which questions to ask next. It’s often more efficient to focus on a few labels at a time, instead of annotating all labels jointly.

prodigy
pos.teach
dataset
spacy_model
source
--loader
--label
--tag-map
--exclude
--unsegmented

Example

prodigy
pos.teach
pos_news
en_core_web_sm
./news_headlines.jsonl
--label VERB,NOUN,PROPN
prodigy pos.teach pos_news en_core_web_sm ./news_headlines.jsonl --label VERB,NOUN,PROPN

Create gold-standard data for dependency parsing by correcting the model’s suggestions. The spaCy model will be used to predict dependencies for the given labels, which the annotator can remove and correct if necessary. If --update is set, the model in the loop will be updated with the annotations and its updated predictions will be reflected in future batches. The recipe performs no example selection and all texts will be shown as they come in.

prodigy
dep.correct
dataset
spacy_model
source
--loader
--label
--update
--wrap
--unsegmented
--exclude

Example

prodigy
dep.correct
deps_news
en_core_web_sm
./news_headlines.jsonl
--label ROOT,csubj,nsubj,dobj,pboj
--update
prodigy dep.correct deps_news en_core_web_sm ./news_headlines.jsonl --label ROOT,csubj,nsubj,dobj,pboj --update

Collect the best possible training data for a dependency parsing model with the model in the loop. Based on your annotations, Prodigy will decide which questions to ask next. It’s often more efficient to focus on a few most relevant labels at a time, instead of annotating all labels jointly.

prodigy
dep.teach
dataset
spacy_model
source
--loader
--label
--exclude
--unsegmented

Example

prodigy
dep.teach
deps_news
en_core_web_sm
./news_headlines.jsonl
--label csubj,nsubj,dobj,pboj
prodigy dep.teach deps_news en_core_web_sm ./news_headlines.jsonl --label csubj,nsubj,dobj,pboj

Create training data for coreference resolution. Coreference resolution is the challenge of linking ambiguous mentions such as “her” or “that woman” back to an antecedent providing more context about the entity in question. This recipe allows you to focus on nouns, proper nouns and pronouns specifically, by disabling all other tokens. You can customize the labels used to extract those using the recipe arguments. Also see the usage guide on coreference annotation.

prodigy
coref.manual
dataset
spacy_model
source
--loader
--label
--pos-tags
--poss-pron-tags
--ner-labels
--exclude

Example

prodigy
coref.manual
coref_movies
en_core_web_sm
./plot_summaries.jsonl
--label COREF
prodigy coref.manual coref_movies en_core_web_sm ./plot_summaries.jsonl --label COREF

Annotate directional relations and dependencies between tokens and expressions by selecting the head, child and dependency label and optionally assign labelled spans for named entities or other expressions. This workflow is extremely powerful and can be used for basic dependency annotation, as well as joint named entity and entity relation annotation. If --span-label defines additional span labels, a second mode for span highlighting is added.

The recipe lets you take advantage of several efficiency tricks: spans can be pre-defined using an existing NER dataset, entities or noun phrases from a model or fully custom match patterns. You can also disable certain tokens to make them unselectable. This lets you focus on what matters and prevents annotators from introducing mistakes. For more details and examples, check out the usage guide on custom relation annotation and see the task-specific recipes dep.correct and coref.manual that include pre-defined configurations.

prodigy
rel.manual
dataset
spacy_model
source
--loader
--label
--span-label
--patterns
--disable-patterns
--add-ents
--add-nps
--wrap
--exclude

Example

prodigy
rel.manual
relation_data
en_core_web_sm
./data.jsonl
--label COREF,OBJECT
--span-label PERSON,PRODUCT,NP
--disable-patterns ./disable_patterns.jsonl
--add-ents
--wrap
prodigy rel.manual relation_data en_core_web_sm ./data.jsonl --label COREF,OBJECT --span-label PERSON,PRODUCT,NP --disable-patterns ./disable_patterns.jsonl --add-ents --wrap

disable_patterns.jsonl{"pattern": [{"is_punct": true}]}
{"pattern": [{"pos": "VERB"}]}
{"pattern": [{"lower": {"in": ["'s", "’s"]}}]}

Annotate images by drawing rectangular bounding boxes and polygon shapes. Each shape will be added to the task’s "spans" with its label and a "points" property containing the [x, y] pixel coordinate tuples. See here for more details on the JSONL format. You can click and drag or click and release to draw boxes. Polygon shapes can also be closed by double-clicking when adding the last point, similar to closing a shape in Photoshop or Illustrator. Clicking on the label will select a shape so you can change the label or delete it.

prodigy
image.manual
dataset
source
--loader
--exclude
--width
--darken
--no-fetch
--remove-base64

Example

prodigy
image.manual
photo_objects
./stock-photos
--label LAPTOP,CUP,PLANT
prodigy image.manual photo_objects ./stock-photos --label LAPTOP,CUP,PLANT

Manually label regions for the given labels in the audio or video file. The recipe expects a directory of audio files as the source argument and will use the audio loader (default) to load the data. To load video files instead, you can set --loader video. Each added region will be added to the "audio_spans" with a start and end timestamp and the selected label.

prodigy
audio.manual
dataset
source
--loader
--label
--autoplay
--keep-base64
--fetch-media
--exclude

Example

prodigy
audio.manual
speaker_data
./recordings
--label SPEAKER_1,SPEAKER_2,NOISE
prodigy audio.manual speaker_data ./recordings --label SPEAKER_1,SPEAKER_2,NOISE

Recipe command

prodigy
audio.manual
speaker_data
./recordings
--loader video
--label SPEAKER_1,SPEAKER_2
prodigy audio.manual speaker_data ./recordings --loader video --label SPEAKER_1,SPEAKER_2

Manually transcribe audio and video files by typing the transcript into a text field. The recipe expects a directory of audio files as the source argument and will use the audio loader (default) to load the data. To load video files instead, you can set --loader video. The transcript will be stored as the key "transcript". To make it easier to toggle play and pause as you transcribe and to prevent clashes with the text input field (like with the default enter), this recipe lets you customize the keyboard shortcuts. To toggle play/pause, you can press command/option/alt/ctrl+enter or provide your own overrides via --playpause-key.

prodigy
audio.transcribe
dataset
source
--loader
--label
--autoplay
--keep-base64
--fetch-media
--playpause-key
--text-rows
--exclude

Example

prodigy
audio.transcribe
speaker_transcripts
./recordings
--text-rows 3
prodigy audio.transcribe speaker_transcripts ./recordings --text-rows 3

Train a model component (NER, text classification, tagger or parser) using one or more Prodigy datasets with annotations. The recipe calls into spaCy directly and can update an existing model or train a new model from scratch. Datasets will be merged and conflicts will be filtered out. If your data contains potentially conflicting annotations, it’s recommended to first use review to resolve them. If you specify an --output directory, the best model will be saved at the end. You can then load it into spaCy by pointing spacy.load at the directory.

prodigy
train
component
datasets
spacy_model
--init-tok2vec
--output
--eval-id
--eval-split
--n-iter
--batch-size
--dropout
--factor
--textcat-exclusive
--ner-missing
--binary
--silent

Example

prodigy
train
ner
fashion_brands_training
en_vectors_web_lg
--eval-id fashion_brands_eval
✔ Loaded model 'en_vectors_web_lg'
Created and merged data for 1235 total examples
Created and merged data for 500 total examples
Using 1235 train / 500 eval (from 'fashion_brands_eval')
Component: ner | Batch size: compounding | Dropout: 0.2 | Iterations: 10

ℹ Baseline accuracy: 0.000
=========================== ✨ Training the model ===========================

#    Loss       Precision   Recall     F-Score
--   --------   ---------   --------   --------
 1    1254.70      72.381     63.866     67.857
 2     575.05      70.079     74.790     72.358
 3     349.95      67.293     75.210     71.032
 4     255.48      65.455     75.630     70.175
 5     293.42      64.643     76.050     69.884
 6     260.93      64.643     76.050     69.884
 7     203.03      64.311     76.471     69.866
 8     139.05      64.085     76.471     69.732
 9     122.61      63.986     76.891     69.847
10      73.69      63.763     76.891     69.714

============================ ✨ Results summary ============================

Label           Precision   Recall   F-Score
-------------   ---------   ------   -------
FASHION_BRAND   70.079      74.790   72.358


Best F-Score   72.358
Baseline        0.000

Train a model component (NER, text classification, tagger or parser) with different portions of the training examples and print the accuracy figures and accuracy improvements with more data. This recipe takes pretty much the same arguments as train. --n-samples sets the number of sample models to train at different stages. For instance, 10 will train models for 10% of the examples, 20%, 30% and so on. This recipe is useful to determine the quality of the collected annotations, and whether more training examples will improve the accuracy. As a rule of thumb, if accuracy improves within the last 25%, training with more examples will likely result in better accuracy.

prodigy
train-curve
component
datasets
spacy_model
--init-tok2vec
--eval-id
--eval-split
--n-iter
--batch-size
--dropout
--n-samples
--textcat-exclusive
--ner-missing
--binary

Example

prodigy
train-curve
ner
news_headlines
en_vectors_web_lg
--n-iter 10
✔ Starting with model 'en_vectors_web_lg'
Training 4 times with 25%, 50%, 75%, 100% of the data
============================== ✨ Train curve ==============================

%      Accuracy   Difference
----   --------   ----------
  0%       0.33   baseline
 25%       0.73   +0.40
 50%       0.67   -0.07
 75%       0.80   +0.13
100%       0.91   +0.11

✔ Accuracy improved in the last sample
As a rule of thumb, if accuracy increases in the last segment, this could indicate that collecting more annotations of the same type might improve the model further.

Combine multiple datasets, merge annotations on the same examples and output a file in spaCy’s JSON format that you can use with spacy train. If an eval_output is provided, the --eval-split will be used to split the examples into training and evaluation data. This recipe will merge annotations for the different model components and outputs a combined training corpus. If an example is only present in one dataset type, its annotations for the other components will be missing values. It’s recommended to use the review recipe on the different annotation types first to resolve conflicts properly.

prodigy
data-to-spacy
output
eval_output
--lang
--ner
--textcat
--tagger
--parser
--textcat-exclusive
--ner-missing
--eval-split
--base-model

Example

prodigy
data-to-spacy
./train-data.json
./eval-data.json
--lang en
--ner news_ner_person,news_ner_org,news_ner_product
--textcat news_cats2018,news_cats2019
--eval-split 0.3
✔ Saved 1050 examples to ./eval-data.json
✔ Saved 2450 examples to ./train-data.json

Training in spaCy

spacy
train
en
./model
./train-data.json
./eval-data.json
--pipeline ner,textcat
--n-iter 20
--textcat-multilabel

Build a terminology list interactively using a model’s word vectors and seed terms, either a comma-separated list or a text file containing one term per line. Based on the seed terms, a target vector is created and only terms similar to that target vector are shown. As you annotate, the recipe iterates over the vector model’s vocab and updates the target vector with the words you accept.

prodigy
terms.teach
dataset
vectors
--seeds
--resume

Example

prodigy
terms.teach
prog_lang_terms
en_vectors_web_lg
--seeds Python,C++,Ruby

Convert a dataset collected with terms.teach or sense2vec.teach to a JSONL-formatted patterns file. You can optionally provide a spaCy model for tokenization to create token-based patterns and make them case-insensitive. If no model is provided, the patterns will be generated as exact string matches. Pattern files can be used in Prodigy to bootstrap annotation and pre-highlight suggestions, for example in ner.manual. You can also use them with spaCy’s EntityRuler for rule-based named entity recognition.

prodigy
terms.to-patterns
dataset
output_file
--label
--spacy-model
--case-sensitive

Example

prodigy
terms.to-patterns
prog_lang_terms
./prog_lang_patterns.jsonl
--label PROGRAMMING_LANGUAGE
--spacy-model blank:en

✨ Exported 59 patterns
./prog_lang_patterns.jsonl

Review existing annotations created by multiple annotators and resolve potential conflicts by creating one final “master annotation”. Can be used for both binary and manual annotations and supports all interfaces except image_manual and compare. If the annotations were created with a manual interface, the “most popular” version, e.g. the version most sessions agreed on, will be pre-selected automatically.

prodigy
review
dataset
in_sets
--label
--view-id
--fetch-media

Example

prodigy
review
food_reviews_final
food_reviews2019,food_reviews2018

Compare the output of your model and the output of a baseline on the same inputs. To prevent bias during annotation, Prodigy will randomly decide which output to suggest as the correct answer. When you exit the application, you’ll see detailed stats, including the preferred output. Expects two JSONL files where each entry has an "id" (to match up the outputs on the same input), and an "input" and "output" object with the content to render, e.g. the "text".

prodigy
compare
dataset
a_file
b_file
--no-random
--diff

prodigy
compare
eval_translation
./model_a.jsonl
./model_b.jsonl

model_a.jsonl{"id": 1, "input": {"text": "FedEx von weltweiter Cyberattacke getroffen"}, "output": {"text": "FedEx hit by worldwide cyberattack"}}

model_b.jsonl{"id": 1, "input": {"text": "FedEx von weltweiter Cyberattacke getroffen"}, "output": {"text": "FedEx from worldwide Cyberattacke hit"}}

Start the annotation server, display whatever comes in with a given interface and collect binary annotations. At the end of the annotation session, a breakdown of the answer counts is printed. The --view-id lets you specify one of the existing annotation interfaces – just make sure your input data includes everything the interface needs, since this recipe does no preprocessing and will just show you whatever is in the data. The recipe is also very useful if you want to re-annotate data exported with db-out.

prodigy
mark
dataset
source
--loader
--label
--view-id
--exclude

prodigy
mark
news_marked
./news_headlines.jsonl
--label INTERESTING
--view-id classification

Select examples based on match patterns and accept or reject the result. Unlike ner.manual with patterns, this recipe will only show examples if they contain pattern matches. It can be used for NER and text classification annotations – for instance, to bootstrap a text category if the classes are very imbalanced and not enough positive examples are presented during manual annotation or textcat.teach. The --label-task and --label-span flags can be used to specify where the label should be added. This will also be reflected via the "label" property (on the top-level task or the spans) in the data you create with the recipe. If --combine-matches is set, all matches will be presented together. Otherwise, each match will be presented as a separate task.

prodigy
match
dataset
spacy_model
source
--loader
--label
--patterns
--label-task
--label-span
--combine-matches
--exclude

prodigy
match
news_matched
blank:en
./news_headlines.jsonl
--patterns ./news_patterns.jsonl
--label ORG,PRODUCT
--label-span

Pretty-print the model’s predictions on the command line. Supports named entities and text categories and will display the annotations if the model components are available. For textcat annotations, only the category with the highest score is shown if the score is greater than 0.5.

prodigy
print-stream
spacy_model
source
--loader

Pretty-print annotations from a given dataset on the command line. Supports plain text, text classification and NER annotations. If no --style is specified, Prodigy will try to infer it from the data via the "_view_id" that’s automatically added since v1.8.

prodigy
print-dataset
dataset
--style

Export annotations in Prodigy’s JSONL format. If the output directory doesn’t exist, it will be created. If no output directory is specified, the data will be printed so it can be redirected to a file.

prodigy
db-out
dataset
out_dir
--dry

Example

prodigy
db-out
news_headlines
>
./news_headlines.jsonl
prodigy db-out news_headlines > ./news_headlines.jsonl

Merge two or more existing datasets into a new set, e.g. to create a final dataset that can be reviewed or used to train a model. Keeps a copy of the original datasets and creates a new set for the merged examples.

prodigy
db-merge
in_sets
out_set
--rehash
--dry

prodigy
db-merge
news_person,news_org,news_product
news_training
✔ Merged 2893 examples from 3 datasets
Created merged dataset 'news_training'

Import existing annotations to the database. Can load all file types supported by Prodigy. To import NER annotations, the files should be converted into Prodigy’s JSONL annotation format.

prodigy
db-in
dataset
in_file
--rehash
--dry

Remove a dataset or annotation session from a project. Can’t be undone. To see all dataset and session IDs in the database, use prodigy stats -ls.

prodigy
drop
dataset
--batch-size

Print Prodigy and database statistics. Specifying a dataset ID will show detailed stats for the dataset, like annotation counts and meta data. You can also choose to list all available dataset or session IDs.

prodigy
stats
dataset
-l
-ls
--no-format

Example

prodigy
stats
news_headlines
-l
============================= ✨ Prodigy Stats =============================

Version            1.9.0
Database Name      SQLite
Database Id        sqlite
Total Datasets     4
Total Sessions     23

================================ ✨ Datasets ================================

news_headlines, news_headlines_eval, github_docs, test

============================= ✨ Dataset Stats =============================

Dataset            news_headlines
Created            2017-07-29 15:29:28
Description        Annotate news headlines
Author             Ines
Annotations        1550
Accept             671
Reject             435
Ignore             444

Run a built-in or custom Prodigy recipe. The -F option lets you load a recipe from a simple Python file, containing one or more recipe functions. All recipe arguments will be available from the command line. To print usage info and a list of available arguments, use the --help flag.

prodigy
recipe_name
...recipe arguments
-F

Example

prodigy
custom-recipe
my_dataset
./data.jsonl
--custom-opt 123
-F recipe.py
prodigy custom-recipe my_dataset ./data.jsonl --custom-opt 123 -F recipe.py

recipe.pypseudocode 
import prodigy
from prodigy.components.loaders import JSONL

@prodigy.recipe(
    "custom-recipe",
    dataset=("The dataset", "positional", None, str),
    source_file=("A positional argument", "positional", None, str),
    custom_opt=("An option", "option", "co", int)
)
def custom_recipe_function(dataset, source_file, custom_opt=10):
    stream = JSONL(source_file)
    print("Custom option pased in via command line:", custom_opt)
    return {
        "dataset": dataset,
        "stream": stream,
        "view_id": "text"
    }