Fetch Concept Batches

Classes

class BatchEmbeddingPipeline(EmbeddingPipeline):
   self.reader = reader
   self.embedder = embedder
   self.store = store

An Embedding Pipeline that reads, embeds, and loads batches of concepts

Parameter	Type	Description
`reader`	ConceptReader	A ConceptReader that extracts concept records and can generate batches of Concepts
`embedder`	ConceptEmbedder	A ConceptEmbedder that can take batches of Concepts and encode them, returning batches of EmbeddedConcepts
`store`	EmbeddingStore	An EmbeddingStore that can take batches of Embedded concepts and store them somewhere

embed_batches() -> Generator[list[EmbeddedConcept]]:

Yields EmbeddedConcepts using the Concept Generator from the reader and passing them through the embedder

Generator for batches of EmbeddedConcepts

run_pipeline() -> None:

Run the complete pipeline, loading batches of concepts to the store

class AllConceptEmbeddingPipeline(EmbeddingPipeline):
    reader: ConceptReader
    embedder: ConceptEmbedder
    store: EmbeddingStore

An Embedding Pipeline that reads, embeds, and loads a complete set of concepts

Parameter	Type	Description
`reader`	ConceptReader	A ConceptReader that extracts concept records and returns them as concepts
`embedder`	ConceptEmbedder	A ConceptEmbedder that can take the list of Concepts and encode them, returning them as EmbeddedConcepts
`store`	EmbeddingStore	An EmbeddingStore that can take the set of embedded concepts and store them somewhere

run_pipeline() -> None:

Run the complete pipeline, extracting and loading the full set of concepts