Ingesting in one call¶
result = sales.autograph.ingest(categories=["legal", "finance"])
One call runs the whole ingestion for the given categories and returns the outcome of every step: it builds the corpus, runs the strategizer, reads the stored strategies and imports the graph. Each step waits for the previous one to finish. There is no single platform endpoint for this; the SDK sequences the four calls for you.
ingest is the call to make when you want the context graph and do not
need to intervene between the steps. Call the steps yourself, see
Building the corpus, Choosing a strategy per cluster,
Reading the stored strategies and Importing the graph, when you want
to inspect the corpus before choosing strategies, pass build or strategizer
options ingest does not expose, or redo one step only.
Parameters¶
Parameter |
Type |
Meaning |
|---|---|---|
|
sequence of |
The categories to ingest. A category is the second label of the scope
the files were uploaded with. At least one is required. Passed to the
corpus build, the strategizer and the import. |
|
|
Passed to the strategizer: what share of the clusters gets a full
knowledge graph. Default |
|
|
Passed to the corpus build only. Default |
|
|
One deadline in seconds for the whole run, shared by the steps: what the build uses is gone for the strategizer and the import. Unset waits without bound. |
|
|
Seconds between two status polls, passed to each step. Default 30. |
|
callable, optional |
Called with an |
Return value¶
An IngestResult with four attributes, one
per step:
buildThe final
GetBuildStatusResponse. A partial build, one that completed witherror_codeset, lands here too; it does not stop the run.strategizeThe final
GetRagStrategizerJobStatusResponse, progress counts only.strategiesThe
GetRagStrategizerStrategyResponseread after the strategizer finished: which cluster got which strategy.orchestrationThe final
GetOrchestrationStatusResponse, orNonewhen every category was already up to date and the import had nothing to do.
Progress¶
on_progress receives one IngestProgress
per poll. Its step is "build", "strategize" or "orchestrate"
and its status is the status object that step just polled, so the fields
on it differ per step:
def show(progress):
if progress.step == "build":
print(f"build {progress.status.progress}%")
elif progress.step == "strategize":
print(f"strategize {progress.status.clusters_done}/{progress.status.clusters_total}")
else:
print(f"orchestrate {progress.status.phase}")
result = sales.autograph.ingest(categories=["legal"], on_progress=show)
Errors¶
ingest stops at the first step that fails and raises whatever that step
raised, so every error listed on the three step pages can come out of it.
The ones you are most likely to handle:
ValueErrorcategoriesis empty. Nothing was sent.AutoGraphNotDeployedErrorThe project has no AutoGraph service. Nothing was sent. See Deploying AutoGraph.
CategoryAlreadyBuiltErrorA category is already built and
incrementalisFalse. Nothing after the build request was sent.AutoGraphBusyErrorA conflicting job is already running on the project. Wait for it and retry.
IngestionFailedErrorA build or strategy run ended as failed, or the import ended as failed or cancelled.
job_statuson the exception is that step’s final status; its type tells you which step it was.JobTimeoutErrorThe shared
timeoutran out, either while waiting on a step or before the next step could start. The step that was running keeps going on the platform.
Any other failure surfaces as one of the general SDK exceptions described in Errors.
When a later step fails, the earlier ones have completed and their results
are stored on the platform. Rerun the failed step on its own rather than the
whole ingest: a second ingest would refuse at the build with
CategoryAlreadyBuiltError unless you pass
incremental=True.
Example¶
from arango_ai_sdk.exceptions import IngestionFailedError
sales = client.project.get(project_name="sales")
def show(progress):
print(progress.step, progress.status.status)
try:
result = sales.autograph.ingest(
categories=["legal", "finance"],
timeout=3600.0,
on_progress=show,
)
except IngestionFailedError as exc:
print(f"Ingestion failed: {exc}")
else:
print(f"{result.build.document_count} documents in the corpus")
print(f"strategies: {dict(result.strategies.strategy_type_counts)}")
if result.orchestration is None:
print("Graph already up to date")
else:
imported = sum(1 for job in result.orchestration.jobs if job.imported)
print(
f"Imported {imported} partitions, "
f"skipped {result.orchestration.skipped_jobs}"
)