Loading...
Skip to main content

List Sources in a Corpus.​

List Sources in a Corpus.

Path Parameters
    corpusId string required

    The parent corpus to list sources in.

Query Parameters
    limit int32

    The maximum number of sources to return. The service may return fewer than this value. If unspecified, Fixie will choose a sensible default. The maximum allowed value is 1000.

    offset int32

    The number of entries in the result set to skip.

Responses

OK

Schema
    sources object[]

    The list of sources.

  • Array [
    • displayName string

    A human-readable name for this source.

    created date-time

    When this source was created.

    modified date-time

    When this source was last modified.

    corpusId string

    The corpus that this source belongs to.

    sourceId string

    The unique ID of this source.

    loadSpec object

    The specification of how to acquire documents for this source.

    maxDocuments int32

    The maximum number of documents to ingest. This cannot exceed 200 in general. If you need more documents in a single corpus, please contact the Fixie team.

    maxDocumentBytes int32

    The maximum size of an individual document in bytes. If unset, a reasonable default will be chosen by Fixie.

    relevantDocumentTypes object

    The types of documents to keep. Any documents surfaced during loading that don't match this filter will be discarded. If unset, all documents will be kept.

    include object

    Mime types must be in this set to be kept. Empty implies the universal set. That is, all mime types will be kept save those in the exclude set.

    mimeTypes string[]
    exclude object

    Mime types must not be in this set to be kept. Empty imples the empty set.

    mimeTypes string[]
    web object

    Allows loading documents by crawling the web.

    Only one of the web or static fields may be populated when creating a new source.

    startUrls string[] required

    The list of start URLs to crawl.

    maxDepth int32

    The maximum depth of links to traverse. If 0 (or unset), there will be no depth limit.

    includeGlobPatterns string[]

    A set of glob patterns matched against any additional discovered URLs. URLs matching these patterns will be included in the crawl, unless the URL matches any of the exclude_glob_patterns.

    excludeGlobPatterns string[]

    A set of glob patterns matched against any additional discovered URLs. URLs matching these patterns will be excluded from the crawl.

    static object

    Allows loading documents from a static source (e.g. a file upload).

    Only one of the web or static fields may be populated when creating a new source.

    documents object[] required

    The documents to load.

  • Array [
    • filename string required

    The filename of the document.

    mimeType string required

    The MIME type of the document.

    contents bytes required

    The contents of the document.

    metadata object

    The metadata to attach to this document.

    publicUrl string

    The public URL of the document, if any.

    language string

    The BCP47 language code of the document, if known.

    title string

    The title of the document, if known.

    description string

    The description of the document, if known.

    published date-time

    The timestamp that the document was published, if known.

  • ]
    • processSteps object[]

    How to process documents for this source. Each step will be run in order over all applicable documents (as defined by each step's relevant_document_types).

    If no steps are specified, the service will add default document transformation steps to convert various document types into text easily understood by an LLM.

  • Array [
    • stepName string required

    The human-readable name of the step.

    relevantDocumentTypes object

    A Filter to apply to mime types.

    include object

    Mime types must be in this set to be kept. Empty implies the universal set. That is, all mime types will be kept save those in the exclude set.

    mimeTypes string[]
    exclude object

    Mime types must not be in this set to be kept. Empty imples the empty set.

    mimeTypes string[]
    htmlToMarkdown object

    Transforms an HTML document into Markdown.

    unstructuredProcessor object

    Transforms a binary document into plain text.

  • ]
    • chunkSpec object

    Specification of how to chunk documents.

    inputSelector object

    The input documents that should be chunked. Only documents that correspond to UTF-8 encoded text can be chunked. Any other kind of document will fail.

    mimeTypeFilter object

    Filters documents based on their mime type.

    include object

    Mime types must be in this set to be kept. Empty implies the universal set. That is, all mime types will be kept save those in the exclude set.

    mimeTypes string[]
    exclude object

    Mime types must not be in this set to be kept. Empty imples the empty set.

    mimeTypes string[]
    originFilter object

    Filters documents based on their origin.

    origins object[]

    Document origins must match one of these to be kept.

  • Array [
    • load boolean
    processStep string
  • ]
    • chunkSize int32

    The desired chunk size for each chunk, in tokens. This is a strict maximum, as well as a target. Adjacent chunks will be combined if their total size is under this limit.

    maxChunksPerDocument int32

    The maximum number of chunks to produce for an individual document.

    maxChunksTotal int32

    The maximum number of chunks to produce in total. This cannot exceed 5000 in general. If you need more chunks in a single source, please contact the Fixie team.

    embedSteps object[]

    How to embed chunks for this source. Defaults will be provided if unset.

  • Array [
    • stepName string

    The human-readable name for this step.

    direct object

    Directly embeds chunks.

    parentChild object

    Embeds chunks using a parent-child strategy. Each chunk is split into multiple children, which are embedded individually. When the child is semantically similar to a query string, the parent is returned. This strategy produces no results for small chunks as it never returns the parent chunk itself. To embed the parent chunks also, use the DirectEmbedStrategy in addition to this one.

  • ]
    • description string

    A human-readable description of this source.

    stats object

    The stats for a source.

    status enum

    Possible values: [SOURCE_STATUS_UNSPECIFIED, SOURCE_STATUS_INITIALIZING, SOURCE_STATUS_READY, SOURCE_STATUS_UPDATING]

    The current status of this source, indicating whether it affects queries.

    This field should not be populated in client requests.

    lastUpdated date-time

    When a job last completed for this source.

    This field should not be populated in client requests.

    numDocs int32

    The total number of documents in this source.

    schedule object

    How/whether to automatically refresh a source.

    enablePeriodicRefresh boolean

    Whether to enable automatic refresh for this source.

  • ]
    • pageInfo object

    Information about the page of results returned.

    requestedPageSize int32

    The number of results requested.

    requestedOffset int32

    The offset specified in the request.

    totalResultCount int32

    The total number of results available.

Loading...