> ## Documentation Index > Fetch the complete documentation index at: https://blackbox.dasha.ai/llms.txt > Use this file to discover all available pages before exploring further. # Add a URL source to a knowledge base > Crawls the pages of a URL into the knowledge base and queues the first crawl at once. Each page becomes one Markdown document that starts with "# " and "Source: <url>"; its URL is stored on the document, and its metadata holds urlSourceId. Re-crawls skip unchanged pages, update changed ones, add new ones and delete the documents of pages that are gone; when a crawl finds fewer than half as many pages as the previous one, nothing is removed and lastRun.warning says why. Within a knowledge base a page is held by one URL source only. robots.txt is respected. ## OpenAPI ````yaml https://blackbox.dasha.ai/swagger/v1/swagger.json post /api/v1/rag/kb/{kbId}/sources openapi: 3.0.4 info: title: Dasha BlackBox Agent API description: API for managing AI agents and calls contact: name: DashaAI Team email: support@dasha.ai version: v1 servers: - url: https://blackbox.dasha.ai description: Dasha BlackBox Agent API security: - ApiKey: [] - OAuth: [] tags: - name: ActivityLogs - name: Agents - name: AgentTestCases - name: AgentToolTestResults - name: CallResults - name: Calls - name: Chats - name: Copilot - name: CustomerData - name: Mcp - name: Media - name: Misc - name: OpsDashboard - name: PronunciationDictionaries - name: Providers - name: SipAliases - name: SipCredentials - name: SipDomains - name: SipPhoneNumbers - name: SipRegisterCallWebhook - name: TextChat - name: TwilioProvider - name: Voice - name: WebhookTest - name: WebIntegrations - name: WebSocket description: WebSocket endpoints for real-time communication paths: /api/v1/rag/kb/{kbId}/sources: post: tags: - KnowledgeBaseSource summary: Add a URL source to a knowledge base description: >- Crawls the pages of a URL into the knowledge base and queues the first crawl at once. Each page becomes one Markdown document that starts with "# <title>" and "Source: <url>"; its URL is stored on the document, and its metadata holds urlSourceId. Re-crawls skip unchanged pages, update changed ones, add new ones and delete the documents of pages that are gone; when a crawl finds fewer than half as many pages as the previous one, nothing is removed and lastRun.warning says why. Within a knowledge base a page is held by one URL source only. robots.txt is respected. parameters: - name: kbId in: path description: Knowledge base ID required: true schema: type: string format: uuid requestBody: description: The source content: application/json-patch+json: schema: allOf: - $ref: '#/components/schemas/UrlSourceCreateRequest' application/json: schema: allOf: - $ref: '#/components/schemas/UrlSourceCreateRequest' text/json: schema: allOf: - $ref: '#/components/schemas/UrlSourceCreateRequest' application/*+json: schema: allOf: - $ref: '#/components/schemas/UrlSourceCreateRequest' responses: '201': description: Created content: text/plain: schema: $ref: '#/components/schemas/UrlSourceInfo' application/json: schema: $ref: '#/components/schemas/UrlSourceInfo' text/json: schema: $ref: '#/components/schemas/UrlSourceInfo' '400': description: Bad Request content: text/plain: schema: $ref: '#/components/schemas/ValidationProblemDetails' application/json: schema: $ref: '#/components/schemas/ValidationProblemDetails' text/json: schema: $ref: '#/components/schemas/ValidationProblemDetails' '404': description: Not Found content: text/plain: schema: {} application/json: schema: {} text/json: schema: {} '500': description: Internal Server Error content: text/plain: schema: {} application/json: schema: {} text/json: schema: {} components: schemas: UrlSourceCreateRequest: type: object properties: type: allOf: - $ref: '#/components/schemas/KnowledgeBaseSourceType' nullable: true url: type: string nullable: true scope: allOf: - $ref: '#/components/schemas/UrlSourceScope' pageLimit: type: integer format: int32 includePaths: type: array items: type: string nullable: true excludePaths: type: array items: type: string nullable: true recrawlIntervalHours: type: integer format: int32 nullable: true crawlerName: type: string nullable: true additionalProperties: false UrlSourceInfo: required: - createdAt - excludePaths - id - includePaths - knowledgeBaseId - pageCount - pageLimit - scope - type - url type: object properties: id: type: string format: uuid knowledgeBaseId: type: string format: uuid type: allOf: - $ref: '#/components/schemas/KnowledgeBaseSourceType' url: minLength: 1 type: string scope: allOf: - $ref: '#/components/schemas/UrlSourceScope' pageLimit: type: integer format: int32 includePaths: type: array items: type: string excludePaths: type: array items: type: string recrawlIntervalHours: type: integer format: int32 nullable: true crawlerName: type: string nullable: true pageCount: type: integer format: int32 createdAt: type: string format: date-time nextRunAt: type: string format: date-time nullable: true lastRun: allOf: - $ref: '#/components/schemas/UrlSourceRunInfo' nullable: true additionalProperties: false ValidationProblemDetails: type: object properties: errors: type: object additionalProperties: type: array items: type: string nullable: true type: type: string nullable: true title: type: string nullable: true status: type: integer format: int32 nullable: true detail: type: string nullable: true instance: type: string nullable: true additionalProperties: {} KnowledgeBaseSourceType: enum: - url type: string UrlSourceScope: enum: - section - page - site type: string UrlSourceRunInfo: required: - counts - id - status - trigger type: object properties: id: type: string format: uuid status: allOf: - $ref: '#/components/schemas/UrlSourceCrawlRunStatus' trigger: allOf: - $ref: '#/components/schemas/UrlSourceCrawlTrigger' startedAt: type: string format: date-time nullable: true finishedAt: type: string format: date-time nullable: true error: type: string nullable: true errorCode: allOf: - $ref: '#/components/schemas/UrlSourceCrawlErrorCode' nullable: true warning: type: string nullable: true counts: allOf: - $ref: '#/components/schemas/UrlSourceRunCounts' additionalProperties: false UrlSourceCrawlRunStatus: enum: - queued - running - completed - failed type: string UrlSourceCrawlTrigger: enum: - initial - scheduled - manual type: string UrlSourceCrawlErrorCode: enum: - not_configured - crawler_refused - page_allowance - crawler_failed type: string UrlSourceRunCounts: required: - covered - discovered - failed - fetched - removed - unchanged type: object properties: discovered: type: integer format: int32 fetched: type: integer format: int32 unchanged: type: integer format: int32 failed: type: integer format: int32 removed: type: integer format: int32 covered: type: integer format: int32 additionalProperties: false securitySchemes: ApiKey: type: http description: API Key Authentication (Bearer {key}) scheme: Bearer OAuth: type: oauth2 flows: implicit: authorizationUrl: https://auth.dasha.ai/connect/authorize scopes: platform_api: Platform API ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.