Crawl websites, upload files, pull integrations, and track every ingestion job.
Larkup gives you one Data page for every source. Add content once, and Larkup indexes it in the background.
The Data page, with a tab for every source
Website
Files
Text
Media
Integrations
Search by keyword or paste a direct URL. The built in local crawler works immediately; switch to the Cloud web crawler in Settings for JavaScript heavy or protected sites.
Starting a crawl by keyword
Jobs run in the background. Track pages processed and status right in the tab.
A running crawl job
Drop in PDF, Word, CSV, JSON, Excel, or plain text, then choose Add file. Or select From URL, paste a public file link, and choose the same Add file action; the remote file is loaded and added in one step. Larkup turns GitHub and Hugging Face share links into downloadable file URLs automatically, checks every redirect is public, and limits remote files to 50 MB.Files, Websites, and Media use the same clearly selected source switch and the same disabled-to-active primary-button behavior when their input is ready.Structured files get a column and schema preview before you confirm. For an Excel workbook, every non-empty worksheet is loaded as its own named dataset so Chat can query the right sheet. Remote imports retain their original URL in document metadata for traceability.
Files staged with a schema preview
Paste notes, transcripts, or any text you want to search, with a title.
Adding raw text directly
Upload images, video, or audio, or paste a YouTube URL. Install the Video Intelligence marketplace tool for transcription and frame analysis. See Chat with Video.
Importing a video from a URL
Connect Notion, Google Drive, Slack, Jira, and more. Choose Configure or Connect, authorize through OAuth or an API key, and Larkup pulls the content in and indexes it.
Available integrations
Service
Category
What gets indexed
Notion
Documentation
Pages, databases, and knowledge bases
Google Drive
Documentation
Files and shared drives
Slack
Communication
Channels, messages, and DMs
Confluence
Documentation
Pages, spaces, and knowledge bases
Jira
Project management
Issues, sprints, and projects
Linear
Project management
Issues, projects, and roadmaps
GitHub
Engineering
Repositories and README knowledge
Salesforce
CRM
Accounts, contacts, and opportunities
Don’t see what you need? Select Ask for integration. Sync again anytime to pull fresh content.
Open Data → Groups to create collections and control whether each collection is available to the Assistant. Use a custom group’s actions menu to rename it or delete it.Only empty custom groups can be deleted. Move their sources to another group from All Data, or delete those sources first. The Default group cannot be deleted.
Configure the web scraper and web search provider under Settings → Search & Scraping. No .env editing required.
Search & Scraping settings
The built-in crawler works through the Larkup API without Docker, a separate browser service, or an API key. For JavaScript-heavy or bot-protected sites, you can optionally select Firecrawl Cloud and add its API key. Organizations that route traffic through a proxy can configure it here too.
Proxy settings for secure scraping
{"docsConfig":{"theme":"mint","$schema":"https://mintlify.com/docs.json","name":"Larkup","colors":{"primary":"#000000","light":"#000000","dark":"#222222"},"logo":{"light":"https://mintcdn.com/larkup/J0lnyYIE7T-PEajl/documentation/logo/logo-light2.png?fit=max\u0026auto=format\u0026n=J0lnyYIE7T-PEajl\u0026q=85\u0026s=a8d0f36de4d9d0698e8a6ee73e6bf1aa","dark":"https://mintcdn.com/larkup/J0lnyYIE7T-PEajl/documentation/logo/logo-dark2.png?fit=max\u0026auto=format\u0026n=J0lnyYIE7T-PEajl\u0026q=85\u0026s=fa0405df7faa06592472499c015d828c"},"favicon":"https://mintcdn.com/larkup/J0lnyYIE7T-PEajl/documentation/images/favicon.ico?fit=max\u0026auto=format\u0026n=J0lnyYIE7T-PEajl\u0026q=85\u0026s=feef65be53c3a3fe73a8289e38d3fd83","background":{"color":{"light":"#F9FAFB","dark":"#0F1117"}},"navbar":{"links":[{"href":"mailto:abdelrahmanaboneda@gmail.com","label":"Support"}],"primary":{"type":"button","label":"GitHub","href":"https://github.com/Larkup-AI/larkup"}},"footer":{"socials":{"github":"https://github.com/Larkup-AI/larkup"}},"contextual":{"options":["copy"],"display":"header"},"styling":{"latex":false},"navigation":{"pages":[]}},"deployment":{"deploymentMetadataValue":{"subdomain":"larkup","hasAudioNarration":false,"actualSubdomain":"larkup","customDomains":["larkup.de"],"gitSource":{"type":"github","owner":"larkup-ai","repo":"larkup","deployBranch":"main","contentDirectory":"docs","isPrivate":false,"mountPath":"","label":""},"chroma":{"collectionId":"41ed47f6-9e92-488c-bf50-0a09ec845389","liveDeploymentHistoryId":"6ad54ec7cfe0d5be20f803dd","schemaVersion":"v3","latestBuiltDeploymentHistoryId":"6ad54ec7cfe0d5be20f803dd"},"entitlements":{},"buildId":"6ad54ec7cfe0d5be20f803dd","clientVersion":"0.0.3971","searchFilterCounts":[{"language":"default","tag":"Documentation","count":13},{"language":"default","tag":"API Reference","count":11},{"language":"default","tag":"Use Cases","count":6},{"language":"default","tag":"Documents","count":5},{"language":"default","tag":"Getting Started","count":5},{"language":"default","tag":"Marketplace","count":5},{"language":"default","tag":"SDK","count":5},{"language":"default","tag":"Developer Guide","count":4},{"language":"default","tag":"Querying","count":2},{"language":"default","tag":"Overview","count":2}]},"assistantConfig":{"enableAskAiButton":false,"enableStarterQuestions":false,"enableContextualStarterQuestions":false,"enableAskAiSearchBar":true},"basePath":"/documentation","hasChatPermissions":false,"hasWebMcpPermissions":false,"toggles":[{"name":"kill-isr","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-mintlify-slack","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-v7-file-tree","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-integrations-settings","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-pr-grouping","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-prefix","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"analytics-v7","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"user-flows-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"no-publish-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-onboarding","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"llms-txt-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"pause-universal-search-reconcile","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving-live","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"stripTemperatureMiddleware","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"per-site-search-delta","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"support-platform-pylon-zendesk","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"instant-rollback","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications-server","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-private-page-search","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-fable","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fixed-ai-action-pricing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automation-resolve-merge-conflict","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"scoped-publishing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"mintql","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"page-level-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-version-history-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-nav-generations","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-page-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workspace-projects","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"assistant-v5","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"byok-sep-25","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fork-pr-previews","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"revalidation-jwt","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"vite-origin","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-agent-model-picker","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-updated-9-15","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-sso-scim","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"hobby-abuse-scan","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"dashboard-inbox","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-signup","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-enabled","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false}],"isAtlas":false,"googleFontsInlined":false},"customCss":[{"filePath":"documentation/style.css","content":""}],"page":{"pagePath":"documentation/guide/data-loading","slug":"/documentation/guide/data-loading","pageMetadata":{"title":"Web Crawl at Scale","description":"Crawl websites, upload files, pull integrations, and track every ingestion job.","href":"/documentation/guide/data-loading"},"description":{"compiledSource":"\"use strict\";\nconst {jsx: _jsx} = arguments[0];\nconst {useMDXComponents: _provideComponents} = arguments[0];\nfunction _createMdxContent(props) {\n const _components = {\n p: \"p\",\n ..._provideComponents(),\n ...props.components\n };\n return _jsx(_components.p, {\n children: \"Crawl websites, upload files, pull integrations, and track every ingestion job.\"\n });\n}\nfunction MDXContent(props = {}) {\n const {wrapper: MDXLayout} = {\n ..._provideComponents(),\n ...props.components\n };\n return MDXLayout ? _jsx(MDXLayout, {\n ...props,\n children: _jsx(_createMdxContent, {\n ...props\n })\n }) : _createMdxContent(props);\n}\nreturn {\n default: MDXContent\n};\n","frontmatter":{},"scope":{}},"pageType":"default","mdx":{"mdxSource":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx,jsxs:_jsxs}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];function _createMdxContent(props){const _components={a:`a`,code:`code`,p:`p`,strong:`strong`,tbody:`tbody`,td:`td`,th:`th`,thead:`thead`,tr:`tr`,..._provideComponents(),...props.components},{Frame,Heading,OptimizedImage,Tab,Table,Tabs}=_components;if(!Frame)_missingMdxReference(`Frame`,true);if(!Heading)_missingMdxReference(`Heading`,true);if(!OptimizedImage)_missingMdxReference(`OptimizedImage`,true);if(!Tab)_missingMdxReference(`Tab`,true);if(!Table)_missingMdxReference(`Table`,true);if(!Tabs)_missingMdxReference(`Tabs`,true);return _jsxs(_Fragment,{children:[_jsxs(_components.p,{children:[`Larkup gives you one `,_jsx(_components.strong,{children:`Data`}),` page for every source. Add content once, and Larkup indexes it in the background.`]}),`\n`,_jsx(Frame,{caption:`The Data page, with a tab for every source`,children:_jsx(OptimizedImage,{src:`https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/website-tab.png?fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=0d40442a5860437f92fee81984e024bd`,alt:`Data page with Files, Text, Website, Media, and Integrations tabs`,\"data-og-width\":`3012`,width:`3012`,\"data-og-height\":`1442`,height:`1442`,\"data-path\":`documentation/images/add/website-tab.png`,\"data-optimize\":`true`,\"data-opv\":`3`,srcset:`https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/website-tab.png?w=280\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=a91f6d8c1aaa490db72f05696ce0d211 280w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/website-tab.png?w=560\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=4c038d59f4e2392c0670abc8ce9fe691 560w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/website-tab.png?w=840\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=560e7b90dbd0aed6b76ce7c5a8b62bfc 840w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/website-tab.png?w=1100\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=b2da05004caea86c5fc601031776215d 1100w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/website-tab.png?w=1650\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=f9f4715a1622cbc402dae23f0c141cd8 1650w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/website-tab.png?w=2500\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=fa818e735bebbe907db8820ed9a3dfd6 2500w`})}),`\n`,_jsxs(Tabs,{children:[_jsxs(Tab,{title:`Website`,id:`website`,children:[_jsx(_components.p,{children:`Search by keyword or paste a direct URL. The built in local crawler works immediately; switch to the Cloud web crawler in Settings for JavaScript heavy or protected sites.`}),_jsx(Frame,{caption:`Starting a crawl by keyword`,children:_jsx(OptimizedImage,{src:`https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/crawl-start.png?fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=fea3373c6a2980a5197463115915b757`,alt:`Website search results ready to add`,\"data-og-width\":`3012`,width:`3012`,\"data-og-height\":`1442`,height:`1442`,\"data-path\":`documentation/images/crawl/crawl-start.png`,\"data-optimize\":`true`,\"data-opv\":`3`,srcset:`https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/crawl-start.png?w=280\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=9debcbb485d95100f4971814636c83eb 280w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/crawl-start.png?w=560\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=76e16ba587aab4168a7f966bddc825ef 560w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/crawl-start.png?w=840\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=fb57b7aada93af89a10684ef0ae725e2 840w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/crawl-start.png?w=1100\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=b82aa521130a5c51ee3ccfaecd8cc830 1100w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/crawl-start.png?w=1650\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=cf5c44b441623f7302331fd9ae44e759 1650w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/crawl-start.png?w=2500\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=3be93a12c0ecd56b82f3c89c14ef67e6 2500w`})}),_jsx(_components.p,{children:`Jobs run in the background. Track pages processed and status right in the tab.`}),_jsx(Frame,{caption:`A running crawl job`,children:_jsx(OptimizedImage,{src:`https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/crawl-jobs-view.png?fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=970c7086938333dc7558ed0a24136142`,alt:`Running crawl job with page count and status`,\"data-og-width\":`3012`,width:`3012`,\"data-og-height\":`1442`,height:`1442`,\"data-path\":`documentation/images/crawl/crawl-jobs-view.png`,\"data-optimize\":`true`,\"data-opv\":`3`,srcset:`https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/crawl-jobs-view.png?w=280\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=3f12675858a15f6588525e9628e49b18 280w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/crawl-jobs-view.png?w=560\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=8f97d3a14ae0ff4b91b3d9092034d12a 560w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/crawl-jobs-view.png?w=840\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=a3bee4a85a5aa781c725247561cc2c85 840w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/crawl-jobs-view.png?w=1100\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=6c1c30c1b747eb28b5c312b4aa4e34a5 1100w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/crawl-jobs-view.png?w=1650\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=4956e9bfa14ce971bf570202c2373784 1650w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/crawl-jobs-view.png?w=2500\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=c5db5347b965efdc20e5f38d20562f2a 2500w`})})]}),_jsxs(Tab,{title:`Files`,id:`files`,children:[_jsxs(_components.p,{children:[`Drop in PDF, Word, CSV, JSON, Excel, or plain text, then choose `,_jsx(_components.strong,{children:`Add file`}),`. Or select `,_jsx(_components.strong,{children:`From URL`}),`, paste a public file link, and choose the same `,_jsx(_components.strong,{children:`Add file`}),` action; the remote file is loaded and added in one step. Larkup turns GitHub and Hugging Face share links into downloadable file URLs automatically, checks every redirect is public, and limits remote files to 50 MB.`]}),_jsx(_components.p,{children:`Files, Websites, and Media use the same clearly selected source switch and the same disabled-to-active primary-button behavior when their input is ready.`}),_jsx(_components.p,{children:`Structured files get a column and schema preview before you confirm. For an Excel workbook, every non-empty worksheet is loaded as its own named dataset so Chat can query the right sheet. Remote imports retain their original URL in document metadata for traceability.`}),_jsx(Frame,{caption:`Files staged with a schema preview`,children:_jsx(OptimizedImage,{src:`https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/files-tab2.png?fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=83e1a4c7d29d00c44d76ea7137c3acb7`,alt:`Files tab with a PDF and an Excel schema preview`,\"data-og-width\":`3012`,width:`3012`,\"data-og-height\":`1442`,height:`1442`,\"data-path\":`documentation/images/add/files-tab2.png`,\"data-optimize\":`true`,\"data-opv\":`3`,srcset:`https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/files-tab2.png?w=280\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=bbd6d86b5a836bc5d4180f08833acb04 280w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/files-tab2.png?w=560\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=32ae345699862e3dec1ed347490a725f 560w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/files-tab2.png?w=840\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=a9e4d0f6a10c91c1b855f7fdadcb81e0 840w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/files-tab2.png?w=1100\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=b4bf28b7cfdb4c1a983d8ba4b787631a 1100w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/files-tab2.png?w=1650\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=6f71be1c1e8ced49b09d26eafa68556d 1650w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/files-tab2.png?w=2500\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=4a575780ee4c5ad9c17d5f7e9721d17c 2500w`})})]}),_jsxs(Tab,{title:`Text`,id:`text`,children:[_jsx(_components.p,{children:`Paste notes, transcripts, or any text you want to search, with a title.`}),_jsx(Frame,{caption:`Adding raw text directly`,children:_jsx(OptimizedImage,{src:`https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/text-tab.png?fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=a248b348f08b4d827a9d48877a36a635`,alt:`Add Text tab with a title and content field`,\"data-og-width\":`3012`,width:`3012`,\"data-og-height\":`1442`,height:`1442`,\"data-path\":`documentation/images/add/text-tab.png`,\"data-optimize\":`true`,\"data-opv\":`3`,srcset:`https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/text-tab.png?w=280\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=684168d61fbb1ff55b9bf622fb7be8fe 280w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/text-tab.png?w=560\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=c32be8ec164ccf183f854230d2dacdb4 560w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/text-tab.png?w=840\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=1978143082879f1d06409ef45226240e 840w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/text-tab.png?w=1100\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=5ca04f4e7fef2d60f28a333dd6f3f132 1100w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/text-tab.png?w=1650\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=1ab114dec4addabe78ad7bfa15efc3c2 1650w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/text-tab.png?w=2500\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=62e40027eab48785d615ceb8845bb5bf 2500w`})})]}),_jsxs(Tab,{title:`Media`,id:`media`,children:[_jsxs(_components.p,{children:[`Upload images, video, or audio, or paste a YouTube URL. Install the Video Intelligence marketplace tool for transcription and frame analysis. See `,_jsx(_components.a,{href:`/documentation/guide/video-chat`,children:`Chat with Video`}),`.`]}),_jsx(Frame,{caption:`Importing a video from a URL`,children:_jsx(OptimizedImage,{src:`https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/media-tab.png?fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=870aee2a1ab5cf00d4386e6f35662ccd`,alt:`Add Media tab with a YouTube URL import`,\"data-og-width\":`3012`,width:`3012`,\"data-og-height\":`1442`,height:`1442`,\"data-path\":`documentation/images/add/media-tab.png`,\"data-optimize\":`true`,\"data-opv\":`3`,srcset:`https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/media-tab.png?w=280\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=0ecce589b7096b3ee4333ae14d1f857b 280w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/media-tab.png?w=560\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=9fd5c07452cdd7cd7190450774e09ffa 560w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/media-tab.png?w=840\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=399df57b9e5e657d66454bafa6921a98 840w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/media-tab.png?w=1100\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=bee66735ea1b50c6ce25764c5f617119 1100w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/media-tab.png?w=1650\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=34a6739e7ba70f4203746e63f27f01f1 1650w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/media-tab.png?w=2500\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=ae3ee4fb236a02628efc8a73d7add137 2500w`})})]}),_jsxs(Tab,{title:`Integrations`,id:`integrations`,children:[_jsxs(_components.p,{children:[`Connect Notion, Google Drive, Slack, Jira, and more. Choose `,_jsx(_components.strong,{children:`Configure`}),` or `,_jsx(_components.strong,{children:`Connect`}),`, authorize through OAuth or an API key, and Larkup pulls the content in and indexes it.`]}),_jsx(Frame,{caption:`Available integrations`,children:_jsx(OptimizedImage,{src:`https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/integrations-tab.png?fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=6294d368e5e1e6c33371ae4ae04d1d15`,alt:`Integrations tab with Notion, GitHub, Jira, Slack, and more`,\"data-og-width\":`3012`,width:`3012`,\"data-og-height\":`1442`,height:`1442`,\"data-path\":`documentation/images/add/integrations-tab.png`,\"data-optimize\":`true`,\"data-opv\":`3`,srcset:`https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/integrations-tab.png?w=280\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=3b7e324e5d362fccc3b3b60e1dcb51ab 280w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/integrations-tab.png?w=560\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=5692b73076a8ff8bfd25563b9eb5a200 560w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/integrations-tab.png?w=840\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=c86ec33323455e8862756069bcef3a9f 840w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/integrations-tab.png?w=1100\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=83614560d0a54749f0f381add44b8fcd 1100w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/integrations-tab.png?w=1650\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=f1bbf9dec761ca989c3645c680e8f4a5 1650w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/integrations-tab.png?w=2500\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=f89cd07ff2d1fd9c7bbb5fe7397f557c 2500w`})}),_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Service`}),_jsx(_components.th,{children:`Category`}),_jsx(_components.th,{children:`What gets indexed`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Notion`}),_jsx(_components.td,{children:`Documentation`}),_jsx(_components.td,{children:`Pages, databases, and knowledge bases`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Google Drive`}),_jsx(_components.td,{children:`Documentation`}),_jsx(_components.td,{children:`Files and shared drives`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Slack`}),_jsx(_components.td,{children:`Communication`}),_jsx(_components.td,{children:`Channels, messages, and DMs`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Confluence`}),_jsx(_components.td,{children:`Documentation`}),_jsx(_components.td,{children:`Pages, spaces, and knowledge bases`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Jira`}),_jsx(_components.td,{children:`Project management`}),_jsx(_components.td,{children:`Issues, sprints, and projects`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Linear`}),_jsx(_components.td,{children:`Project management`}),_jsx(_components.td,{children:`Issues, projects, and roadmaps`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`GitHub`}),_jsx(_components.td,{children:`Engineering`}),_jsx(_components.td,{children:`Repositories and README knowledge`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Salesforce`}),_jsx(_components.td,{children:`CRM`}),_jsx(_components.td,{children:`Accounts, contacts, and opportunities`})]})]})]}),_jsxs(_components.p,{children:[`Don’t see what you need? Select `,_jsx(_components.strong,{children:`Ask for integration`}),`. Sync again anytime to pull fresh content.`]})]})]}),`\n`,_jsx(Heading,{level:`2`,id:`your-corpus`,children:`Your corpus`}),`\n`,_jsxs(_components.p,{children:[`Every loaded document lands in `,_jsx(_components.strong,{children:`Data → All Data`}),`, with its source, status, and character count.`]}),`\n`,_jsx(Frame,{caption:`All Data, every indexed document in one table`,children:_jsx(OptimizedImage,{src:`https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/data-table-view.png?fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=d485f3a248f354e3c171dfa412132db7`,alt:`All Data table with indexed documents`,\"data-og-width\":`1811`,width:`1811`,\"data-og-height\":`868`,height:`868`,\"data-path\":`documentation/images/add/data-table-view.png`,\"data-optimize\":`true`,\"data-opv\":`3`,srcset:`https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/data-table-view.png?w=280\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=eb31237c0641a5b7cdf29b1cf703d8c9 280w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/data-table-view.png?w=560\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=4c2605ed68a9494749ed004f89fe6c40 560w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/data-table-view.png?w=840\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=cc3a69af21d15afdc6f662a411605dbb 840w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/data-table-view.png?w=1100\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=bdef7d2cdca2f072d7a1966c0dab2607 1100w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/data-table-view.png?w=1650\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=eee27ade5a042a88229f694653284b44 1650w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/add/data-table-view.png?w=2500\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=7c9a13092ac3cc66d062ecadee8acaf5 2500w`})}),`\n`,_jsx(Heading,{level:`3`,id:`organize-sources-with-groups`,children:`Organize sources with groups`}),`\n`,_jsxs(_components.p,{children:[`Open `,_jsx(_components.strong,{children:`Data → Groups`}),` to create collections and control whether each collection is available to the Assistant. Use a custom group’s actions menu to rename it or delete it.`]}),`\n`,_jsxs(_components.p,{children:[`Only empty custom groups can be deleted. Move their sources to another group from `,_jsx(_components.strong,{children:`All Data`}),`, or delete those sources first. The Default group cannot be deleted.`]}),`\n`,_jsx(Heading,{level:`2`,id:`scraping-settings`,children:`Scraping settings`}),`\n`,_jsxs(_components.p,{children:[`Configure the web scraper and web search provider under `,_jsx(_components.strong,{children:`Settings → Search \u0026 Scraping`}),`. No `,_jsx(_components.code,{children:`.env`}),` editing required.`]}),`\n`,_jsx(Frame,{caption:`Search \u0026 Scraping settings`,children:_jsx(OptimizedImage,{src:`https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/search-scraping-settings.png?fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=b73545a55f5826c4423c732d14270f08`,alt:`Web search and web scraper provider settings`,\"data-og-width\":`3012`,width:`3012`,\"data-og-height\":`1442`,height:`1442`,\"data-path\":`documentation/images/crawl/search-scraping-settings.png`,\"data-optimize\":`true`,\"data-opv\":`3`,srcset:`https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/search-scraping-settings.png?w=280\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=989be9140d7cebf6f8c0acc49ffc5aac 280w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/search-scraping-settings.png?w=560\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=da6b4a5df09324733d397ca2398a0849 560w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/search-scraping-settings.png?w=840\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=1c63d2a90bf30a5d1482ab7e5514d2cf 840w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/search-scraping-settings.png?w=1100\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=77a7ead74186127d7b7b923e628b9b23 1100w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/search-scraping-settings.png?w=1650\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=6fcd132dcfe2e8a0b9805660c3d2a8c0 1650w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/search-scraping-settings.png?w=2500\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=5eaafbe9f437258e9fb038170eac9236 2500w`})}),`\n`,_jsx(_components.p,{children:`The built-in crawler works through the Larkup API without Docker, a separate browser service, or an API key. For JavaScript-heavy or bot-protected sites, you can optionally select Firecrawl Cloud and add its API key. Organizations that route traffic through a proxy can configure it here too.`}),`\n`,_jsx(Frame,{caption:`Proxy settings for secure scraping`,children:_jsx(OptimizedImage,{src:`https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/proxy-settings.png?fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=1d2ea5f899fe70209e782bfdee23d6ee`,alt:`Proxy server, username, and password fields`,\"data-og-width\":`3012`,width:`3012`,\"data-og-height\":`1442`,height:`1442`,\"data-path\":`documentation/images/crawl/proxy-settings.png`,\"data-optimize\":`true`,\"data-opv\":`3`,srcset:`https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/proxy-settings.png?w=280\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=0b5af00ed2b896d09dbafaab6cf8cdb2 280w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/proxy-settings.png?w=560\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=ca838b38fb3a1f0eda78311e5742b691 560w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/proxy-settings.png?w=840\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=7e04d130537ac307622bafb3f774aac1 840w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/proxy-settings.png?w=1100\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=0069eefafb95f6ea11939e5d3f105516 1100w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/proxy-settings.png?w=1650\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=0f36dc3ec40522570a484912e71d54c5 1650w, https://mintcdn.com/larkup/13l_fOT1PJRw-f4o/documentation/images/crawl/proxy-settings.png?w=2500\u0026fit=max\u0026auto=format\u0026n=13l_fOT1PJRw-f4o\u0026q=85\u0026s=9adf14d18012ac71bd7925dec25b4015 2500w`})})]})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{default:MDXContent};function _missingMdxReference(id,component){throw new Error(`Expected `+(component?`component`:`object`)+\" `\"+id+\"` to be defined: you likely forgot to import, pass, or provide it.\")}","frontmatter":{},"scope":{"config":{},"pageMetadata":{"title":"Web Crawl at Scale","description":"Crawl websites, upload files, pull integrations, and track every ingestion job.","href":"/documentation/guide/data-loading"}}},"mdxSourceWithNoJs":null},"mdxExtracts":{"tableOfContents":[{"title":"Your corpus","slug":"your-corpus","depth":2,"children":[{"title":"Organize sources with groups","slug":"organize-sources-with-groups","depth":3,"children":[]}]},{"title":"Scraping settings","slug":"scraping-settings","depth":2,"children":[]}],"codeExamples":{}},"jsLinks":[],"subdomain":"larkup","siteName":"Larkup","title":"Web Crawl at Scale - Larkup","navMaps":{"firstHrefInVersion":{},"firstHrefInLanguage":{}},"lastModified":"2026-10-05T16:06:24.863Z","originalFileLocation":"documentation/guide/data-loading.mdx"},"headHtml":"\u003cscript data-mint-head>(function(){var d=document.documentElement,p=null;if(!p){try{p=localStorage.getItem(\"isDarkMode\")}catch(e){}if(p==='true')p='dark';else if(p==='false')p='light';if(p!=='light'\u0026\u0026p!=='dark'\u0026\u0026p!=='system')p=\"system\"}var r=p==='system'?(matchMedia(\"(prefers-color-scheme: dark)\").matches?'dark':'light'):p;d.classList.remove(r==='dark'?'light':'dark');d.classList.add(r);d.setAttribute(\"data-theme-preference\",p)})();\u003c/script>\u003cmeta data-mint-head name=\"description\" content=\"Crawl websites, upload files, pull integrations, and track every ingestion job.\">\u003cmeta data-mint-head name=\"application-name\" content=\"Larkup\">\u003cmeta data-mint-head name=\"generator\" content=\"Mintlify\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/larkup/O-SbaNQDkZFAZqod/_generated/favicon/android-chrome-192x192.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"180x180\" href=\"/mintlify-assets/_mintlify/favicons/larkup/O-SbaNQDkZFAZqod/_generated/favicon/apple-touch-icon.png\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/larkup/O-SbaNQDkZFAZqod/_generated/favicon/favicon-16x16.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/larkup/O-SbaNQDkZFAZqod/_generated/favicon/favicon-32x32.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/larkup/O-SbaNQDkZFAZqod/_generated/favicon/favicon.ico\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/larkup/O-SbaNQDkZFAZqod/_generated/favicon-dark/android-chrome-192x192.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/larkup/O-SbaNQDkZFAZqod/_generated/favicon-dark/favicon-16x16.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/larkup/O-SbaNQDkZFAZqod/_generated/favicon-dark/favicon-32x32.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/larkup/O-SbaNQDkZFAZqod/_generated/favicon-dark/favicon.ico\" media=\"(prefers-color-scheme: dark)\">\u003cmeta data-mint-head property=\"og:title\" content=\"Web Crawl at Scale - Larkup\">\u003cmeta data-mint-head property=\"og:description\" content=\"Crawl websites, upload files, pull integrations, and track every ingestion job.\">\u003cmeta data-mint-head property=\"og:site_name\" content=\"Larkup\">\u003cmeta data-mint-head property=\"og:url\" content=\"https://larkup.de/documentation/guide/data-loading\">\u003cmeta data-mint-head property=\"og:type\" content=\"website\">\u003cmeta data-mint-head property=\"og:image\" content=\"https://larkup.mintlify.app/_mintlify/api/og/v2?division=Use+Cases\u0026amp;title=Web+Crawl+at+Scale\u0026amp;description=Crawl+websites%2C+upload+files%2C+pull+integrations%2C+and+track+every+ingestion+job.\u0026amp;theme=bde185ca85eb94a198a8ed1e\u0026amp;signature=80ac08a48500696701441d59f35643947c20eb1bde9780e24e0cf701a705f575\">\u003cmeta data-mint-head property=\"og:image:width\" content=\"1200\">\u003cmeta data-mint-head property=\"og:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"twitter:card\" content=\"summary_large_image\">\u003cmeta data-mint-head name=\"twitter:title\" content=\"Web Crawl at Scale - Larkup\">\u003cmeta data-mint-head name=\"twitter:description\" content=\"Crawl websites, upload files, pull integrations, and track every ingestion job.\">\u003cmeta data-mint-head name=\"twitter:image\" content=\"https://larkup.mintlify.app/_mintlify/api/og/v2?division=Use+Cases\u0026amp;title=Web+Crawl+at+Scale\u0026amp;description=Crawl+websites%2C+upload+files%2C+pull+integrations%2C+and+track+every+ingestion+job.\u0026amp;theme=bde185ca85eb94a198a8ed1e\u0026amp;signature=80ac08a48500696701441d59f35643947c20eb1bde9780e24e0cf701a705f575\">\u003cmeta data-mint-head name=\"twitter:image:width\" content=\"1200\">\u003cmeta data-mint-head name=\"twitter:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"msapplication-config\" content=\"/mintlify-assets/_mintlify/favicons/larkup/O-SbaNQDkZFAZqod/_generated/favicon/browserconfig.xml\">\u003cmeta data-mint-head name=\"apple-mobile-web-app-title\" content=\"Larkup\">\u003cmeta data-mint-head name=\"msapplication-TileColor\" content=\"#000000\">\u003clink data-mint-head rel=\"canonical\" href=\"https://larkup.de/documentation/guide/data-loading\">\u003clink data-mint-head rel=\"alternate\" type=\"application/xml\" href=\"/sitemap.xml\">\u003clink data-mint-head rel=\"alternate\" type=\"text/markdown\" href=\"/documentation/guide/data-loading.md\">\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https://larkup.de/#organization\",\"name\":\"Larkup\",\"url\":\"https://larkup.de\",\"logo\":{\"@type\":\"ImageObject\",\"url\":\"https://mintcdn.com/larkup/J0lnyYIE7T-PEajl/documentation/logo/logo-light2.png?fit=max\u0026auto=format\u0026n=J0lnyYIE7T-PEajl\u0026q=85\u0026s=a8d0f36de4d9d0698e8a6ee73e6bf1aa\"}},{\"@type\":\"WebSite\",\"@id\":\"https://larkup.de#website\",\"name\":\"Larkup\",\"url\":\"https://larkup.de\",\"publisher\":{\"@id\":\"https://larkup.de/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https://larkup.de/documentation/guide/data-loading#webpage\",\"url\":\"https://larkup.de/documentation/guide/data-loading\",\"name\":\"Web Crawl at Scale\",\"description\":\"Crawl websites, upload files, pull integrations, and track every ingestion job.\",\"dateModified\":\"2026-10-05T16:06:24.863Z\",\"isPartOf\":{\"@id\":\"https://larkup.de#website\"},\"breadcrumb\":{\"@id\":\"https://larkup.de/documentation/guide/data-loading#breadcrumb\"}},{\"@type\":\"BreadcrumbList\",\"@id\":\"https://larkup.de/documentation/guide/data-loading#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Use Cases\",\"item\":\"https://larkup.de/documentation/guide/build-rag-server\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Web Crawl at Scale\",\"item\":\"https://larkup.de/documentation/guide/data-loading\"}]},{\"@type\":[\"Article\",\"TechArticle\"],\"@id\":\"https://larkup.de/documentation/guide/data-loading#article\",\"headline\":\"Web Crawl at Scale\",\"name\":\"Web Crawl at Scale\",\"description\":\"Crawl websites, upload files, pull integrations, and track every ingestion job.\",\"url\":\"https://larkup.de/documentation/guide/data-loading\",\"mainEntityOfPage\":{\"@id\":\"https://larkup.de/documentation/guide/data-loading#webpage\"},\"dateModified\":\"2026-10-05T16:06:24.863Z\",\"publisher\":{\"@id\":\"https://larkup.de/#organization\"},\"isPartOf\":{\"@id\":\"https://larkup.de#website\"}}]}\u003c/script>\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@type\":\"WebSite\",\"name\":\"Larkup\",\"creator\":{\"@type\":\"Organization\",\"name\":\"Mintlify\",\"url\":\"https://mintlify.com\"}}\u003c/script>","siteName":"Larkup","nav":{"global":null,"tabs":[{"tab":"Documentation","groups":[{"group":"Getting Started","pages":[{"title":"Introduction","href":"/documentation"},{"title":"Installation","href":"/documentation/guide/installation"},{"title":"Quickstart","href":"/documentation/guide/quickstart"}]},{"group":"Use Cases","pages":[{"title":"Build Your RAG Server","description":"Generate, launch, and deploy a complete RAG backend.","href":"/documentation/guide/build-rag-server"},{"title":"Web Crawl at Scale","href":"/documentation/guide/data-loading"},{"title":"Chat with Video","description":"Ask questions about a video and get answers with a timestamp.","href":"/documentation/guide/video-chat"},{"title":"Talk to Your Custom AI from Social Media","sidebarTitle":"AI in Social Media","href":"/documentation/guide/public-webhooks"},{"title":"Sign and Fill Documents","href":"/documentation/guide/document-editor"},{"title":"Analyze Tabular Data","href":"/documentation/guide/data-analysis"}]},{"group":"Developer Guide","pages":[{"title":"Agent Server API","href":"/documentation/developer/agent-runtime-api"},{"title":"Deploy Your Server","href":"/documentation/developer/deploy"},{"title":"CLI Reference","href":"/documentation/developer/cli"},{"title":"Configuration Reference","href":"/documentation/developer/configuration"}]}]},{"tab":"Marketplace","pages":[{"title":"Marketplace","href":"/documentation/marketplace/overview"}]},{"tab":"SDK","pages":[{"title":"SDK","href":"/documentation/sdk/overview"}]},{"tab":"API Reference","pages":[{"title":"API Reference","href":"/documentation/api-reference/overview"}]}]},"title":"Web Crawl at Scale - Larkup","buildId":"dpl_DFJrtvUi8HS5F9nkooKuDJsuD5BM","deploymentId":"dpl_DFJrtvUi8HS5F9nkooKuDJsuD5BM","dynamics":["07c022d89b","a74be98ba7","a9bb46feb3"],"chunks":["Frame-COGkNuA6.js","DynamicMarkdown-85clUA0i.js","zoom-image-lightbox-BZxD9MAj.js","lightbox-handle-r17bVUz3.js","Markdown-Cfiy0xZO.js","clsx-rL8ZCCw4.js","lib--scc4DXi.js","lib-B8HbPgcC.js","html-void-elements-mCSPHezU.js","dist-aeztFa9U.js","katex-YMM-FMeD.js","marked.esm-CZWythHk.js","AssistantPageLink-BywkCr5C.js"]}
