Autonomous AI crawler n8n Workflow Template

n8n workflow template · 28 nodes · OpenAI

Autonomous AI crawler is a free n8n workflow template that starts when you run it manually in the n8n editor and works with OpenAI and Supabase. It has 28 nodes, 5 of them AI nodes (Call n8n Workflow Tool, OpenAI Chat Model, Structured Output Parser and AI Agent). The notes inside the workflow say: “This workflow autonomously navigates through given websites and retrieves social media profile links.” To run it, you add your own credentials for OpenAI API and Supabase API.

Download JSON How to import into n8n

At a glance

Trigger
Manual Trigger (Execute workflow)
Nodes
28
Apps
OpenAI and Supabase
AI nodes
5
Credentials
2 types
Category
Spreadsheets, databases and docs

Apps and integrations

  • OpenAI
  • Supabase

Node types used

  • Call n8n Workflow Tool
  • OpenAI Chat Model
  • Structured Output Parser
  • Edit Fields (Set)
  • Manual Trigger
  • Supabase
  • Merge
  • Markdown
  • HTML
  • Split Out
  • Remove Duplicates
  • Filter
  • Aggregate
  • HTTP Request
  • AI Agent

How it works

The text below comes from the sticky notes inside the workflow. We removed links, code samples and anything that looks like a key or token.

Autonomous AI crawler

This workflow autonomously navigates through given websites and retrieves social media profile links.

You can modify this workflow to retrieve other type of data (e.g. contact details or company profile summary).

Set parameters for execution

Pass only name and website values from database.

If you use other field namings, update this node.

Crawling agent (retrieve social media profile links)

Crawl website to extract social media profile links and return them in unified JSON format.

You can change type of retrieved data by editing prompt and parser schema.

Text scraper tool

This tool is designed to return all text from the given webpage.

Consider adding proxy for better crawling accuracy.

URL scraper tool

This tool is designed to return all links (URLs) from the given webpage.

Consider adding proxy for better crawling accuracy.

Setup notes

The text below comes from the sticky notes inside the workflow. We removed links, code samples and anything that looks like a key or token.

Note

  1. Complete video guide for this workflow is available on my YouTube.
  2. Remember to add your credentials and configure nodes.
  3. If you like this workflow, please subscribe to my YouTube channel and/or my newsletter.

Thank you for your support!

Get companies from database

Retrieve names and websites of companies from Supabase table to process crawling.

You can replace Supabase with other database of your choice.

Insert data to database

Add new rows in database table with extracted data.

You can replace Supabase with other database of your choice.

Nodes in this workflow

The 28 nodes in file order (sticky notes excluded). Branches and connections determine execution order in n8n:

  1. Text (Call n8n Workflow Tool)
  2. URLs (Call n8n Workflow Tool)
  3. OpenAI Chat Model (OpenAI Chat Model)
  4. JSON Parser (Structured Output Parser)
  5. Map company name and website (Edit Fields (Set))
  6. Execute workflow (Manual Trigger)
  7. Get companies (Supabase)
  8. Select company name and website (Edit Fields (Set))
  9. Set social media array (Edit Fields (Set))
  10. Merge all data (Merge)
  11. Insert new row (Supabase)
  12. Convert HTML to Markdown (Markdown)
  13. Retrieve URLs (HTML)
  14. Split out URLs (Split Out)
  15. Remove duplicated (Remove Duplicates)
  16. Set domain to path (Edit Fields (Set))
  17. Filter out invalid URLs (Filter)
  18. Aggregate URLs (Aggregate)
  19. Filter out empty hrefs (Filter)
  20. Set domain (text) (Edit Fields (Set))
  21. Add protocool to domain (text) (Edit Fields (Set))
  22. Get website (text) (HTTP Request)
  23. Set response (text) (Edit Fields (Set))
  24. Set domain (URL) (Edit Fields (Set))
  25. Get website (URL) (HTTP Request)
  26. Set response (URL) (Edit Fields (Set))
  27. Add protocool to domain (URL) (Edit Fields (Set))
  28. Crawl website (AI Agent)

Credentials you need

  • OpenAI API – used by OpenAI Chat Model
  • Supabase API – used by Get companies and Insert new row

Configure your own credentials after importing. Create each credential in n8n, then select it in the node.

How to import into n8n

  1. Download the JSON file with the button above.
  2. In n8n, open a new workflow, select the ... menu and choose Import from File. You can also paste the JSON straight into the editor canvas.
  3. Open each node that shows a warning and select or create its credential.
  4. Read the notes on the canvas, test the workflow with sample data, then activate it.

Do you need somewhere to run it? Read our guide to self-hosted n8n.

Need this customised?

We can adapt this workflow to your apps, data and process, then deploy and support it.

Contact Us