How to Integrate Flowise with Zenrows

Integrate Zenrows with Flowise to build AI-powered web scraping workflows using a visual drag-and-drop builder for data collection pipelines.

Integrate Zenrows with Flowise to add reliable web scraping capabilities to your AI workflows. This guide shows you how to build data collection pipelines that combine Zenrows’ scraping technology with Flowise’s visual workflow builder.

What is Flowise?

Flowise is an open-source, low-code platform for visually building, managing, and interacting with AI agents and large language model (LLM) workflows in a chat-based environment. Its drag-and-drop interface allows users to compose AI agents into interconnected, LLM-powered workflows with little to no programming knowledge.

Use Cases

You can leverage Flowise-Zenrows integration to drive several business opportunities. Here are some of them:

• Product research: Discover the best marketing position for new products with insights from scraped data.
• Demand forecasting: Scrape products’ data to predict the demand ahead of competitors. Make data-driven decisions on re-stocks and pricing.
• Real-estate intelligence: Collect real-estate data and analyze it to gain data-driven investment insights.
• Competitor’s research: Gather competitor’s data for information gain and predict their next moves.
• Product optimization: Collect customers’ reviews, ratings, and other metrics from social platforms, review sites, and brand pages and fine-tune products to meet consumers’ expectations.
• Lead generation: Collect quality lead contacts from various sources and automate captures, outreaches, and follow-ups.
• Product recommendation: Scrape and analyze relevant product data points to recommend the best deals to potential buyers.

Basic Integration: Amazon Product Scraper

Let’s start by building a simple Amazon product scraper workflow that fetches data using Zenrows and returns it as a response via the Flowise chat box. We’ll use the Flowise Agentflows for this tutorial.

The Flowise platform is accessible locally or remotely via the official Flowise website. If you use Flowise remotely via the website, no installation steps are required. However, the local setup requires installing Flowise globally to run a local server.

In this guide, we’ll use the local setup, starting with the installation and server startup steps.

Step 1: Install Flowise and set up a server

This tutorial assumes you’ve set up Node.js on your machine. Otherwise, download and install it from the official website.

1. Install the flowise package globally:
   npm install -g flowise
   
2. Run the flowise server. The command below starts a server that defaults to http://localhost:3000:
   npx flowise start
   
3. Visit this URL to launch the Flowise interface.
4. Complete the initial registration steps, and you’ll get to the platform’s dashboard, as shown below:

Step 2: Add Zenrows API Key to Flowise environment variables

Adding your Zenrows API key to the Flowise environment variable makes it accessible across the platforms.

1. From the dashboard, go to Variables.
2. Click + Add Variable at the top-right.
3. Fill in the Variable Name field with the name of your API key. Then, paste your Zenrows API key in the Value field.
4. Click Add.

Step 3: Initiate a new Agentflow

1. Go to Agentflows and click + Add New at the top right to open the Flowise canvas.

   

2. Once in the Flowise Canvas, click the save icon at the top right.

   

3. Enter a name for your new workflow and click Save.

   

Step 4: Create a scraper flow with Zenrows

We’ll integrate the Zenrows API using an HTTP agent node, which enables you to send an API request via Flowise.

1. Click the + icon at the top left and search for “HTTP” using the search bar.

   

2. Drag and drop the HTTP node into your Canvas, placing it directly in front of the Start agent.

   

3. Link the agents by dragging a line from the Start node to connect with the HTTP agent node.

   

4. Double-click the HTTP agent node.

5. Click the edit icon at the top of the modal to change the agent’s name. Then, press Enter on your keyboard.

   

6. Click + Add Query Params to set the target URL parameter. We’ll retrieve this URL dynamically from the chat box:

   • Type url in the Key field.
   • Type double braces ({{) in the Value field. This loads dynamic input options.
   • Select question from the options to load the URL from the chat box.

7. Similarly, for the Zenrows API key, click + Add Query Params and type apikey in the Key field. Type {{ inside the Value and select $vars.ZENROWS_API_KEY.

8. Click + Add Query Params repeatedly to add each of the following Zenrows parameters in order:

   • js_render = true
   • premium_proxy = true
   • wait = 2500
   • css_extractor =
   • original_status = true
   

9. Add your CSS selectors as an array in the css_extractor value: We’ll use the following css_extractor for this tutorial:

   {
       "name":"span#productTitle",
       "price":"span.a-price.aok-align-center.reinventPricePriceToPayMargin.priceToPay",
       "ratings":"#acrPopover",
       "description":"ul.a-unordered-list.a-vertical.a-spacing-mini",
       "review summary":"#product-summary p span",
       "review count":"#acrCustomerReviewLink"
   }
   

10. Under Response Type, select Text.

    

11. Click on the Flowise Canvas. Then, click the Validate Nodes icon at the top right and select Validate Nodes to confirm that your setup is correct.

    

12. Finally, click the save icon at the top right to save your changes.

Step 5: Run the flow

1. Click the chat icon at the top right.

2. Type an Amazon product page URL in the box and press Enter to send your request.

3. Click the Process Flow banner in the chat box.

4. Click the expansion icon next to Scraper Agent.

   

5. You’ll see the following response page, showing the scraped data:

   

However, we want the LLM to return the data as the sole response in the chat. Let’s set this up using OpenAI.

Step 6: Integrate an LLM flow

1. Click the + icon at the top left. Search LLM. Then, drag and drop it in the canvas next to the Scraper Agent.

2. Link the Scraper Agent to the LLM node.

   

3. Double-click the LLM agent and rename it.

   

4. Under Model, search and select ChatOpenAI.

5. Click ChatOpenAI Parameters:

   • Click on Connect Credentials, then on Create New and set up your OpenAI API key and click Add.
   
   • Select your desired model, temperature (preferably, 0.1 or 0.2 for increased accuracy), and other parameters.
   

6. Click + Add Messages:

   • Under Role, select Assistant.
   • Type {{ inside the Content field and select httpAgentflow_0.
   

7. Toggle on Enable Memory.

   

8. Type the following prompt in the Input Message field. We’ve instructed the LLM to only show {{ httpAgentflow_0 }} as the output and not attempt to visit any URL provided in the chat:

   You are a web scraping assistant. Only display the response provided by the Scraper Agent in the chat box. Do not attempt to access or retrieve information from external URLs yourself; rely solely on the Scraper Agent for all web data. Your output must be {{ httpAgentflow_0 }}.
   

9. Set the Return Response As field to Assistant Message. This allows the chat model to return the Scraper Agent’s output in its response.

   

10. Click anywhere in the canvas. Then click the save icon at the top right.

    

11. To run the flow, click the chat icon and send the URL of an Amazon product page in the chat. The model now streams the scraped data as a response in the chat box:

    

Nice! You just created your first scraping flow by integrating Zenrows with Flowise.

Advanced Integration: Demand Analysis System

Besides the HTTP option, you can also integrate Zenrows with Flowise using custom functions. We’ll see a real-life use case using the following agents:

• Create an agent that scrapes an Amazon listing page based on a search term using Zenrows.
• Build an agent node to handle data cleaning.
• Create an agent to analyze the data using LLM.

Step 1: Create a custom web scraper function with Zenrows

1. Go to Agentflows and click + Add New to start a new flow.

2. Click the save icon and name your flow.

3. Click + at the top left of the Flowise canvas.

4. Search for Custom Function and drag its card to the canvas.

5. Link the Start node with the Custom Function agent.

   

6. Double-click the Custom Function agent and rename it.

7. Initiate the Zenrows scraping logic inside the JavaScript Function field. Import axios, configure the target URL to accept a search term, and load your Zenrows API key from the Flowise environment variables.

   Node.js

   const axios = require("axios");
   
   // declare the search term variable
   const searchTerm = $searchTerm;
   
   // load your API key from the environment variables
   const apikey = $vars.ZENROWS_API_KEY;
   
   const url = `https://www.amazon.com/s?k=${encodeURIComponent(searchTerm)}`;
   

8. To create the searchTerm variable, click + Add Input Variables.

9. Type searchTerm inside the Variable Name field.

10. We want the scraping logic to accept this search term dynamically from the user’s question in the chat box. Type {{ in the Variable Value and select question.

    

11. Extend the scraping logic by adding the required Zenrows parameters. Scrape the data using the css_extractor and return the response data. See the complete code below:

    Node.js

    const axios = require("axios");
    
    // declare the search term variable
    const searchTerm = $searchTerm;
    
    // load your API key from the environment variables
    const apikey = $vars.ZENROWS_API_KEY;
    
    const url = `https://www.amazon.com/s?k=${encodeURIComponent(searchTerm)}`;
    
    try {
    const response = await axios({
        url: "https://api.zenrows.com/v1/",
        method: "GET",
        params: {
        url: url,
        apikey: apikey,
        js_render: "true",
        wait: "2500",
        premium_proxy: "true",
        css_extractor: JSON.stringify({
            name: "div.s-title-instructions-style",
            url: "div.s-title-instructions-style a @href",
            listingPrice: "span.a-price",
            demandHistory:
            "div.a-section.a-section.a-spacing-none.a-spacing-top-micro span.a-size-base.a-color-secondary",
            reviewCount: "span.a-size-base.s-underline-text",
        }),
        },
    });
    
    return response.data;
    
    } catch (error) {
    console.error("Error fetching data:", error.message);
    throw error;
    }
    

12. Click the save icon at the top right to save your changes.

    

Step 2: Create a data cleaning agent

1. Click the + icon.

2. Search for Custom Function and drag and drop it onto the canvas.

3. Link this new custom function agent with the Scraper agent.

   

4. Double-click the Custom Function 1 agent node and rename it (e.g., Data Cleaner).

   

5. We need to load the Scraper agent’s output into the Data Cleaner agent node. Create this variable by clicking + Add Input Variables. Then, type scrapedData in the Variable Name field. Then, enter {{ in the Variable Value field and select customFunctionAgentflow_0.

   

6. Enter the following data cleaning code in the JavaScript Function field. This code first converts the scraped data into an object, merges it, and fixes anomalies, such as data inconsistencies.

   Node.js

   // convert the scraped data to an object
   let data;
   try {
   data = JSON.parse($scrapedData);
   } catch (e) {
   return [];
   }
   
   // ensure all arrays exist
   const demand = Array.isArray(data.demandHistory) ? data.demandHistory : [];
   const name = Array.isArray(data.name) ? data.name : [];
   const listingPrice = Array.isArray(data.listingPrice) ? data.listingPrice : [];
   const reviewCount = Array.isArray(data.reviewCount) ? data.reviewCount : [];
   const url = Array.isArray(data.url) ? data.url : [];
   
   const baseUrl = "https://www.amazon.com";
   
   // check if the demand entry is a real monthly demand
   const isValidDemand = (d) =>
   typeof d === "string" && d.includes("bought in past month");
   
   // merge and clean the data
   const products = [];
   let demandIdx = 0;
   
   for (let i = 0; i < name.length; i++) {
   // filter out sponsored/ad names
   if (
       typeof name[i] !== "string" ||
       name[i].toLowerCase().startsWith("sponsored") ||
       name[i].toLowerCase().includes("ad feedback")
   ) {
       continue;
   }
   
   // find the next valid demand entry
   let demandValue = "Unavailable";
   while (demandIdx < demand.length) {
       if (isValidDemand(demand[demandIdx])) {
       demandValue = demand[demandIdx];
       demandIdx++;
       break;
       }
       demandIdx++;
   }
   
   // clean doubled price issues
   let price = listingPrice[i] || "Unavailable";
   if (typeof price === "string") {
       const match = price.match(/\$\d+(\.\d{2})?/);
       if (match) {
       price = match[0];
       }
   }
   
   // ensure URL is absolute
   let productUrl = url[i] || "Unavailable";
   if (
       typeof productUrl === "string" &&
       productUrl !== "Unavailable" &&
       !productUrl.startsWith("http")
   ) {
       productUrl = baseUrl + productUrl;
   }
   
   products.push({
       name: name[i] || "Unavailable",
       demand: demandValue,
       price: price,
       reviews: reviewCount[i] || "Unavailable",
       url: productUrl,
   });
   }
   
   return products;
   

7. Click anywhere on the canvas and click the save icon.

   

Step 3: Create a Data Analyst Agent LLM

1. Click the + icon.

2. Search for LLM and drag and drop it into the canvas.

3. Link the LLM agent with the Data Cleaner agent. The LLM will act on the cleaned data returned from the Data Cleaner agent.

   

4. Double-click the LLM agent and give it a descriptive name.

5. From the Model dropdown, search and select ChatOpenAI or your preferred model.

6. Set up your credentials, select your model, and choose a temperature.

   

7. Click + Add Messages under the Messages section.

8. Create a prompt that instructs the Analyst agent to get the most in-demand products from the cleaned data and establish a correlation between price and demand. We’ll use the following prompt, where customFunctionAgentflow_1 is the cleaned data from the Data Cleaner agent node.

   Analyze the products in {{ customFunctionAgentflow_1 }}. Select the 4 most in-demand products based on the demand history (highest demand first). Also, tell if there is a correlation between price and demand across these products. For each product, format the product's URL as an HTML anchor tag with the text: Check product out.
   

9. Under Role, select Assistant. Then, enter the prompt into the Content field.

   

10. Under Return Response As, select Assistant Message.

    

11. Finally, validate and save your workflow by clicking the validation and save icons at the top right, respectively.

Step 4: Run the demand analysis workflow

Click the message icon to initiate a chat. Then, prompt the Analyst agent with a product search term (e.g., headphones, mouse, or teddy bears).

Here’s a sample response for “mouse” product listings:

Congratulations! 🎉 You just built a product research workflow with Zenrows and Flowise integration.

Troubleshooting

Error 401 or Zenrows authentication error

• Solution 1: Ensure you’ve entered the correct Zenrows API key in the Flowise environment variables.
• Solution 2: Ensure you load the Zenrows API key into your scraping workflow.

Error 400

• Solution 1: Make sure the supplied Zenrows parameters are correct and don’t contain unsupported strings.
• Solution 2: Use the correct CSS selectors and format the css_extractor array correctly.

LLM agent fails

• Solution 1: Check your network connectivity and try your prompt again.
• Solution 2: Ensure you still have enough API credit for your chosen LLM.
• Solution 3: Make sure you’ve supplied the correct API key of your LLM agent.

Custom function fails

• Solution 1: Execute the code in an external text editor, such as VS Code, and simulate the expected output for debugging.
• Solution 2: Make sure all variables in your code are listed as Input Variables.

localhost unavailable or the Flowise interface doesn’t load

• Solution: Stop and restart the running server. Then, reload the Flowise interface.

Conclusion

You’ve learned to connect Zenrows with Flowise for a streamlined data flow across several agents, including data collection, cleaning, and analysis. While Flowise enables you to build agentic workflows for data automation and other purposes, Zenrows ensures the integrity of your data pipeline with a consistent data supply.

Frequently Asked Questions (FAQ)

Will Zenrows charge me extra for integrating Flowise?

Zenrows doesn’t charge you extra for integrating with Flowise. Flowise is also an open-source tool with no initial cost implications. Although Flowise charges for advanced features like cloud hosting, the free plan provides most of what you need, including building Agentflows, Chatflows, Assistants, and more.

Are there alternatives to Flowise?

Yes, Zenrows integrates with other AI-powered low-code and no-code workflow tools, including Pipedream, Make, Clay, Zapier, and more. Check our integration options for more details.

Can I save the extracted data from Flowise to Google Sheets?

Although Flowise supports Google Sheets as a tool, connecting to it involves some technical steps. Currently, Flowise has limitations in terms of Google Sheets write operations compared to tools like Zapier, which offer easier and more seamless integration.

Does Zenrows bypass anti-bots on Flowise?

Yes, integrating Zenrows with Flowise enables your scraper agent to bypass anti-bots easily.