By the end of this tutorial, you\u2019ll have a reliable system that can monitor indexing status for hundreds of URLs without lifting a finger.<\/p>\n\n\n\n
Step 1: Setting Up the Trigger<\/strong><\/p>\n\n\n\n The first building block of this workflow is the trigger<\/strong>. Without it, you\u2019d have to manually start the process every time, which defeats the purpose of automation. Instead, we want our indexing checks to run on a schedule\u2014quietly in the background\u2014so that our Google Sheet is always up to date with the latest results.<\/p>\n\n\n\n In n8n, this is handled by the Cron Trigger node<\/strong>. The Cron node is incredibly flexible; it lets you schedule workflows to run at fixed intervals, specific times of day, or even just once on a certain date. For our indexing checker, we set it to run every 8 hours daily<\/strong>. This ensures that if Google\u2019s index changes\u2014whether pages get added or drop out\u2014we\u2019ll detect it within a reasonable timeframe without constantly bombarding Google with requests.<\/p>\n\n\n\n Here\u2019s how the Cron node was configured:<\/p>\n\n\n\n This means the workflow automatically fires three times per day (every 8 hours). That frequency is a sweet spot: it\u2019s frequent enough to catch indexation changes quickly, but not so frequent that it risks unnecessary load or triggering anti-bot defenses.<\/p>\n\n\n\n Why periodic scheduling matters<\/strong><\/p>\n\n\n\n Indexing is not static. A URL that is indexed today might be deindexed tomorrow due to crawl budget issues, canonical conflicts, noindex tags, or algorithmic changes. By running checks automatically, you create a historical log of indexation health, which is incredibly valuable for diagnosing long-term SEO problems. For example:<\/p>\n\n\n\n Once this node is in place, the workflow doesn\u2019t need any manual input to start. It becomes a hands-free system<\/strong> that wakes itself up every 8 hours, prepares the next batch of URLs, and checks their status.<\/p>\n\n\n\n Step 2: Defining the URL List<\/strong><\/p>\n\n\n\n With the trigger in place, the next step is to tell our workflow which URLs<\/strong> we want to monitor. In n8n, the easiest way to do this is with the Set Node<\/strong>. This node lets us create and store static values that can be passed down to other nodes in the workflow. In our case, we\u2019re going to define a list of URLs that need their indexing status checked.<\/p>\n\n\n\n Inside the Set Node<\/strong>, I created a new field called urlList. Under the Value<\/strong> section, I entered all the URLs I want to track, separated by commas. For example:<\/p>\n\n\n\nhttps:\/\/www.vesbhusha.com\/types-of-uniforms-in-the-hotel-industry\/,\nhttps:\/\/www.vesbhusha.com\/common-mistakes-to-avoid-when-choosing-corporate-uniforms\/,\nhttps:\/\/www.vesbhusha.com\/importance-of-uniforms-in-branding\/\n\n\n\n When this node executes, the output is a JSON object containing that single field:<\/p>\n\n\n\n{\n “urlList”: “https:\/\/www.vesbhusha.com\/types-of-uniforms-in-the-hotel-industry\/, https:\/\/www.vesbhusha.com\/common-mistakes-to-avoid-when-choosing-corporate-uniforms\/, https:\/\/www.vesbhusha.com\/importance-of-uniforms-in-branding\/”\n}\n\n\n\n\n Why use the Set Node?<\/strong><\/p>\n\n\n\n Once we have a list of URLs defined in the Set Node, the next step is to prepare them in a format that can actually be queried on Google. If we were to paste all of them directly into Google at once, the workflow would quickly fail. That\u2019s because Google needs one query at a time to return clear results, and sending multiple URLs in bulk will either break the process or return mixed results that are hard to analyze.<\/p>\n\n\n\n To solve this, we use a Function Node<\/strong> in n8n that runs a short JavaScript script. The purpose of this node is simple: take the comma-separated list of URLs we created earlier and split them into individual items. Each item is then transformed into a properly formatted Google search query using the site:<\/strong> operator. This operator is one of the most useful tricks for SEOs\u2014it allows us to check if a specific page exists in Google\u2019s index. For example, searching site:example.com\/page will either show the indexed page or display a message that no results were found.<\/p>\n\n\n\n The Function Node code does three things:<\/p>\n\n\n\n By doing this, the workflow ensures that each URL is ready to be processed one by one in the upcoming loop. The output of this node is a structured list where every item contains both the original URL and its corresponding Google search query.<\/p>\n\n\n\n This is a crucial step because it transforms our simple static list into actionable search requests that Google can understand. Without this preparation, the following nodes would not know how to handle multiple URLs in a systematic and reliable way.<\/p>\n\n\n\n Step 4: Looping Over Items Safely<\/strong><\/p>\n\n\n\n Now that we have a clean list of prepared Google search queries, the next step is to make sure they are executed one by one in a controlled manner. This is where the Loop Over Items node<\/strong> in n8n comes in. By default, n8n can process multiple items in parallel, but when we\u2019re dealing with Google requests, parallel execution can easily cause issues. Too many simultaneous requests look suspicious to Google, and that may trigger captchas or even temporary blocks.<\/p>\n\n\n\n To avoid this, we configure the workflow to process only one URL at a time<\/strong>. In the Loop Over Items node, we set the Batch Size<\/strong> to 1. This ensures that each Google query is executed individually, giving us full control over the sequence of events. With this setup, each URL will go through the next steps in isolation: randomized User-Agent assignment, timed delay, request execution, and result parsing.<\/p>\n\n\n\n This design also helps in debugging. If something goes wrong with a specific URL, it is much easier to track the error when URLs are processed sequentially rather than in parallel. For large websites with hundreds of URLs, sequential looping might take longer, but it\u2019s safer and much more reliable.<\/p>\n\n\n\n Another benefit of using the Loop node is scalability<\/strong>. Whether you are testing 10 URLs or 1,000, the logic remains the same\u2014one at a time, steady, and predictable. You can even extend this logic with conditional branches. For example, you could skip certain URLs under specific conditions or route failed URLs into a separate log for review.<\/p>\n\n\n\n In short, the Loop Over Items node ensures that our indexing checker behaves responsibly, reduces the risk of being flagged by Google, and maintains a professional-grade level of reliability. Without this step, the workflow could overwhelm Google\u2019s systems or produce incomplete results.<\/p>\n\n\n\n Step 5: Adding Random User-Agent and Delay<\/strong><\/p>\n\n\n\n One of the key challenges when automating Google queries is avoiding detection. If every request comes from the same \u201cbrowser fingerprint\u201d and occurs at the same interval, Google can quickly recognize the activity as automated. This might result in temporary IP blocks, captchas, or incomplete search results. To make our workflow behave more like a human user, we need to introduce randomness in two critical areas: the browser identity<\/strong> and the timing of requests<\/strong>.<\/p>\n\n\n\n This is handled through a Function Node in n8n<\/strong>. Inside the node, we define an array of common User-Agent strings representing different browsers and devices. This includes Chrome on Windows, Firefox on Linux, Safari on iOS, Edge on macOS, and more. For each URL request, the workflow randomly selects one of these User-Agents and attaches it to the HTTP header. As a result, one request may look like it came from a Windows desktop, while the next might appear as if it was sent from an iPhone.<\/p>\n\n\n\n In addition to rotating User-Agents, the code also generates a random delay between 12 and 30 seconds. This ensures that requests are not sent at fixed intervals, which would otherwise look suspicious. Instead, the workflow pauses unpredictably, much like how a real person might pause while browsing.<\/p>\n\n\n\n This step is critical because it adds a layer of authenticity to the automation. Without it, even the best-designed workflow could be flagged by Google after only a few runs. With User-Agent rotation and randomized delays, however, the workflow remains stable, resilient, and safe to run multiple times a day without issue.<\/p>\n\n\n\n This combination of tactics\u2014browser rotation and natural timing\u2014transforms the workflow from a simple script into a robust SEO tool that can operate at scale without tripping Google\u2019s defenses.<\/p>\n\n\n\n Step 6: Implementing the Wait Node<\/strong><\/p>\n\n\n\n Once the workflow has selected a random User-Agent and generated a randomized delay, we need to actually apply that pause before sending the request to Google. This is where the Wait Node<\/strong> in n8n comes into play.<\/p>\n\n\n\n The Wait Node is designed to temporarily hold the workflow until certain conditions are met. In our case, the condition is simply time. Instead of sending requests instantly back-to-back, the Wait Node forces each iteration of the loop to pause for a number of seconds. By using the delay value generated in the previous step, the workflow introduces realistic timing between queries.<\/p>\n\n\n\n This delay is absolutely crucial. Think of how a human behaves when searching Google: nobody enters ten different \u201csite:\u201d searches per second. There are always pauses\u2014typing the query, reading results, clicking through. By inserting these pauses, our automation mimics real-world behavior and avoids looking like a bot.<\/p>\n\n\n\n In the Wait Node, the configuration is set to Resume After Time Interval<\/strong>. The input value is dynamic, meaning we don\u2019t hardcode \u201c20 seconds\u201d or \u201c30 seconds.\u201d Instead, the node reads {{$json.delay}} directly from the output of the Random Agent step. That means every iteration can have a different delay\u2014sometimes 18 seconds, sometimes 25, sometimes 30. This unpredictability is what makes the workflow resilient.<\/p>\n\n\n\n Without this step, even rotating User-Agents would not be enough. Google could still detect an unnatural pattern if requests were being sent at perfectly regular intervals. With the Wait Node in place, every request becomes staggered and inconsistent\u2014just like genuine human searches.<\/p>\n\n\n\n By the end of this step, each URL in your list is ready to be queried in a safe and natural way. The workflow now behaves almost indistinguishably from a person manually performing \u201csite:\u201d searches throughout the day. This makes the entire system both effective and sustainable for long-term use.<\/p>\n\n\n\n Step 7: Executing the Google Search<\/strong><\/p>\n\n\n\n With delays and User-Agent rotation in place, the workflow is now ready to perform the actual search on Google. This is handled using the HTTP Request Node<\/strong> in n8n. The purpose of this node is straightforward: it sends a GET request to Google with the prepared site: query, and captures the raw HTML response.<\/p>\n\n\n\n The most important part of this step is ensuring that Google receives the request in a way that looks natural. To do this, we configure the HTTP Request node with the following key settings:<\/p>\n\n\n\n For example, if your URL is:<\/p>\n\n\n\n https:\/\/www.google.com\/search?q=site:https:\/\/www.vesbhusha.com\/types-of-uniforms-in-the-hotel-industry\/<\/p>\n\n\n\n The HTTP Request node fetches the full HTML of the search results page for that query. This raw HTML doesn\u2019t look readable at first, but it contains the clues we need to determine whether the page is indexed or not.<\/p>\n\n\n\n By capturing the Google search result this way, we have complete control. We are no longer relying on external SEO tools or even the Search Console API. Instead, we\u2019re directly replicating what a human SEO would do\u2014search for a URL using the site: operator\u2014but in an automated, scalable way.<\/p>\n\n\n\n This step is the gateway between preparation and analysis. Once the HTML is captured, the next node can parse it and decide whether the target URL is indexed.<\/p>\n\n\n\n Step 8: Parsing Results and Checking Indexing Status<\/strong><\/p>\n\n\n\n At this stage, we already have the raw HTML from Google Search for each site: query. The next challenge is to interpret that response and decide if the page is indexed or not. Google doesn\u2019t provide a simple \u201cyes\/no\u201d answer in the search result HTML\u2014it has to be inferred from the content.<\/p>\n\n\n\n In our workflow, we solve this problem using a Function Node with JavaScript<\/strong>. The logic is simple but effective: search the returned HTML for a specific phrase that Google displays when no results are found. Typically, this message looks like \u201cdid not match any documents\u201d<\/em>. If this phrase is present in the response, it means Google couldn\u2019t find the URL\u2014so the page is not indexed. If the phrase is missing, the page is assumed to be indexed.<\/p>\n\n\n\n The function performs three main tasks:<\/p>\n\n\n\n This makes the workflow extremely efficient. Instead of dealing with messy HTML or regex parsing, we rely on a single reliable text check. Of course, this logic can be extended later to handle edge cases or localized Google messages, but for most scenarios, it works perfectly.<\/p>\n\n\n\n The output from this node is now structured, human-readable, and ready to be logged. With every URL clearly marked as Indexed or Not Indexed, we can confidently pass the data into our reporting system\u2014in this case, Google Sheets.<\/p>\n\n\n\n Step 9: Logging Results in Google Sheets<\/strong><\/p>\n\n\n\n Once the indexing status has been determined for each URL, the final step is to save these results somewhere we can easily monitor them over time. For this workflow, we use the Google Sheets node in n8n<\/strong>. Sheets is a great choice because it provides an accessible, shareable, and visual way to track indexation changes without requiring any additional infrastructure.<\/p>\n\n\n\n In the Google Sheets node, we configure the workflow to either append a new row<\/strong> or update an existing row<\/strong> depending on whether the URL already exists in the sheet. This makes the system flexible: if a URL is being checked for the first time, it\u2019s added as a new entry. If it\u2019s already there, its isIndexed status simply updates with the latest result.<\/p>\n\n\n\n Typical Configuration:<\/strong><\/p>\n\n\n\n The beauty of this approach is that it provides long-term visibility<\/strong> into indexation trends. Over time, you can see which URLs frequently drop out of Google\u2019s index, which remain stable, and how changes to your site may affect crawling and indexing. This data can also be exported, filtered, or visualized in Google Data Studio for deeper reporting.<\/p>\n\n\n\n By storing results in Google Sheets, we make the workflow not just automated, but also actionable. The information is always there, organized, and ready for SEO decision-making<\/p>\n\n\n\n Step 10: Final Node (No Operation)<\/strong><\/p>\n\n\n\n After processing each URL, applying delays, running Google searches, parsing the results, and saving everything into Google Sheets, the workflow needs a clear stopping point. This is where the No Operation (Do Nothing)<\/strong> node comes in.<\/p>\n\n\n\n At first glance, this step may seem unnecessary because it doesn\u2019t actively perform any task. However, in practice, it serves an important role in keeping the workflow clean and organized. The No Operation node acts as a placeholder that signals the official end of the process. This makes the entire sequence easier to read and maintain, especially in larger workflows where many branches of logic may converge.<\/p>\n\n\n\n Why use a No Operation node?<\/strong><\/p>\n\n\n\n Error Handling<\/strong>: In some workflows, ending abruptly on a Google Sheets update could leave things unclear if you later debug issues. Having a dedicated endpoint node keeps the workflow structured.<\/p>\n\n\n\n Conclusion<\/strong><\/p>\n\n\n\n Managing SEO for large websites can be overwhelming, especially when it comes to monitoring whether hundreds or even thousands of URLs remain indexed in Google. Manually checking these with \u201csite:\u201d searches is time-consuming, inconsistent, and simply not practical at scale. That\u2019s why automation is no longer a luxury\u2014it\u2019s a necessity.<\/p>\n\n\n\n By leveraging the power of n8n<\/strong>, we built a fully automated workflow that performs all the heavy lifting for you. From triggering the workflow on a schedule, splitting URLs into manageable queries, introducing random delays and User-Agent rotation, fetching results with HTTP requests, parsing responses for indexing status, and finally logging everything neatly into Google Sheets\u2014this workflow covers the entire lifecycle of indexation monitoring.<\/p>\n\n\n\n The beauty of this setup is its simplicity and scalability. Once configured, it runs in the background, updating your Sheets automatically with fresh data every few hours. You\u2019ll always have a historical record of indexation trends that can help diagnose problems early, spot sudden drops, or confirm improvements after implementing SEO changes.<\/p>\n\n\n\n More importantly, the workflow is flexible. The No Operation node<\/strong> at the end is a deliberate design choice\u2014it allows you to extend the system whenever you\u2019re ready. For example, you could add Slack alerts for URLs that remain unindexed after several runs, email reports for clients, or even connect it to a database or BI dashboard.<\/p>\n\n\n\n This isn\u2019t just about saving time (though it will save hours each week). It\u2019s about creating a reliable, repeatable, and scalable process<\/strong> for one of SEO\u2019s most fundamental needs: making sure your content is actually findable in Google.<\/p>\n\n\n\n If you\u2019re an agency, this workflow can be a game-changer for client reporting. If you\u2019re an in-house SEO, it can give you peace of mind knowing nothing slips through the cracks<\/p>\n","protected":false},"excerpt":{"rendered":" If you\u2019ve ever managed SEO for a large website, you already know how painful it can be to manually check whether all your important URLs are indexed in Google. For smaller sites, running the occasional \u201csite:\u201d search works fine. But when you\u2019re handling hundreds or even thousands of URLs across different projects, it becomes nearly […]<\/p>\n","protected":false},"author":1,"featured_media":1374,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"yoast_head":"\n\n
![\"\"](\"https:\/\/www.softnoesis.com\/blog\/wp-content\/uploads\/2025\/09\/image-1-1024x445.png\")
<\/figure>\n\n\n\n\n
\n
![\"\"](\"https:\/\/www.softnoesis.com\/blog\/wp-content\/uploads\/2025\/09\/image-2-1024x454.png\")
<\/figure>\n\n\n\nStep 3: Preparing the Google Search Query<\/span><\/h2><\/span><\/p>\n\n\n\n
\n
\n\n
![\"\"](\"https:\/\/www.softnoesis.com\/blog\/wp-content\/uploads\/2025\/09\/53e7ae8c-9256-44d3-9ed0-cdd99efd5801-1024x421.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/www.softnoesis.com\/blog\/wp-content\/uploads\/2025\/09\/102f14f8-fe22-45cc-815e-ae9d4d4cad35-1024x447.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/www.softnoesis.com\/blog\/wp-content\/uploads\/2025\/09\/870eb395-9e17-4529-b89c-da3b7065c66f-1024x437.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/www.softnoesis.com\/blog\/wp-content\/uploads\/2025\/09\/564348cd-5b4d-4214-b4d0-946150c6f676-1024x423.png\")
<\/figure>\n\n\n\n\n\/\/ Wait Google Search Node\n\/\/ Uses dynamic delay value from previous step\n\/\/ Example: delay = 20 seconds\n\n{\n \"searchUrl\": \"https:\/\/www.google.com\/search?q=site:https:\/\/www.vesbhusha.com\/types-of-uniforms-in-the-hotel-industry\/\",\n \"originalUrl\": \"https:\/\/www.vesbhusha.com\/types-of-uniforms-in-the-hotel-industry\/\",\n \"userAgent\": \"Mozilla\/5.0 (iPad; CPU OS 17_2 like Mac OS X)...Safari\/604.1\",\n \"delay\": 20\n}\n<\/code><\/pre>\n\n\n\n\n
![\"\"](\"https:\/\/www.softnoesis.com\/blog\/wp-content\/uploads\/2025\/09\/88e4da21-428b-4c56-a98d-05781070c905-1024x434.png\")
<\/figure>\n\n\n\n\n\/\/ HTTP Request Node Configuration\nRequest Method: GET\nURL: {{$json[\"searchUrl\"]}}\nResponse Format: String\nProperty Name: data\nHeaders:\n User-Agent: {{$json.userAgent}}\n<\/code><\/pre>\n\n\n\n\n
<\/li>\n\n\n\n
\n\n
<\/li>\n<\/ul>\n<\/li>\n\n\n\n
\n\n
![\"\"](\"https:\/\/www.softnoesis.com\/blog\/wp-content\/uploads\/2025\/09\/547406b1-d317-45da-be58-333079b9ac61-1024x434.png\")
<\/figure>\n\n\n\n\nconst html = $json[\"body\"];\nconst found = typeof html === \"string\" && html.includes(\"did not match any documents\") ? false : true;\n\nconst itemIndex = $runIndex;\nconst allPreviousData = $node[\"Wait Google Search\"].json;\n\nreturn [{\n json: {\n searchUrl: allPreviousData.originalUrl || 'TEST_URL',\n isIndexed: found,\n }\n}];\n<\/code><\/pre>\n\n\n\n\n
![\"\"](\"https:\/\/www.softnoesis.com\/blog\/wp-content\/uploads\/2025\/09\/a0215c55-b089-4757-ba20-f383f0fd30c2-1024x427.png\")
<\/figure>\n\n\n\n\n \n
\n \n URL<\/th>\n isIndexed<\/th>\n Last Checked<\/th>\n <\/tr>\n <\/thead>\n \n https:\/\/www.vesbhusha.com\/types-of-uniforms-in-the-hotel-industry\/<\/a><\/td>\n true<\/td>\n 2025-09-11<\/td>\n <\/tr>\n \n https:\/\/www.vesbhusha.com\/common-mistakes-to-avoid-when-choosing-corporate-uniforms\/<\/a><\/td>\n false<\/td>\n 2025-09-11<\/td>\n <\/tr>\n <\/tbody>\n<\/table>\n\n\n\n \n
<\/li>\n\n\n\n
<\/li>\n<\/ol>\n\n\n\n