# Is OpenAI ChatGPT blocked from crawling the dopus website and manual?

**URL:** <https://resource.dopus.com/t/is-openai-chatgpt-blocked-from-crawling-the-dopus-website-and-manual/53654>\
**Category:** Site Feedback\
**Created:** [November 29, 2024, 6:48pm UTC](https://resource.dopus.com/t/is-openai-chatgpt-blocked-from-crawling-the-dopus-website-and-manual/53654 "2024-11-29T18:48:33Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Hardkorn](https://resource.dopus.com/user_avatar/resource.dopus.com/hardkorn/32/48254_2.png) [@Hardkorn](https://resource.dopus.com/u/Hardkorn)\
**Post date:** [November 29, 2024, 6:48pm UTC](https://resource.dopus.com/t/is-openai-chatgpt-blocked-from-crawling-the-dopus-website-and-manual/53654/1 "2024-11-29T18:48:33Z")

</div>

Hello,

**i have asked OpenAI ChatGPT the following question:**  
Hello, can you learn how Directory Opus works from [Introduction [Directory Opus Manual]](https://docs.dopus.com/doku.php?id=introduction), or is it not allowed for you?

**OpenAI ChatGPT answered:**  
Hello! I cannot directly browse the webpage **[Introduction [Directory Opus Manual]](https://docs.dopus.com/doku.php?id=introduction)** because I don’t have access to external websites. However, I can give you a general explanation of **Directory Opus** based on my current knowledge.

**Second question:**  
Is this due to the robots.txt?

**OpenAI ChatGPT answered:**  
Yes, that could indeed be due to the **robots.txt** file of the website.

The **robots.txt** file is a file that instructs web crawlers and search engines about which parts of a website they are allowed to index and which parts they are not. If a website specifies in its **robots.txt** file that crawlers are not allowed to access certain sections or the entire site, I would not be able to retrieve that content.

Since I can only access websites that are allowed for crawlers, and I don’t have a direct way of checking the **robots.txt** , it’s possible that such a restriction is in place if you cannot access the page directly.

If you'd like to learn more about the **robots.txt** of a website, you can simply visit the URL of the **robots.txt** file by appending **/robots.txt** to the domain (e.g., `https://docs.dopus.com/robots.txt`).

The last sentence is not true ([https://docs.dopus.com/robots.txt](https://docs.dopus.com/robots.txt) do not exist) and i don't care if you allow it, but it would explain the wrong answers to dopus from ChatGPT. Especially when it comes to scripting in dopus.

---

<div class="post-metadata">

**Author:** ![Leo](https://resource.dopus.com/user_avatar/resource.dopus.com/leo/32/69732_2.png) [@Leo](https://resource.dopus.com/u/Leo)\
**Post date:** [November 29, 2024, 6:53pm UTC](https://resource.dopus.com/t/is-openai-chatgpt-blocked-from-crawling-the-dopus-website-and-manual/53654/2 "2024-11-29T18:53:03Z")

</div>

> because I don’t have access to external websites

It seems to be saying it has no access to any websites at all. Maybe it is confused about itself, like AI is confused about everything else.