Hello,
i have asked OpenAI ChatGPT the following question:
Hello, can you learn how Directory Opus works from Introduction [Directory Opus Manual], or is it not allowed for you?
OpenAI ChatGPT answered:
Hello! I cannot directly browse the webpage Introduction [Directory Opus Manual] because I don’t have access to external websites. However, I can give you a general explanation of Directory Opus based on my current knowledge.
Second question:
Is this due to the robots.txt?
OpenAI ChatGPT answered:
Yes, that could indeed be due to the robots.txt file of the website.
The robots.txt file is a file that instructs web crawlers and search engines about which parts of a website they are allowed to index and which parts they are not. If a website specifies in its robots.txt file that crawlers are not allowed to access certain sections or the entire site, I would not be able to retrieve that content.
Since I can only access websites that are allowed for crawlers, and I don’t have a direct way of checking the robots.txt, it’s possible that such a restriction is in place if you cannot access the page directly.
If you'd like to learn more about the robots.txt of a website, you can simply visit the URL of the robots.txt file by appending /robots.txt to the domain (e.g., https://docs.dopus.com/robots.txt).
The last sentence is not true (https://docs.dopus.com/robots.txt do not exist) and i don't care if you allow it, but it would explain the wrong answers to dopus from ChatGPT. Especially when it comes to scripting in dopus.