I am considering devising a pipeline to create a more structured wiki but would need to scrape the forum. What is the policy on scraping?
4 Likes
Are you feeding the data into an AI or something else?
I actually havent thought that far ahead. The first step is just determining the availiability of the data. I could concieve of doing it with or without AI. Having an AI in the pipeline would simplify classification but also increase error. If not using it for a first pass, it would probably have to be far more restrictive at first.
Iām not really sure how youād turn scraped data from the forum into a useful wiki without a lot of manual work, given the number of posts, some of which are old and no longer definitive.
This seems like an idea without a plan. 
Happy to think about it more. My plans tend to develop as i go. That is the fun of it. If i get around to it, i'll give you a more concrete plan as it seems like the potential availability of the data is contingent upon it.
I have a longstanding interest in information distribution and the current system of answering each query individually on here seems very inefficient to me. It seems like solutions would be far more accessible in a traversible wiki. From a systems perspective, if the simple questions are easily answered, they give way to ever more complex and interesting questions and problems (to us tinkerers). It would also solve the age old problem of the loss of organisational knowldge that inevitably occurs when existing gurus retire etc.
From my perspective, indexability, interconnected linking, and applied/working examples, with explainers for frequent pitfalls are essential. This doesnt address the problem of people not wanting to read the wiki and posing the question anyway, however...
1 Like
Artificial Intelligence cannot āthinkā (although some people seem to believe that), but for purposes like what you propose, it would probably save you a million hours. Further tuning could be done by humans making corrections (to which AI could react by correcting similar errors, etc. etc.) - I think itās a great idea. In spite of some help I received for searching stuff (I learned about using escape characters by reading the help page for the forum software makers) I still find it hard to find stuff. I donāt think the forum supports AND and OR properly, let alone NEAR.
Others did scraping here I believe, at least to save all DO Manual mages, and compile them into a Microsoft .chm file (which I used for a while, much easier to find stuff).
Using many more categories and/or tags could have been helpful. When you create a post, choosing āButtons/Scriptsā is extremely undifferentiated. Optional tags: which script language? You can choose between āscriptā and ājscriptā. No āvbscriptā available. Good luck if you want help with vbscript. But those are silly details. The power of the forum is in the first place in its quick answers by professionals - I have no complaints about that.
Back to AI: some engine to convert jscript into vbscript and vice versa would probably be vey welcome for many users here - I think AI could help with that. I used an online converter when I decided to convert all my vb scripts to jscript - I think it was VB.NET to JavaScript Converter - but I had to do it in chunks for large scripts (didnāt want to pay) and the conversion was far from perfect (which was why I didnāt want to pay). Iām sure there are already better ones available, very probably supported by AI.
2 Likes
You mean you want to make a get rich quick app.
Yes it can.
They donāt just give us the reigns to the real Gemini, to do with her as we please. There is a reason they need all those data centers, and that reason isnāt to help us write emails.
A profit motive didnt even occur to me. I am just idealistic and perfectionistic. Which is to say, i just like efficiency 
I would add that while cynicism is usually founded, the cost of being a cynic, like the cost of being a liar, isnt that you cannot be trusted, it is that you can trust no one else. (Not calling you a liar to be clear; just making a philosophical point)
1 Like
This is not the place for a philosophical discussion, but AI canot āthinkā the way humans do. AI can produce summaries from large piles of knowledge but thatās not the same thing. Experts know this. I also know it from experience, as Iām a long time debater on forums, FB etc. where folks are discussing politics, philosophy, world views, etc. and where some people counter-argued with what they could get from Gemini, Grok3/Grok4 or whatever engine out there, and they just made a laughing stock of themselves whenever they did the copy/paste thing rather than using the AI results to feed their own thinking and then give a decent response. You can usually tell within seconds whether or not youāre talking to some AI bot.
PS. By the way, I donāt do conspiracy stuff. Not even against Google. The āBig Sevenā are American companies because the US is a country where they understand the value of free enterpreneurship. Itās that simple.
But thatāll be my only words on AI, I think. Too far off-topic. I shouldnāt even have mentioned it - but I wanted to emphasize that the suggestion to use AI was really very good for the purpose of structuring the data of a forum.
1 Like