ChatGPT doesn't provide any information on how it got the answer it give you.
So you have to trust that it not feeding you complete bullshit.
Unlike Google who always provides a source link to where it got it answer so you can always research the answer yourself.
1. I tested it by asking it to write Python scripts around an area where I consider myself to be a domain expert - AWS and the Python script was correct. I asked it to make changes and the changes were correct.
2. I gave it some Python scripts I wrote using the AWS SDK and it described what it did as well as I would expect in an interview. On top of that, I asked it why I would use it and the purpose of one of the methods. The method was trying to determine whether it was running from the root account or a delegate account.
It has to run slightly different depending on where it’s running from (Cloudformation stacksets). It answered the question as well as I would expect an interview candidate to.
The answers to my questions - write code to do $x was proven correct by the Python interpreter.
I also asked it generic AWS questions like I would in an interview. I knew the answers.
If you look for a random high school math word problem on the internet. It won’t just tell you the answer, it will tell you how it derived it.
I haven’t used it for random search. I had it open and was using it to generate code and documentation for a project I was working on.